Conversation Parsing: JSONL, Events, and SQLite Read Boundaries

Decoding a file into objects completes only the first parsing step. Conversation imports also identify message records, restore ordering, associate tool calls, and establish whether the database state being read is complete. This article follows ChatCrystal’s pinned implementation and uses synthetic fixtures to test line and snapshot boundaries without reading personal conversation directories.

Locate the read path by storage format

The current parsers have three different entry paths. One generic JSON.parse does not cover every tool.

Path	Current read strategy	Semantics requiring separate checks
Claude Code and Codex JSONL	createReadStream with readline	Event types, message identity, tool association, and deduplication scope
Copilot JSON / JSONL	Whole JSON file; first line only for JSONL	Snapshot wrappers and message content within requests
Cursor and Trae vscdb	Read main database bytes and load them into sql.js	Internal keys and values, database snapshots, and timestamp origins

See SourceAdapter design for the shared output interface. A common interface neither makes input formats identical nor replaces runtime validation of JSON fields.

Line reads reduce input buffering, without fixing total memory

The Claude Code parser reads lines, skips malformed JSON, retains records matching role and relevance rules, cleans some XML content, and accumulates and sorts messages. Input is not loaded into one initial string, but the message array still grows and a single line can be large. Using readline does not establish constant memory or immunity to OOM.

Save this as lines.mjs and run with Node 24+ without dependencies. It demonstrates teaching rules for line framing and selection, rather than a complete tool parser. Two stream chunks deliberately split a JSON record, and lines use CRLF.

import assert from 'node:assert/strict';
import { Readable } from 'node:stream';
import { createInterface } from 'node:readline';

const input = [
  '{"type":"message","text":"first"}',
  '{broken',
  '{"type":"progress","text":"noise"}',
  '{"type":"message","text":"second"}',
].join('\r\n');
const stream = Readable.from([input.slice(0, 18), input.slice(18)]);
const lines = createInterface({ input: stream, crlfDelay: Infinity });
const messages = [];
let malformed = 0;
for await (const line of lines) {
  if (!line.trim()) continue;
  let event;
  try {
    event = JSON.parse(line);
  } catch {
    malformed++;
    continue;
  }
  if (event?.type === 'message' && typeof event.text === 'string') {
    messages.push(event.text);
  }
}
assert.deepEqual(messages, ['first', 'second']);
assert.equal(malformed, 1);
console.log('line framing, selection, and malformed count pass');

Change the expected malformed count to 0 and the program should fail; restore 1 and it passes. The teaching code adds this diagnostic count. It does not establish that existing parsers log every damaged record. These assertions cover the constructed input, without proving recovery from every invalid JSON record or oversized line.

Event sequences need format-specific normalization

The Codex parser handles session_meta, event_msg, and response_item separately. An answer can appear in more than one event: the current code skips response_item assistant text when it exactly matches the last assistant text. That local rule does not provide global event deduplication. Tool calls also mark the previous assistant’s hasToolUse, so retaining only events with text fields cannot establish complete tool semantics.

The Copilot parser reads the first snapshot line on its JSONL path, accepts the current snapshot wrapper or a legacy object, and extracts user messages and responses from requests. It does not replay incremental patches from later lines. An empty, damaged, or unexpected first line is not necessarily repaired by subsequent lines. Identify the exact read path before diagnosing missing messages.

Public regression fixtures should separately include duplicate answer events, tool-only records, damaged lines, unknown events, and missing times. Expected results should specify ordering, roles, text, and tool flags, rather than only asserting that parsing does not throw.

Database byte copies have a distinct read time

The current vscdb.ts reads the main file with readFileSync and opens an in-memory sql.js database. On failure it waits 500ms and retries once, returning null after a second failure. It does not read WAL or coordinate a consistent backup with the source program. One retry does not establish recovery from all lock contention or damaged reads.

For a source using WAL mode, committed content can still reside in WAL. Copying only the main file does not establish a complete, current snapshot; see the SQLite WAL documentation. This explains the read strategy’s boundary, without claiming that the exercise reproduced a user’s data loss.

Install dependencies in a new temporary directory, save the following snapshot.mjs, and run with Node 24+:

npm init -y
npm install --ignore-scripts --no-audit --no-fund sql.js@1.14.2

import assert from 'node:assert/strict';
import initSqlJs from 'sql.js';

const SQL = await initSqlJs();
const live = new SQL.Database();
let copy;
try {
  live.run('CREATE TABLE kv (key TEXT PRIMARY KEY, value TEXT)');
  live.run("INSERT INTO kv VALUES ('message', 'first')");
  const bytes = live.export();
  copy = new SQL.Database(bytes);
  live.run("UPDATE kv SET value = 'second' WHERE key = 'message'");
  const read = (db) => db.exec('SELECT value FROM kv')[0].values[0][0];
  assert.equal(read(copy), 'first');
  assert.equal(read(live), 'second');
  console.log('exported bytes are a separate snapshot');
} finally {
  copy?.close();
  live.close();
}

Change read(copy)’s expectation to second and it should fail; restore first and it passes. This in-memory experiment establishes only that exported bytes do not follow subsequent changes in the original instance. It does not test file locks, WAL, concurrent writes, or openVscdb recovery.

Locate message content through SQLite keys

Once database bytes are available, the parser must select the right table, key, and conversation. These paths describe the pinned implementation, rather than a universal specification for every vscdb file.

Source	Conversation lookup	Message extraction
Cursor	Read allComposers from workspace ItemTable’s composer.composerData; scanning also discovers orphan conversations without workspace entries	Match bubbleId:conversationID: prefixes in global cursorDiskKV and parse each value
Trae	Parse ItemTable’s memento/icube-ai-agent-storage value and find sessionId in list	Read that conversation’s messages, order by turnIndex, and extract user or assistant content

Cursor treats bubble.type 1 as user and other values as assistant, skips empty assistant text, and finally sorts by message time. A newer bubble._v produces a warning; that does not establish full compatibility with its structure.

Trae’s SQL value can be a string or a Uint8Array requiring UTF-8 decoding. Direct assistant content takes precedence; otherwise extraction reads reply material from agentTaskContent’s proposal, finish-step thoughts in guideline.planItems, and related locations, while collecting thinking separately. Ordering timestamps can be synthesized as increasing values from the conversation’s start time when necessary. They are not all exact original message times.

These are current behaviors in the source linked below. The snapshot exercise did not run either adapter. Format integration checks should construct the relevant tables, keys, TEXT/BLOB values, missing conversations, and empty messages, then assert actual adapter output.

Check metadata separately from message time

The Cursor scan uses conversation creation time when available, falling back to database modification time. The Trae scan prefers conversation update time. Neither supports a blanket statement that file modification time is every message’s timestamp. Multiple conversations in one database can also share its file size.

Normalized output gives downstream consumers a common shape. It does not guarantee collision-free identities, observable omissions, or complete compatibility with new formats. Keep fixtures and failure evidence for scanning, parsing, and persistence separately. Measure time and memory using the bulk import measurement method; the offline assertions here produce no performance conclusion.