--- title: "Conversation Parsing: JSONL, Events, and SQLite Read Boundaries" description: Trace ChatCrystal parsers with offline fixtures to test line framing and SQLite snapshots, separating decoding from message normalization guarantees. image: "https://zengliangyi.com/og/blog/en/multi-format-conversation-parsing.png" --- Decoding a file into objects completes only the first parsing step. Conversation imports also identify message records, restore ordering, associate tool calls, and establish whether the database state being read is complete. This article follows ChatCrystal’s pinned implementation and uses synthetic fixtures to test line and snapshot boundaries without reading personal conversation directories. ## Locate the read path by storage format The current parsers have three different entry paths. One generic JSON.parse does not cover every tool. | Path | Current read strategy | Semantics requiring separate checks | | --- | --- | --- | | Claude Code and Codex JSONL | createReadStream with readline | Event types, message identity, tool association, and deduplication scope | | Copilot JSON / JSONL | Whole JSON file; first line only for JSONL | Snapshot wrappers and message content within requests | | Cursor and Trae vscdb | Read main database bytes and load them into sql.js | Internal keys and values, database snapshots, and timestamp origins | See [SourceAdapter design](/en/blog/source-adapter-design/) for the shared output interface. A common interface neither makes input formats identical nor replaces runtime validation of JSON fields. ## Line reads reduce input buffering, without fixing total memory The [Claude Code parser](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/adapters/claude-code.ts) reads lines, skips malformed JSON, retains records matching role and relevance rules, cleans some XML content, and accumulates and sorts messages. Input is not loaded into one initial string, but the message array still grows and a single line can be large. Using readline does not establish constant memory or immunity to OOM. Save this as `lines.mjs` and run with Node 24+ without dependencies. It demonstrates **teaching rules for line framing and selection**, rather than a complete tool parser. Two stream chunks deliberately split a JSON record, and lines use CRLF. ```js import assert from 'node:assert/strict' ; import { Readable } from 'node:stream' ; import { createInterface } from 'node:readline' ; const input = [ '{"type":"message","text":"first"}' , '{broken' , '{"type":"progress","text":"noise"}' , '{"type":"message","text":"second"}' , ]. join ( ' \r\n ' ); const stream = Readable. from ([input. slice ( 0 , 18 ), input. slice ( 18 )]); const lines = createInterface ({ input: stream, crlfDelay: Infinity }); const messages = []; let malformed = 0 ; for await ( const line of lines) { if ( ! line. trim ()) continue ; let event; try { event = JSON . parse (line); } catch { malformed ++ ; continue ; } if (event?.type === 'message' && typeof event.text === 'string' ) { messages. push (event.text); } } assert. deepEqual (messages, [ 'first' , 'second' ]); assert. equal (malformed, 1 ); console. log ( 'line framing, selection, and malformed count pass' ); ``` Change the expected malformed count to 0 and the program should fail; restore 1 and it passes. The teaching code adds this diagnostic count. It does not establish that existing parsers log every damaged record. These assertions cover the constructed input, without proving recovery from every invalid JSON record or oversized line. ## Event sequences need format-specific normalization The [Codex parser](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/adapters/codex.ts) handles session_meta, event_msg, and response_item separately. An answer can appear in more than one event: the current code skips response_item assistant text when it exactly matches the last assistant text. That local rule does not provide global event deduplication. Tool calls also mark the previous assistant’s hasToolUse, so retaining only events with text fields cannot establish complete tool semantics. The [Copilot parser](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/adapters/copilot.ts) reads the first snapshot line on its JSONL path, accepts the current snapshot wrapper or a legacy object, and extracts user messages and responses from requests. It does not replay incremental patches from later lines. An empty, damaged, or unexpected first line is not necessarily repaired by subsequent lines. Identify the exact read path before diagnosing missing messages. Public regression fixtures should separately include duplicate answer events, tool-only records, damaged lines, unknown events, and missing times. Expected results should specify ordering, roles, text, and tool flags, rather than only asserting that parsing does not throw. ## Database byte copies have a distinct read time The current [vscdb.ts](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/vscdb.ts) reads the main file with readFileSync and opens an in-memory sql.js database. On failure it waits 500ms and retries once, returning null after a second failure. It does not read WAL or coordinate a consistent backup with the source program. One retry does not establish recovery from all lock contention or damaged reads. For a source using WAL mode, committed content can still reside in WAL. Copying only the main file does not establish a complete, current snapshot; see the [SQLite WAL documentation](https://www.sqlite.org/wal.html). This explains the read strategy’s boundary, without claiming that the exercise reproduced a user’s data loss. Install dependencies in a new temporary directory, save the following `snapshot.mjs`, and run with Node 24+: ```sh npm init -y npm install --ignore-scripts --no-audit --no-fund sql.js@1.14.2 ``` ```js import assert from 'node:assert/strict' ; import initSqlJs from 'sql.js' ; const SQL = await initSqlJs (); const live = new SQL . Database (); let copy; try { live. run ( 'CREATE TABLE kv (key TEXT PRIMARY KEY, value TEXT)' ); live. run ( "INSERT INTO kv VALUES ('message', 'first')" ); const bytes = live. export (); copy = new SQL . Database (bytes); live. run ( "UPDATE kv SET value = 'second' WHERE key = 'message'" ); const read = ( db ) => db. exec ( 'SELECT value FROM kv' )[ 0 ].values[ 0 ][ 0 ]; assert. equal ( read (copy), 'first' ); assert. equal ( read (live), 'second' ); console. log ( 'exported bytes are a separate snapshot' ); } finally { copy?. close (); live. close (); } ``` Change read(copy)’s expectation to second and it should fail; restore first and it passes. This in-memory experiment establishes only that exported bytes do not follow subsequent changes in the original instance. **It does not test file locks, WAL, concurrent writes, or openVscdb recovery.** ## Locate message content through SQLite keys Once database bytes are available, the parser must select the right table, key, and conversation. These paths describe the pinned implementation, rather than a universal specification for every vscdb file. | Source | Conversation lookup | Message extraction | | --- | --- | --- | | Cursor | Read allComposers from workspace ItemTable’s composer.composerData; scanning also discovers orphan conversations without workspace entries | Match bubbleId:conversationID: prefixes in global cursorDiskKV and parse each value | | Trae | Parse ItemTable’s memento/icube-ai-agent-storage value and find sessionId in list | Read that conversation’s messages, order by turnIndex, and extract user or assistant content | Cursor treats bubble.type 1 as user and other values as assistant, skips empty assistant text, and finally sorts by message time. A newer bubble._v produces a warning; that does not establish full compatibility with its structure. Trae’s SQL value can be a string or a Uint8Array requiring UTF-8 decoding. Direct assistant content takes precedence; otherwise extraction reads reply material from agentTaskContent’s proposal, finish-step thoughts in guideline.planItems, and related locations, while collecting thinking separately. Ordering timestamps can be synthesized as increasing values from the conversation’s start time when necessary. They are not all exact original message times. These are current behaviors in the source linked below. The snapshot exercise did not run either adapter. Format integration checks should construct the relevant tables, keys, TEXT/BLOB values, missing conversations, and empty messages, then assert actual adapter output. ## Check metadata separately from message time The [Cursor](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/adapters/cursor.ts) scan uses conversation creation time when available, falling back to database modification time. The [Trae](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/adapters/trae.ts) scan prefers conversation update time. Neither supports a blanket statement that file modification time is every message’s timestamp. Multiple conversations in one database can also share its file size. Normalized output gives downstream consumers a common shape. It does not guarantee collision-free identities, observable omissions, or complete compatibility with new formats. Keep fixtures and failure evidence for scanning, parsing, and persistence separately. Measure time and memory using the [bulk import measurement method](/en/blog/conversation-import-memory-budget/); the offline assertions here produce no performance conclusion. Share: ## Related Posts [## AI Conversation Formats: JSONL Events vs SQLite Snapshots Compare 5 AI adapter formats: separate JSONL events from SQLite keys, and inspect indexed, orphan, and unknown-version chat transcripts in memory stores. 曾亮懿 October 7, 2026 4 min read #ChatCrystal #Node.js #SQLite](/en/blog/ai-conversation-format-divergence) [## How Claude Code, Cursor, Copilot, and Trae Store Chats on Disk Compare Claude Code, Cursor, Copilot, and Trae storage paths. Distinguish raw copies from exports, and verify migration retention with test manifests. 曾亮懿 October 7, 2026 4 min read #ChatCrystal #Node.js #SQLite](/en/blog/ai-conversation-management-options) [## sql.js WASM Persistence: Memory, Exports & Nested Savepoints Trace ChatCrystal’s database lifecycle with sql.js 1.14.2 fixtures, separating SQL commits, exports, file writes, foreign-key restoration, and nested rollback. 曾亮懿 October 7, 2026 5 min read #ChatCrystal #Node.js #SQLite](/en/blog/sqljs-wasm-persistence) ## From writing to code Explore ChatCrystal, developer tools and daily learning notes on my GitHub. Questions about the implementation are welcome. [Visit my GitHub](https://github.com/ZengLiangYi) ```json { "@context": "https://schema.org", "@type": "WebSite", "name": "ZengLiangYi Blog", "alternateName": [ "Zeng Liangyi", "ZengLiangYi", "zengliangyi.com" ], "url": "https://zengliangyi.com", "description": "曾亮懿的个人网站。记录 AI Native 探索、开发者工具与全栈产品实践。" } { "@context": "https://schema.org", "@type": "BlogPosting", "headline": "Conversation Parsing: JSONL, Events, and SQLite Read Boundaries", "description": "Trace ChatCrystal parsers with offline fixtures to test line framing and SQLite snapshots, separating decoding from message normalization guarantees.", "url": "https://zengliangyi.com/en/blog/multi-format-conversation-parsing/", "image": "https://zengliangyi.com/images/blog/multi-format-conversation-parsing.svg", "datePublished": "2026-10-06T00:00:00.000Z", "dateModified": "2026-10-23T00:00:00.000Z", "author": { "@type": "Person", "name": "曾亮懿", "url": "https://zengliangyi.com/en/about" }, "publisher": { "@type": "Organization", "name": "ZengLiangYi Blog", "logo": { "@type": "ImageObject", "url": "https://zengliangyi.com/favicon-96x96.png" } }, "mainEntityOfPage": { "@type": "WebPage", "@id": "https://zengliangyi.com/en/blog/multi-format-conversation-parsing/" } } { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://zengliangyi.com/en" }, { "@type": "ListItem", "position": 2, "name": "Blog", "item": "https://zengliangyi.com/en/blog" }, { "@type": "ListItem", "position": 3, "name": "Conversation Parsing: JSONL, Events, and SQLite Read Boundaries", "item": "https://zengliangyi.com/en/blog/multi-format-conversation-parsing/" } ] } ```