--- title: "Five AI conversation parsers: retained fields and loss boundaries" description: Build an acceptance matrix from Claude Code, Codex, Cursor, Trae, and Copilot parsers, checking tool flags, thinking, order, and snapshot coverage. image: "https://zengliangyi.com/og/blog/en/ai-conversation-parser-boundaries.png" --- Successful parsing does not establish complete preservation. Five sources can output ParsedConversation while selecting text, tool information, thinking, and order differently. [Multi-format parsing](/en/blog/multi-format-conversation-parsing/) covers inputs and database locations; this article focuses on output acceptance, without treating shared fields as a lossless format conversion. ## Start with an acceptance matrix for five sources These observations describe the pinned [built-in adapters](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/index.ts), not the storage specification of future tool versions. | Source | Input location and text selection | Boundary to observe during acceptance | | --- | --- | --- | | Claude Code | Project JSONL; user/assistant messages with uuid; strings or content blocks | System messages and known deltas are discarded; tool_result is excluded from text; thinking is separate and tools set a flag | | Codex | Rollout JSONL; selected event_msg and response_item branches | User/developer response_items are skipped; full text is compared only with the latest assistant text; calls mark the preceding assistant | | Cursor | Workspace ItemTable session index; global cursorDiskKV bubble prefixes and orphan sessions | Type=1 means user, others become assistant; empty assistants are skipped first; new _v only warns; timestamp sorting follows | | Trae | ItemTable memento/icube-ai-agent-storage; sessionId selects data.list entry | turnIndex sorting; direct text and nested agent content have precedence; timestamps can be synthetic | | Copilot | Entire legacy .json; literal first-line snapshot in .jsonl | Later patches are not replayed; response blocks map separately to text, thinking, and tool flags | Cursor and Trae do not share a SQL-value schema, JSON shape, or session lookup. Shared fields reduce downstream branching without replacing source-specific sample tests. ## Tool flags, thinking, and empty messages differ [Claude Code](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/adapters/claude-code.ts) tool blocks set hasToolUse without preserving the complete tool input/result trace. A message with only that flag can survive without text or thinking. Tags such as system-reminder are removed from text. This extraction policy does not prove the removed information has no value for every use. [Cursor](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/adapters/cursor.ts) skips empty assistant text before extracting thinking or tool flags, so a thinking-only assistant bubble can be lost too. HasCode can come from code fences or codeBlocks/suggestedCodeBlocks arrays, rather than text alone. [Trae](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/adapters/trae.ts) prefers direct content; otherwise it combines nested content such as proposal and finish with thinking. Final retention requires text or thinking; a tool flag alone is insufficient. [Copilot](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/adapters/copilot.ts) likewise adds an assistant message only when text or thinking has a value. Claude Code’s tool-only behavior does not apply to every source. ## Codex deduplication and ordering have defined scopes The [Codex parser](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/adapters/codex.ts) extracts user input from event_msg/user_message and assistant text from agent_message. Response_item/message combines only assistant output_text. If the full text equals lastAssistantMsg.content, that response_item is skipped. There is no deduplication set for every event or all historical text. Function_call/custom_tool_call sets hasToolUse on the latest assistant. Without one, it does not create a tool-call message. Unhandled branches such as reasoning do not populate thinking; this does not establish that all raw reasoning is absent or encrypted. Missing timestamps use the current time, and final sorting does not reconstruct every event’s causal relationship. Trae first sorts by turnIndex, then keeps output timestamps monotonic. Those times can be constructed ordering markers, unsuitable for measuring actual reading or model-response duration. Preserve raw event position or source samples when checking order; normalized timestamp alone is insufficient. ## Test snapshots and response blocks with a Copilot sample Save as `lesson.mjs` and run directly with Node 24+. This teaching extractor covers only first-line wrapping, selected response blocks, and assistant retention. **It is not the complete Copilot adapter:** there is no file scanning, user-message handling, timestamp processing, database, or runtime schema check. ```js import assert from 'node:assert/strict' ; function firstSnapshot ( lines ) { try { const snapshot = JSON . parse (lines. split ( ' \n ' )[ 0 ]); if (snapshot.kind !== undefined && snapshot.kind !== 0 ) return null ; return snapshot.v ?? snapshot; } catch { return null ; } } function assistant ( response ) { const texts = []; const thoughts = []; let hasToolUse = false ; for ( const item of response) { if (item.kind === 'thinking' && item.value) thoughts. push (item.value); else if ((item.kind === 'text' || item.kind === undefined ) && item.value) texts. push (item.value); else if (item.kind === 'toolInvocationSerialized' || item.kind === 'prepareToolInvocation' ) hasToolUse = true ; } const content = texts. join ( ' \n ' ). trim (); const thinking = thoughts. length ? thoughts. join ( ' \n ' ) : null ; return content || thinking ? { content, thinking, hasToolUse } : null ; } const snapshot = { kind: 0 , v: { requests: [ { response: [ { kind: 'text' , value: 'first answer' }, { kind: 'thinking' , value: 'fixture thought' }, { kind: 'prepareToolInvocation' }, { kind: 'inlineReference' , value: 'not response text' }, ], }, ], }, }; const patch = { kind: 1 , v: { requests: [{ response: [{ kind: 'text' , value: 'later answer' }] }] }, }; const lines = JSON . stringify (snapshot) + ' \n ' + JSON . stringify (patch); const selected = firstSnapshot (lines); assert. notEqual (selected, null ); assert. deepEqual ( assistant (selected.requests[ 0 ].response), { content: 'first answer' , thinking: 'fixture thought' , hasToolUse: true , }); assert. equal ( firstSnapshot ( 'broken \n ' + JSON . stringify (snapshot)), null ); assert. equal ( firstSnapshot ( JSON . stringify (patch)), null ); assert. equal ( assistant ([{ kind: 'prepareToolInvocation' }]), null ); assert. deepEqual ( assistant ([{ kind: 'thinking' , value: 'thought only' }]), { content: '' , thinking: 'thought only' , hasToolUse: false , }); console. log ( 'snapshot and selected response boundaries pass' ); ``` Set hasToolUse to false in its assignment and the mixed-response assertion fails. Select the last line rather than the first and the initial-snapshot assertion fails too. Restoring the code passes. This verifies the choices for these samples, without proving later patches never affect conversation content. ## Record missing content as well as message counts Prepare normal text, tool-only, thinking-only, mixed blocks, empty responses, repeated text, missing timestamps, malformed JSON, and unknown-version samples for each source. Record expected text, role, tool flags, thinking, order, identity, and reasons for omission. Claude/Codex can skip malformed JSON lines; a damaged first line in Copilot can instead produce an empty conversation. These are not one universal recovery policy. Parsed output also reaches the [import pipeline](/en/blog/source-adapter-internals/), including its fewer-than-two-message filter and content comparison. Distinguish a parser discarding a message, an importer skipping a conversation, and a database remaining unchanged. A structurally valid output JSON establishes its shape, without proving source compatibility or completeness. Share: ## Related Posts [## AI SDK Provider Abstraction: Capabilities and Protocols Test 7 ChatCrystal providers and AI SDK factories with offline mocks for Responses, Chat Completions, embeddings, and structured output rejection. 曾亮懿 October 7, 2026 4 min read #ChatCrystal #Node.js #TypeScript](/en/blog/ai-sdk-provider-abstraction) [## SourceAdapter internals: identity, content hashes, and reimport Trace ChatCrystal’s registry, importer, and remote payload to verify identity, canonical hashes, metadata shortcuts, and parser-version boundaries. 曾亮懿 October 7, 2026 4 min read #ChatCrystal #TypeScript #Node.js](/en/blog/source-adapter-internals) [## Code tooling: the actual roles of Biome, ESLint, and formatting Read ChatCrystal’s scripts and Flat Config, verify ESLint diagnostics offline, and distinguish syntax rules, type checking, formatting, and CI gates. 曾亮懿 October 6, 2026 4 min read #ChatCrystal #TypeScript #Node.js](/en/blog/lint-format-tooling) ## From writing to code Explore ChatCrystal, developer tools and daily learning notes on my GitHub. Questions about the implementation are welcome. [Visit my GitHub](https://github.com/ZengLiangYi) ```json { "@context": "https://schema.org", "@type": "WebSite", "name": "ZengLiangYi Blog", "alternateName": [ "Zeng Liangyi", "ZengLiangYi", "zengliangyi.com" ], "url": "https://zengliangyi.com", "description": "曾亮懿的个人网站。记录 AI Native 探索、开发者工具与全栈产品实践。" } { "@context": "https://schema.org", "@type": "BlogPosting", "headline": "Five AI conversation parsers: retained fields and loss boundaries", "description": "Build an acceptance matrix from Claude Code, Codex, Cursor, Trae, and Copilot parsers, checking tool flags, thinking, order, and snapshot coverage.", "url": "https://zengliangyi.com/en/blog/ai-conversation-parser-boundaries/", "image": "https://zengliangyi.com/images/blog/ai-conversation-parser-boundaries.svg", "datePublished": "2026-10-07T00:00:00.000Z", "dateModified": "2026-10-07T00:00:00.000Z", "author": { "@type": "Person", "name": "曾亮懿", "url": "https://zengliangyi.com/en/about" }, "publisher": { "@type": "Organization", "name": "ZengLiangYi Blog", "logo": { "@type": "ImageObject", "url": "https://zengliangyi.com/favicon-96x96.png" } }, "mainEntityOfPage": { "@type": "WebPage", "@id": "https://zengliangyi.com/en/blog/ai-conversation-parser-boundaries/" } } { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://zengliangyi.com/en" }, { "@type": "ListItem", "position": 2, "name": "Blog", "item": "https://zengliangyi.com/en/blog" }, { "@type": "ListItem", "position": 3, "name": "Five AI conversation parsers: retained fields and loss boundaries", "item": "https://zengliangyi.com/en/blog/ai-conversation-parser-boundaries/" } ] } ```