---
title: Understanding Five AI Conversation Data Formats
description: How ChatCrystal parses Claude Code, Codex, Cursor, Trae, and Copilot records, and where JSONL messages, event streams, SQLite KV stores, and snapshots differ.
image: "https://zengliangyi.com/og/blog/en/ai-conversation-data-formats.png"
---

Discussing architecture in Claude Code, debugging in Cursor, and continuing an implementation in Copilot leaves related records in different stores. Before searching across them, you need to decide what constitutes a message, what constitutes a conversation, and which content to retain.

[ChatCrystal](/en/projects/chatcrystal) reads these records through separate adapters, then passes them to shared import, note, and retrieval workflows. This article was revised against the implementation on September 23, 2026, at commit [`7642768`](https://github.com/ZengLiangYi/ChatCrystal/tree/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser). It describes formats the project handles, not stable public contracts offered by the tools or a specification for lossless backups.

## Claude Code: a line is not necessarily a conversation message

The default scan location is `~/.claude/projects/`. Each JSONL line can be parsed independently, but not every line belongs in a readable conversation. Files also contain system records, progress, file-history snapshots, and tool-related content.

This minimal example illustrates field relationships rather than a complete log:



```jsonl
{  "type"  :  "user"  ,  "uuid"  :  "u1"  ,  "message"  :{  "role"  :  "user"  ,  "content"  :  "Explain this error"  }}  
  {  "type"  :  "assistant"  ,  "uuid"  :  "a1"  ,  "parentUuid"  :  "u1"  ,  "message"  :{  "role"  :  "assistant"  ,  "content"  :[{  "type"  :  "text"  ,  "text"  :  "Start by checking the input type."  }]}}
```



The current adapter requires a `uuid`, selects `user` and `assistant` records, and preserves `parentUuid`. This is a filtering policy for knowledge extraction. Skipped records are not necessarily worthless; they are outside the conversation body this importer aims to retain.

Two type layers matter: the outer record’s `type` and the types of blocks inside `message.content`. Content may be a string or an array mixing text and tool blocks. Calling `String(content)` is insufficient, and tool results should not automatically become human messages.

Cleaning injected text also needs a defined scope. Removing specific system tags does not justify deleting everything inside angle brackets: the user might be discussing HTML. Paths, record fields, and parent relationships can all supply metadata; project and timing information should not be assumed to come exclusively from filenames.

## Codex: choose message sources within an event stream

The default scan location is `~/.codex/sessions/`. These files are also JSONL, but records represent events. Parsing requires checking both the outer event type and `payload.type`.

| Record | Use in the current adapter |

| --- | --- |

| `session_meta` | Extract session metadata |

| `event_msg` + `user_message` | Extract the actual user prompt |

| `event_msg` + `agent_message` | Extract assistant text |

| `response_item` + `message` | Handle full message objects |

| `response_item` + `function_call` / `custom_tool_call` | Mark tool use on an assistant message |

Here, `function_call` is a payload type, not a field at the same level as `session_meta`. Duplication is another concern: a prompt can appear in more than one event representation. The adapter prefers `event_msg/user_message`, skips user messages inside `response_item`, and handles repeated assistant text.

Deduplication depends on event order: the `response_item/message` branch compares text with the most recent assistant message, while the `event_msg/agent_message` branch appends directly. If identical text arrives first as a `response_item` and then as an `event_msg`, it can still appear twice. This is not global deduplication by event ID.

The implementation sorts messages by timestamp at the end, but sorting alone does not reconstruct every causal relationship. Missing or equal timestamps and format changes need dedicated fixtures. A readable imported conversation is also not a complete replay of tool execution.

## Cursor: SQLite is the container; KV is the model

Cursor uses a VS Code-style data directory. A common Linux root is `~/.config/Cursor/User/`; other operating systems use different locations. Two paths are particularly relevant:



```text
workspaceStorage/<workspace>/state.vscdb  
    └─ Workspace and Composer metadata  
    
  globalStorage/state.vscdb  
    └─ Global conversation and bubble data
```



Common entry points include `composer.composerData` in `ItemTable` and keys shaped like `bubbleId:<composerId>:<bubbleId>` in `cursorDiskKV`. SQLite supplies query operations, but the application model remains encoded in JSON values and key conventions. The databases and tables are not interchangeable.

Important boundaries include:

- Metadata and message bodies are separate, so their IDs must be joined correctly.

- Empty assistant messages may be intermediate states and should not produce empty notes.

- Conversations in the global database may no longer be discoverable through workspace metadata.

- Value types, nested fields, and version markers can change; anomalies should be diagnosed rather than silently discarded.

ChatCrystal uses sql.js to read an in-memory copy of the database file instead of connecting directly to Cursor’s live database. This avoids writing to the source, but it does not guarantee a current, consistent snapshot. Concurrent writes and the database’s WAL still matter. Check the source-file state during troubleshooting; do not modify the original database to accommodate the parser.

An in-memory copy also has a cost. Reading the file and constructing the database consume memory. Process many workspaces individually and close instances promptly instead of keeping every database resident.

## Trae: the same container with a different message model

The current adapter reads `ItemTable` in the workspace’s `state.vscdb` and looks up `memento/icube-ai-agent-storage`. Sharing a file extension with Cursor does not make Cursor’s bubble queries reusable.

The current TypeScript model contains sessions with `sessionId` and `messages`. An assistant message can have direct `content`, or a proposal and plan steps under `agentTaskContent`. Describing the entire store as one uniform array of task messages misses this structure.

Extraction prefers direct text, then reads nested content through known fields. Plan steps, tool names, and reasoning-related fields have different meanings. Serializing the whole object as JSON is not a substitute for extracting the message body. The adapter should distinguish a valid structure with no readable text from a parsing failure; otherwise the summarizer cannot tell why its input is empty.

## GitHub Copilot: snapshots and patches are different records

The current scanner covers workspace `chatSessions` and global `emptyWindowChatSessions`, supporting both `.json` and `.jsonl` files.

In the JSONL format handled by this adapter, the first line is a `kind: 0` snapshot, while subsequent lines can be `kind: 1` state patches. Treating every line as a full conversation and retaining the final line is incorrect.



```json
{  
    "kind"  :   0  ,  
    "v"  : {  
      "sessionId"  :   "example-session"  ,  
      "requests"  : [  
        {  "message"  : {  "text"  :   "Explain this function"  },   "response"  : [{  "kind"  :   "text"  ,   "value"  :   "It reads the configuration."  }]}  
      ]  
    }  
  }
```



This is a simplified illustration. `response` belongs to each request rather than a separate top-level array. Response entries may also contain tool calls and other kinds of content that need distinct handling.

The current implementation reads the first JSONL snapshot; it does not replay every subsequent patch. This is a material limitation: changes stored only in patches may not appear in the imported conversation. Complete reconstruction requires patch handling and regression fixtures before claiming that the entire session can be recovered.

## Unify the interface without erasing the differences

| Source | Main container | Core parsing responsibility |

| --- | --- | --- |

| Claude Code | JSONL message records | Filter records, extract blocks, preserve message relationships |

| Codex | JSONL event stream | Choose event sources, handle duplication, assemble messages |

| Cursor | SQLite KV | Join workspace metadata with global messages |

| Trae | SQLite KV | Read sessions and nested task content |

| Copilot | JSON / JSONL snapshots and patches | Recognize snapshots and state the patch-replay limitation |

ChatCrystal’s `SourceAdapter` separates `detect()`, `scan()`, and `parse()`: detect a source, scan metadata, and parse a normalized result. The import workflow does not need to know about bubbles or task fields.

Normalized output should retain provenance, original IDs, timestamps, and traceability. Parsing failures should remain distinguishable from genuinely empty content. File size and modification time help incremental importing, but a changed file does not mean all of its content is new.

## Which cases should parser tests cover first?

A minimum fixture set should extend beyond a successful exchange: malformed JSON lines, empty files, tool-result-only messages, duplicate events, missing timestamps, orphaned Composers, unknown fields, Windows and Unix paths, and Copilot patches.

For each fixture, specify the expected message count, retained content, skipped content, and diagnostics. Use sanitized or synthetic records so real project material does not enter the test repository.

These boundaries determine the quality of the knowledge base’s input. Better retrieval cannot restore essential information silently lost during import.

Continue with [local vector database selection](/en/blog/local-vector-database-selection) for the retrieval layer, or the [Obsidian workflow](/en/blog/chatcrystal-to-obsidian) for managing exported notes.

Share:

##  Related Posts

[##  Five AI conversation parsers: retained fields and loss boundaries

Build an acceptance matrix from Claude Code, Codex, Cursor, Trae, and Copilot parsers, checking tool flags, thinking, order, and snapshot coverage.          曾亮懿       October 7, 2026      4 min read

#ChatCrystal

#TypeScript

#Node.js](/en/blog/ai-conversation-parser-boundaries)    [##  AI SDK Provider Abstraction: Capabilities and Protocols

Test 7 ChatCrystal providers and AI SDK factories with offline mocks for Responses, Chat Completions, embeddings, and structured output rejection.          曾亮懿       October 7, 2026      4 min read

#ChatCrystal

#Node.js

#TypeScript](/en/blog/ai-sdk-provider-abstraction)    [##  SourceAdapter internals: identity, content hashes, and reimport

Trace ChatCrystal’s registry, importer, and remote payload to verify identity, canonical hashes, metadata shortcuts, and parser-version boundaries.          曾亮懿       October 7, 2026      4 min read

#ChatCrystal

#TypeScript

#Node.js](/en/blog/source-adapter-internals)

##  From writing to code

Explore ChatCrystal, developer tools and daily learning notes on my GitHub. Questions about the implementation are welcome.

[Visit my GitHub](https://github.com/ZengLiangYi)

```json
{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "name": "ZengLiangYi Blog",
  "alternateName": [
    "Zeng Liangyi",
    "ZengLiangYi",
    "zengliangyi.com"
  ],
  "url": "https://zengliangyi.com",
  "description": "曾亮懿的个人网站。记录 AI Native 探索、开发者工具与全栈产品实践。"
}
{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "Understanding Five AI Conversation Data Formats",
  "description": "How ChatCrystal parses Claude Code, Codex, Cursor, Trae, and Copilot records, and where JSONL messages, event streams, SQLite KV stores, and snapshots differ.",
  "url": "https://zengliangyi.com/en/blog/ai-conversation-data-formats/",
  "image": "https://zengliangyi.com/images/blog/ai-conversation-data-formats.svg",
  "datePublished": "2026-06-09T00:45:00.000Z",
  "dateModified": "2026-09-24T00:00:00.000Z",
  "author": {
    "@type": "Person",
    "name": "曾亮懿",
    "url": "https://zengliangyi.com/en/about"
  },
  "publisher": {
    "@type": "Organization",
    "name": "ZengLiangYi Blog",
    "logo": {
      "@type": "ImageObject",
      "url": "https://zengliangyi.com/favicon-96x96.png"
    }
  },
  "mainEntityOfPage": {
    "@type": "WebPage",
    "@id": "https://zengliangyi.com/en/blog/ai-conversation-data-formats/"
  }
}
{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    {
      "@type": "ListItem",
      "position": 1,
      "name": "Home",
      "item": "https://zengliangyi.com/en"
    },
    {
      "@type": "ListItem",
      "position": 2,
      "name": "Blog",
      "item": "https://zengliangyi.com/en/blog"
    },
    {
      "@type": "ListItem",
      "position": 3,
      "name": "Understanding Five AI Conversation Data Formats",
      "item": "https://zengliangyi.com/en/blog/ai-conversation-data-formats/"
    }
  ]
}
```