Understanding Five AI Conversation Data Formats

Discussing architecture in Claude Code, debugging in Cursor, and continuing an implementation in Copilot leaves related records in different stores. Before searching across them, you need to decide what constitutes a message, what constitutes a conversation, and which content to retain.

ChatCrystal reads these records through separate adapters, then passes them to shared import, note, and retrieval workflows. This article was revised against the implementation on September 23, 2026, at commit 7642768. It describes formats the project handles, not stable public contracts offered by the tools or a specification for lossless backups.

Claude Code: a line is not necessarily a conversation message

The default scan location is ~/.claude/projects/. Each JSONL line can be parsed independently, but not every line belongs in a readable conversation. Files also contain system records, progress, file-history snapshots, and tool-related content.

This minimal example illustrates field relationships rather than a complete log:

{"type":"user","uuid":"u1","message":{"role":"user","content":"Explain this error"}}
{"type":"assistant","uuid":"a1","parentUuid":"u1","message":{"role":"assistant","content":[{"type":"text","text":"Start by checking the input type."}]}}

The current adapter requires a uuid, selects user and assistant records, and preserves parentUuid. This is a filtering policy for knowledge extraction. Skipped records are not necessarily worthless; they are outside the conversation body this importer aims to retain.

Two type layers matter: the outer record’s type and the types of blocks inside message.content. Content may be a string or an array mixing text and tool blocks. Calling String(content) is insufficient, and tool results should not automatically become human messages.

Cleaning injected text also needs a defined scope. Removing specific system tags does not justify deleting everything inside angle brackets: the user might be discussing HTML. Paths, record fields, and parent relationships can all supply metadata; project and timing information should not be assumed to come exclusively from filenames.

Codex: choose message sources within an event stream

The default scan location is ~/.codex/sessions/. These files are also JSONL, but records represent events. Parsing requires checking both the outer event type and payload.type.

Record	Use in the current adapter
session_meta	Extract session metadata
event_msg + user_message	Extract the actual user prompt
event_msg + agent_message	Extract assistant text
response_item + message	Handle full message objects
response_item + function_call / custom_tool_call	Mark tool use on an assistant message

Here, function_call is a payload type, not a field at the same level as session_meta. Duplication is another concern: a prompt can appear in more than one event representation. The adapter prefers event_msg/user_message, skips user messages inside response_item, and handles repeated assistant text.

Deduplication depends on event order: the response_item/message branch compares text with the most recent assistant message, while the event_msg/agent_message branch appends directly. If identical text arrives first as a response_item and then as an event_msg, it can still appear twice. This is not global deduplication by event ID.

The implementation sorts messages by timestamp at the end, but sorting alone does not reconstruct every causal relationship. Missing or equal timestamps and format changes need dedicated fixtures. A readable imported conversation is also not a complete replay of tool execution.

Cursor: SQLite is the container; KV is the model

Cursor uses a VS Code-style data directory. A common Linux root is ~/.config/Cursor/User/; other operating systems use different locations. Two paths are particularly relevant:

workspaceStorage/<workspace>/state.vscdb
  └─ Workspace and Composer metadata

globalStorage/state.vscdb
  └─ Global conversation and bubble data

Common entry points include composer.composerData in ItemTable and keys shaped like bubbleId:<composerId>:<bubbleId> in cursorDiskKV. SQLite supplies query operations, but the application model remains encoded in JSON values and key conventions. The databases and tables are not interchangeable.

Important boundaries include:

• Metadata and message bodies are separate, so their IDs must be joined correctly.
• Empty assistant messages may be intermediate states and should not produce empty notes.
• Conversations in the global database may no longer be discoverable through workspace metadata.
• Value types, nested fields, and version markers can change; anomalies should be diagnosed rather than silently discarded.

ChatCrystal uses sql.js to read an in-memory copy of the database file instead of connecting directly to Cursor’s live database. This avoids writing to the source, but it does not guarantee a current, consistent snapshot. Concurrent writes and the database’s WAL still matter. Check the source-file state during troubleshooting; do not modify the original database to accommodate the parser.

An in-memory copy also has a cost. Reading the file and constructing the database consume memory. Process many workspaces individually and close instances promptly instead of keeping every database resident.

Trae: the same container with a different message model

The current adapter reads ItemTable in the workspace’s state.vscdb and looks up memento/icube-ai-agent-storage. Sharing a file extension with Cursor does not make Cursor’s bubble queries reusable.

The current TypeScript model contains sessions with sessionId and messages. An assistant message can have direct content, or a proposal and plan steps under agentTaskContent. Describing the entire store as one uniform array of task messages misses this structure.

Extraction prefers direct text, then reads nested content through known fields. Plan steps, tool names, and reasoning-related fields have different meanings. Serializing the whole object as JSON is not a substitute for extracting the message body. The adapter should distinguish a valid structure with no readable text from a parsing failure; otherwise the summarizer cannot tell why its input is empty.

GitHub Copilot: snapshots and patches are different records

The current scanner covers workspace chatSessions and global emptyWindowChatSessions, supporting both .json and .jsonl files.

In the JSONL format handled by this adapter, the first line is a kind: 0 snapshot, while subsequent lines can be kind: 1 state patches. Treating every line as a full conversation and retaining the final line is incorrect.

{
  "kind": 0,
  "v": {
    "sessionId": "example-session",
    "requests": [
      {"message": {"text": "Explain this function"}, "response": [{"kind": "text", "value": "It reads the configuration."}]}
    ]
  }
}

This is a simplified illustration. response belongs to each request rather than a separate top-level array. Response entries may also contain tool calls and other kinds of content that need distinct handling.

The current implementation reads the first JSONL snapshot; it does not replay every subsequent patch. This is a material limitation: changes stored only in patches may not appear in the imported conversation. Complete reconstruction requires patch handling and regression fixtures before claiming that the entire session can be recovered.

Unify the interface without erasing the differences

Source	Main container	Core parsing responsibility
Claude Code	JSONL message records	Filter records, extract blocks, preserve message relationships
Codex	JSONL event stream	Choose event sources, handle duplication, assemble messages
Cursor	SQLite KV	Join workspace metadata with global messages
Trae	SQLite KV	Read sessions and nested task content
Copilot	JSON / JSONL snapshots and patches	Recognize snapshots and state the patch-replay limitation

ChatCrystal’s SourceAdapter separates detect(), scan(), and parse(): detect a source, scan metadata, and parse a normalized result. The import workflow does not need to know about bubbles or task fields.

Normalized output should retain provenance, original IDs, timestamps, and traceability. Parsing failures should remain distinguishable from genuinely empty content. File size and modification time help incremental importing, but a changed file does not mean all of its content is new.

Which cases should parser tests cover first?

A minimum fixture set should extend beyond a successful exchange: malformed JSON lines, empty files, tool-result-only messages, duplicate events, missing timestamps, orphaned Composers, unknown fields, Windows and Unix paths, and Copilot patches.

For each fixture, specify the expected message count, retained content, skipped content, and diagnostics. Use sanitized or synthetic records so real project material does not enter the test repository.

These boundaries determine the quality of the knowledge base’s input. Better retrieval cannot restore essential information silently lost during import.

Continue with local vector database selection for the retrieval layer, or the Obsidian workflow for managing exported notes.