--- title: "SourceAdapter design: separate discovery, scanning, and parsing" description: Read ChatCrystal’s adapter and import contracts, verify selective parsing with fixtures, and check registration, enablement, identity, and version boundaries. image: "https://zengliangyi.com/og/blog/en/source-adapter-design.png" --- AI tools can store conversations as JSONL, event streams, or SQLite key-value data. A shared import pipeline needs clear responsibilities for discovering data, deciding whether it changed, and understanding its format. This article reads ChatCrystal’s SourceAdapter interface, then uses fixtures to verify the control flow that avoids parsing unchanged candidates. ## Share output contracts while retaining format differences [adapter.ts](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/adapter.ts) defines name, displayName, optional parserVersion, and three asynchronous methods. The core excerpt below depends on shared types and **cannot compile alone**. ```ts export interface SourceAdapter { readonly name : string ; readonly displayName : string ; readonly parserVersion ?: string ; detect () : Promise < SourceInfo | null >; scan () : Promise < ConversationMeta []>; parse ( meta : ConversationMeta ) : Promise < ParsedConversation >; } ``` `detect` returns source information or null; `scan` returns candidate metadata such as identity, path, size, and modification time; `parse` produces a normalized conversation and messages. Shared output lets the import service reuse persistence logic, but a TypeScript interface does not validate external JSON at runtime. Do not assume every detect implementation performs one directory-existence check. The [Claude Code adapter](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/adapters/claude-code.ts) enumerates project directories and JSONL files to count conversations; scan also skips files below 100 bytes. Read a method’s responsibility separately from the I/O its implementation performs. ## Registration does not establish enablement [registry.ts](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/registry.ts) stores adapters in a Map, warning and overwriting on duplicate names. The current [module entry](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/parser/index.ts) registers five built-in sources: Claude Code, Codex, Cursor, Trae, and Copilot. Registration happens statically at module import; it is not an interface for uploading arbitrary plugins at runtime. `detectAllSources` discovers all registered sources concurrently. [importAll](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/services/import.ts) first filters by enabledSources, then discovers and scans sequentially. These paths differ. Adding an adapter to the Map also requires checking enablement; otherwise the import path may never call it. ## Verify selective parsing with a fixture adapter Save this as `lesson.mjs` and run with Node 24+ without dependencies. It is a **teaching control flow**, modeling only enablement, discovery, metadata comparison, and parsing counts. It is not the complete importAll and has no database, content hash, or replacement policy. ```js import assert from 'node:assert/strict' ; const calls = []; const registry = new Map (); registry. set ( 'fixture' , { name: 'fixture' , async detect () { calls. push ( 'detect' ); return { name: 'fixture' }; }, async scan () { calls. push ( 'scan' ); return [ { id: 'unchanged' , fileSize: 120 , fileMtime: 't1' }, { id: 'changed' , fileSize: 240 , fileMtime: 't2' }, ]; }, async parse ( meta ) { calls. push ( 'parse:' + meta.id); return { id: meta.id }; }, }); const previous = new Map ([[ 'unchanged' , { fileSize: 120 , fileMtime: 't1' }]]); async function collect ( enabled ) { const results = []; for ( const adapter of registry. values ()) { if ( ! enabled. includes (adapter.name)) continue ; if ( ! ( await adapter. detect ())) continue ; for ( const meta of await adapter. scan ()) { const old = previous. get (meta.id); if (old?.fileSize === meta.fileSize && old?.fileMtime === meta.fileMtime) continue ; results. push ( await adapter. parse (meta)); } } return results; } assert. deepEqual ( await collect ([]), []); assert. deepEqual (calls, []); assert. deepEqual ( await collect ([ 'fixture' ]), [{ id: 'changed' }]); assert. deepEqual (calls, [ 'detect' , 'scan' , 'parse:changed' ]); console. log ( 'enabled source and parse boundaries pass' ); ``` Change 120 to 121 in `previous` and the previously unchanged candidate is parsed, failing the result assertion. Restore it and the program passes. This verifies the metadata shortcut, without establishing that identical size and timestamp always mean identical content. ## The actual importer compares content too The real importAll first skips unchanged size/time pairs, then computes a normalized content hash after parsing. An existing conversation whose file metadata changed but content hash stayed equal receives a metadata update. Changed content follows the replacement policy. Conversations with fewer than two parsed messages are skipped. Failures have different scopes too. Discovery and scanning sit outside the per-candidate try/catch: a thrown detect or scan error terminates that importAll call. Candidate parsing sits inside the guard, so a thrown parse error increments the error count and allows later candidates to continue. Per-candidate recovery does not establish source-wide fault isolation. The fast path still has a limit: unchanged size and timestamp prevent reparsing, so this is not a content check on every import. parserVersion records the parsing contract; its optional interface field does not establish that changing the version forces reimport automatically. See [bulk import measurement](/en/blog/conversation-import-memory-budget/) for persistence and replacement details. ## Check the whole path when adding a source A reliable integration needs reproducible format samples, identity rules, ordering and noise handling, and tests of missing and damaged data. Implementing and registering three methods is only part of the work. Also check the [configuration](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/config.ts) defaults and migration, path settings, interface entry points, remote transfer, and watchers for recognition of the new name. The current configuration restores built-in sources when loading persisted settings. That does not support a promise that disablement survives every restart scenario. Copying another tool’s default directory does not establish compatibility either. Check detect, scan, and parse separately with synthetic or authorized samples, then check persistence and failures through the shared importer. The interface reduces the influence of format differences on the main pipeline. It does not automatically eliminate identity collisions, omitted configuration, or parsing errors. Integration evidence should come from samples and counterexamples through the complete import path rather than interface type checking alone. Share: ## Related Posts [## Five AI conversation parsers: retained fields and loss boundaries Build an acceptance matrix from Claude Code, Codex, Cursor, Trae, and Copilot parsers, checking tool flags, thinking, order, and snapshot coverage. 曾亮懿 October 7, 2026 4 min read #ChatCrystal #TypeScript #Node.js](/en/blog/ai-conversation-parser-boundaries) [## AI SDK Provider Abstraction: Capabilities and Protocols Test 7 ChatCrystal providers and AI SDK factories with offline mocks for Responses, Chat Completions, embeddings, and structured output rejection. 曾亮懿 October 7, 2026 4 min read #ChatCrystal #Node.js #TypeScript](/en/blog/ai-sdk-provider-abstraction) [## SourceAdapter internals: identity, content hashes, and reimport Trace ChatCrystal’s registry, importer, and remote payload to verify identity, canonical hashes, metadata shortcuts, and parser-version boundaries. 曾亮懿 October 7, 2026 4 min read #ChatCrystal #TypeScript #Node.js](/en/blog/source-adapter-internals) ## From writing to code Explore ChatCrystal, developer tools and daily learning notes on my GitHub. Questions about the implementation are welcome. [Visit my GitHub](https://github.com/ZengLiangYi) ```json { "@context": "https://schema.org", "@type": "WebSite", "name": "ZengLiangYi Blog", "alternateName": [ "Zeng Liangyi", "ZengLiangYi", "zengliangyi.com" ], "url": "https://zengliangyi.com", "description": "曾亮懿的个人网站。记录 AI Native 探索、开发者工具与全栈产品实践。" } { "@context": "https://schema.org", "@type": "BlogPosting", "headline": "SourceAdapter design: separate discovery, scanning, and parsing", "description": "Read ChatCrystal’s adapter and import contracts, verify selective parsing with fixtures, and check registration, enablement, identity, and version boundaries.", "url": "https://zengliangyi.com/en/blog/source-adapter-design/", "image": "https://zengliangyi.com/images/blog/source-adapter-design.svg", "datePublished": "2026-10-06T00:00:00.000Z", "dateModified": "2026-10-06T00:00:00.000Z", "author": { "@type": "Person", "name": "曾亮懿", "url": "https://zengliangyi.com/en/about" }, "publisher": { "@type": "Organization", "name": "ZengLiangYi Blog", "logo": { "@type": "ImageObject", "url": "https://zengliangyi.com/favicon-96x96.png" } }, "mainEntityOfPage": { "@type": "WebPage", "@id": "https://zengliangyi.com/en/blog/source-adapter-design/" } } { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://zengliangyi.com/en" }, { "@type": "ListItem", "position": 2, "name": "Blog", "item": "https://zengliangyi.com/en/blog" }, { "@type": "ListItem", "position": 3, "name": "SourceAdapter design: separate discovery, scanning, and parsing", "item": "https://zengliangyi.com/en/blog/source-adapter-design/" } ] } ```