--- title: "Memory Budgets for Large Chat Imports: Measure Before Batching" description: Profile memory peaks in large chat imports with sql.js and task queues. Size batches from reproducible benchmarks instead of arbitrary fixed thresholds. image: "https://zengliangyi.com/og/blog/en/conversation-import-memory-budget.png" --- Large imports combine several memory peaks: sql.js keeps the ChatCrystal database in memory, a source adapter opens raw files, and saving the database creates an export copy. Do not predict every machine with “conversation count × fixed KB”; measure your sources, message lengths, and runtime mode. This article follows the [import loop](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/services/import.ts), [database implementation](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/db/index.ts), [remote import](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/services/remoteImport.ts), and [queue](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/queue/index.ts) at fixed commit [`7642768`](https://github.com/ZengLiangYi/ChatCrystal/tree/764276814e5c5a33ecbfeae63754f74b9261a3a8). ## Separate resident memory from peaks sql.js reads the database file and keeps it in a WASM database; a larger database raises resident memory. `saveDatabase()` exports the database before writing it, so an additional typed array and Buffer exist during the export. Automatic saving defaults to every 30 seconds, but the peak depends on the actual database size. `importAll()` first collects metadata from enabled adapters, then parses and writes one item at a time; this local path does not collect every complete conversation into one array. It explicitly saves at the end, while periodic autosaves can persist some data earlier. An interruption can still lose changes made since the last save. `importAll()` itself does not automatically summarize, and the current file watcher only triggers import; confirm separately whether summary jobs have been submitted. Summary and vector jobs add memory when they run. Queue in the cover represents this optional subsequent stage. ## Source formats have different peaks - Claude Code and Codex JSONL parsers read through streams and readline but accumulate the complete conversation’s parsed message array. Memory is not limited to one line, and an unusually long record creates an additional large temporary object. - Cursor and Trae open source SQLite databases with sql.js, so the source database and ChatCrystal’s database coexist during parsing. - Copilot’s JSONL snapshot path reads only the first snapshot line; its JSON path still reads and parses the full file. These are implementation paths, not a guarantee that any file size is safe. Measure each source family to find which files dominate your peak. ## Queue limits the processing peak, not completion time Queued summarization and related background work share a p-queue with concurrency 1 and at most one queued task starting per second. Errors containing `429` are retried with backoff, up to three additional attempts. Serial work reduces the objects these jobs hold simultaneously, but synchronous APIs and other processes are not serialized by this queue. It does not promise completion time or eliminate concurrent-write risks. Remote import first collects every parsed upload object into an array, then filters and splits it using limits of 25 items per batch, 8 MiB per item, and 20 MiB per serialized batch. Bounded transport batches do not bound collection-stage memory. Local `importAll()` has different metadata and save behavior as well. ## Measure the server process with an isolated sample RSS is the memory currently resident in physical RAM for a process. This procedure covers Linux, a local CLI server, and Claude Code JSONL. It does not combine Electron, an upload client, or an external model into the same figure. It requires an installed `crystal`, Node 24 or newer, Python 3, and curl. Confirm that port 13731 is free and replace `/replace/with/sample.jsonl` with a copy of a sample that parses correctly. Preserve conversation metadata; the source layout is `claude/sample/session.jsonl`. `DATA_DIR` isolates the application database only; configure all five source paths separately. The script points other sources at an empty directory and enables only Claude Code after startup. This fixed version adds missing enabledSources on restart, so reapply the source selection each run instead of relying on a previously saved single-source setting. See the [runtime paths](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/runtime/paths.ts) and [configuration loader](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/config.ts). ```bash set -e sample_root = $( mktemp -d ) mkdir -p " $sample_root /data" " $sample_root /claude/sample" " $sample_root /empty" cp /replace/with/sample.jsonl " $sample_root /claude/sample/session.jsonl" DATA_DIR = " $sample_root /data" CLAUDE_PROJECTS_DIR = " $sample_root /claude" \ CODEX_SESSIONS_DIR=" $sample_root /empty" CURSOR_DATA_DIR=" $sample_root /empty" \ TRAE_DATA_DIR=" $sample_root /empty" COPILOT_DATA_DIR=" $sample_root /empty" \ CHATCRYSTAL_CLOUD_MODE= false CHATCRYSTAL_API_TOKEN='' \ crystal serve --port 13731 > " $sample_root /server.log" 2>&1 & server_pid = $! trap 'kill -TERM "$server_pid" 2>/dev/null || true; wait "$server_pid" 2>/dev/null || true' EXIT ready = false for attempt in $( seq 1 20 ); do kill -0 " $server_pid " || exit 1 if curl --fail --silent http://127.0.0.1:13731/api/health > /dev/null ; then ready = true break fi sleep 0.5 done [ " $ready " = true ] || exit 1 curl --fail --silent --show-error --request POST \ --header 'Content-Type: application/json' \ --data '{"enabledSources":["claude-code"]}' \ http://127.0.0.1:13731/api/config > " $sample_root /config.json" python3 - " $sample_root /config.json" << 'PYCONFIG' import json, sys assert json.load(open(sys.argv[1]))['success'] is True PYCONFIG ps -p " $server_pid " -o pid=,args= python3 - " $server_pid " " $sample_root /rss.csv" " $sample_root /sampler.ready" << 'PYSAMPLE' & import csv, sys, time from pathlib import Path status = Path('/proc') / sys.argv[1] / 'status' with open(sys.argv[2], 'w', buffering=1) as out: writer = csv.writer(out) writer.writerow(['unix_seconds', 'rss_kib', 'process_peak_kib']) ready = Path(sys.argv[3]) while status.exists(): try: fields = dict(line.split(':', 1) for line in status.read_text().splitlines() if ':' in line) writer.writerow([time.time(), fields['VmRSS'].split()[0], fields['VmHWM'].split()[0]]) out.flush() if not ready.exists(): ready.write_text('sample recorded') except (FileNotFoundError, KeyError, ProcessLookupError): break time.sleep(0.05) PYSAMPLE sampler_pid = $! for attempt in $( seq 1 100 ); do kill -0 " $sampler_pid " || exit 1 [ ! -s " $sample_root /sampler.ready" ] || break sleep 0.05 done [ -s " $sample_root /sampler.ready" ] || exit 1 printf 'import_start %s\n' "$( date -Is )" > " $sample_root /phases.log" curl --fail --silent --show-error --request POST \ http://127.0.0.1:13731/api/import/scan > " $sample_root /import.json" python3 - " $sample_root /import.json" << 'PYRESULT' import json, sys result = json.load(open(sys.argv[1])) assert result['success'] is True print(result['data']) PYRESULT printf 'import_end %s\n' "$( date -Is )" >> " $sample_root /phases.log" kill -TERM " $server_pid " wait " $server_pid " wait " $sampler_pid " trap - EXIT python3 - " $sample_root /rss.csv" << 'PYVERIFY' import csv, sys rows = list(csv.DictReader(open(sys.argv[1]))) assert rows, 'No RSS samples recorded' print('samples:', len(rows), 'max_rss_kib:', max(int(row['rss_kib']) for row in rows)) PYVERIFY printf '%s\n' " $sample_root " ``` Confirm that `ps` shows the Node process for this run’s `crystal serve`. Keep the log, configuration, import result, Node/application versions, and sample size. A new database may contain bundled demo data; record that baseline. The external sampler reads RSS every 50 milliseconds in KiB. `VmHWM` is the process’s cumulative RSS high-water mark since startup. Use a new temporary directory and process for each small, medium, or large sample so that an earlier run does not become the next baseline. Start/end markers delimit the whole import; they do not identify scan, individual-file parsing, or synchronous-export peaks. Sampling can miss a brief peak. The high-water mark adds a process-level peak but cannot establish that it occurred during saving. The [Linux kernel documentation](https://docs.kernel.org/filesystems/proc.html) notes that asynchronous RSS accounting can be imprecise; a finer snapshot can use the slower `smaps`. To attribute a peak to database export, add timestamped diagnostic markers around `saveDatabase()` in an isolated code copy and correlate them with external sampling. An import-completion log is not an export-start marker. For remote import, sample the upload CLI and server PIDs separately. The local Electron server runs inside the main process and shares its PID; measure actual renderer, GPU, and other child processes separately. For a remote connection, measure the server independently, and record a remote-import worker separately as well. Measure a local model independently and avoid double-counting shared pages. This article provides neither a combined peak for these modes nor measured memory numbers. If an observed peak approaches available memory, reduce batches, disable unneeded sources, or schedule summaries separately, then retest with headroom for the OS, other processes, and sampling error. ## Batch and recover deliberately Enable one source at a time in settings, complete its scan, and confirm a successful database save before adding another. The local CLI currently does not forward `--source` to the import entry point, so that flag does not guarantee a single-source scan. For Cursor or Trae, organize measurements by workspace in isolated copies that retain required metadata; for long JSONL, identify unusually large files first. Avoid multiple processes sharing one data directory during import. Keep each scan’s JSON/SSE result and server logs to reconcile imported, replaced, skipped, and error counts. `crystal status` only gives current totals, and note listings cannot reconstruct those four counts for an import pass. After an interruption, use these records and conversation details to decide whether to rerun. Imported records with the same file size and modification time are skipped; changed content is hashed, and replacement can enqueue vector cleanup for old notes. An empty queue does not prove that all summaries and embeddings are complete. Restart normally when you need to release process-local objects: ```bash crystal serve stop crystal serve --daemon crystal status ``` After recovery, validate one known conversation and one long conversation by checking the source, note state, and vector state. A performance conclusion is meaningful only with its sample, configuration, and measurement procedure. Share: ## Related Posts [## Bulk import with 1000 conversations: cost and performance checks Separate parsing, SQL writes, WASM export, and summary queues; use synthetic inputs to check retention and data equivalence. 曾亮懿 October 7, 2026 4 min read #ChatCrystal #SQLite #Performance](/en/blog/bulk-import-performance-plan) [## Fixing 1px Carousel Seams: Subpixel Layout and Scale Fixes Debug 1px mobile carousel seams caused by subpixel rounding. Compare negative margins, CSS translation, and scale transforms to eliminate rendering gaps. 曾亮懿 October 13, 2026 5 min read #CSS #Web Development #Swiper #Performance](/en/blog/carousel-subpixel-1px-seam) [## AI Conversation Formats: JSONL Events vs SQLite Snapshots Compare 5 AI adapter formats: separate JSONL events from SQLite keys, and inspect indexed, orphan, and unknown-version chat transcripts in memory stores. 曾亮懿 October 7, 2026 4 min read #ChatCrystal #Node.js #SQLite](/en/blog/ai-conversation-format-divergence) ## From writing to code Explore ChatCrystal, developer tools and daily learning notes on my GitHub. Questions about the implementation are welcome. [Visit my GitHub](https://github.com/ZengLiangYi) ```json { "@context": "https://schema.org", "@type": "WebSite", "name": "ZengLiangYi Blog", "alternateName": [ "Zeng Liangyi", "ZengLiangYi", "zengliangyi.com" ], "url": "https://zengliangyi.com", "description": "曾亮懿的个人网站。记录 AI Native 探索、开发者工具与全栈产品实践。" } { "@context": "https://schema.org", "@type": "BlogPosting", "headline": "Memory Budgets for Large Chat Imports: Measure Before Batching", "description": "Profile memory peaks in large chat imports with sql.js and task queues. Size batches from reproducible benchmarks instead of arbitrary fixed thresholds.", "url": "https://zengliangyi.com/en/blog/conversation-import-memory-budget/", "image": "https://zengliangyi.com/images/blog/conversation-import-memory-budget.svg", "datePublished": "2026-10-03T00:00:00.000Z", "dateModified": "2026-10-23T00:00:00.000Z", "author": { "@type": "Person", "name": "曾亮懿", "url": "https://zengliangyi.com/en/about" }, "publisher": { "@type": "Organization", "name": "ZengLiangYi Blog", "logo": { "@type": "ImageObject", "url": "https://zengliangyi.com/favicon-96x96.png" } }, "mainEntityOfPage": { "@type": "WebPage", "@id": "https://zengliangyi.com/en/blog/conversation-import-memory-budget/" } } { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://zengliangyi.com/en" }, { "@type": "ListItem", "position": 2, "name": "Blog", "item": "https://zengliangyi.com/en/blog" }, { "@type": "ListItem", "position": 3, "name": "Memory Budgets for Large Chat Imports: Measure Before Batching", "item": "https://zengliangyi.com/en/blog/conversation-import-memory-budget/" } ] } ```