Memory Budgets for Large Chat Imports: Measure Before Batching

Large imports combine several memory peaks: sql.js keeps the ChatCrystal database in memory, a source adapter opens raw files, and saving the database creates an export copy. Do not predict every machine with “conversation count × fixed KB”; measure your sources, message lengths, and runtime mode.

This article follows the import loop, database implementation, remote import, and queue at fixed commit 7642768.

Separate resident memory from peaks

sql.js reads the database file and keeps it in a WASM database; a larger database raises resident memory. saveDatabase() exports the database before writing it, so an additional typed array and Buffer exist during the export. Automatic saving defaults to every 30 seconds, but the peak depends on the actual database size.

importAll() first collects metadata from enabled adapters, then parses and writes one item at a time; this local path does not collect every complete conversation into one array. It explicitly saves at the end, while periodic autosaves can persist some data earlier. An interruption can still lose changes made since the last save. importAll() itself does not automatically summarize, and the current file watcher only triggers import; confirm separately whether summary jobs have been submitted. Summary and vector jobs add memory when they run. Queue in the cover represents this optional subsequent stage.

Source formats have different peaks

• Claude Code and Codex JSONL parsers read through streams and readline but accumulate the complete conversation’s parsed message array. Memory is not limited to one line, and an unusually long record creates an additional large temporary object.
• Cursor and Trae open source SQLite databases with sql.js, so the source database and ChatCrystal’s database coexist during parsing.
• Copilot’s JSONL snapshot path reads only the first snapshot line; its JSON path still reads and parses the full file.

These are implementation paths, not a guarantee that any file size is safe. Measure each source family to find which files dominate your peak.

Queue limits the processing peak, not completion time

Queued summarization and related background work share a p-queue with concurrency 1 and at most one queued task starting per second. Errors containing 429 are retried with backoff, up to three additional attempts. Serial work reduces the objects these jobs hold simultaneously, but synchronous APIs and other processes are not serialized by this queue. It does not promise completion time or eliminate concurrent-write risks.

Remote import first collects every parsed upload object into an array, then filters and splits it using limits of 25 items per batch, 8 MiB per item, and 20 MiB per serialized batch. Bounded transport batches do not bound collection-stage memory. Local importAll() has different metadata and save behavior as well.

Measure the server process with an isolated sample

RSS is the memory currently resident in physical RAM for a process. This procedure covers Linux, a local CLI server, and Claude Code JSONL. It does not combine Electron, an upload client, or an external model into the same figure. It requires an installed crystal, Node 24 or newer, Python 3, and curl. Confirm that port 13731 is free and replace /replace/with/sample.jsonl with a copy of a sample that parses correctly. Preserve conversation metadata; the source layout is claude/sample/session.jsonl.

DATA_DIR isolates the application database only; configure all five source paths separately. The script points other sources at an empty directory and enables only Claude Code after startup. This fixed version adds missing enabledSources on restart, so reapply the source selection each run instead of relying on a previously saved single-source setting. See the runtime paths and configuration loader.

set -e
sample_root=$(mktemp -d)
mkdir -p "$sample_root/data" "$sample_root/claude/sample" "$sample_root/empty"
cp /replace/with/sample.jsonl "$sample_root/claude/sample/session.jsonl"
DATA_DIR="$sample_root/data" CLAUDE_PROJECTS_DIR="$sample_root/claude" \
  CODEX_SESSIONS_DIR="$sample_root/empty" CURSOR_DATA_DIR="$sample_root/empty" \
  TRAE_DATA_DIR="$sample_root/empty" COPILOT_DATA_DIR="$sample_root/empty" \
  CHATCRYSTAL_CLOUD_MODE=false CHATCRYSTAL_API_TOKEN='' \
  crystal serve --port 13731 >"$sample_root/server.log" 2>&1 &
server_pid=$!
trap 'kill -TERM "$server_pid" 2>/dev/null || true; wait "$server_pid" 2>/dev/null || true' EXIT
ready=false
for attempt in $(seq 1 20); do
  kill -0 "$server_pid" || exit 1
  if curl --fail --silent http://127.0.0.1:13731/api/health >/dev/null; then
    ready=true
    break
  fi
  sleep 0.5
done
[ "$ready" = true ] || exit 1
curl --fail --silent --show-error --request POST \
  --header 'Content-Type: application/json' \
  --data '{"enabledSources":["claude-code"]}' \
  http://127.0.0.1:13731/api/config >"$sample_root/config.json"
python3 - "$sample_root/config.json" <<'PYCONFIG'
import json, sys
assert json.load(open(sys.argv[1]))['success'] is True
PYCONFIG
ps -p "$server_pid" -o pid=,args=
python3 - "$server_pid" "$sample_root/rss.csv" "$sample_root/sampler.ready" <<'PYSAMPLE' &
import csv, sys, time
from pathlib import Path
status = Path('/proc') / sys.argv[1] / 'status'
with open(sys.argv[2], 'w', buffering=1) as out:
    writer = csv.writer(out)
    writer.writerow(['unix_seconds', 'rss_kib', 'process_peak_kib'])
    ready = Path(sys.argv[3])
    while status.exists():
        try:
            fields = dict(line.split(':', 1) for line in status.read_text().splitlines() if ':' in line)
            writer.writerow([time.time(), fields['VmRSS'].split()[0], fields['VmHWM'].split()[0]])
            out.flush()
            if not ready.exists():
                ready.write_text('sample recorded')
        except (FileNotFoundError, KeyError, ProcessLookupError):
            break
        time.sleep(0.05)
PYSAMPLE
sampler_pid=$!
for attempt in $(seq 1 100); do
  kill -0 "$sampler_pid" || exit 1
  [ ! -s "$sample_root/sampler.ready" ] || break
  sleep 0.05
done
[ -s "$sample_root/sampler.ready" ] || exit 1
printf 'import_start %s\n' "$(date -Is)" >"$sample_root/phases.log"
curl --fail --silent --show-error --request POST \
  http://127.0.0.1:13731/api/import/scan >"$sample_root/import.json"
python3 - "$sample_root/import.json" <<'PYRESULT'
import json, sys
result = json.load(open(sys.argv[1]))
assert result['success'] is True
print(result['data'])
PYRESULT
printf 'import_end %s\n' "$(date -Is)" >>"$sample_root/phases.log"
kill -TERM "$server_pid"
wait "$server_pid"
wait "$sampler_pid"
trap - EXIT
python3 - "$sample_root/rss.csv" <<'PYVERIFY'
import csv, sys
rows = list(csv.DictReader(open(sys.argv[1])))
assert rows, 'No RSS samples recorded'
print('samples:', len(rows), 'max_rss_kib:', max(int(row['rss_kib']) for row in rows))
PYVERIFY
printf '%s\n' "$sample_root"

Confirm that ps shows the Node process for this run’s crystal serve. Keep the log, configuration, import result, Node/application versions, and sample size. A new database may contain bundled demo data; record that baseline. The external sampler reads RSS every 50 milliseconds in KiB. VmHWM is the process’s cumulative RSS high-water mark since startup. Use a new temporary directory and process for each small, medium, or large sample so that an earlier run does not become the next baseline.

Start/end markers delimit the whole import; they do not identify scan, individual-file parsing, or synchronous-export peaks. Sampling can miss a brief peak. The high-water mark adds a process-level peak but cannot establish that it occurred during saving. The Linux kernel documentation notes that asynchronous RSS accounting can be imprecise; a finer snapshot can use the slower smaps. To attribute a peak to database export, add timestamped diagnostic markers around saveDatabase() in an isolated code copy and correlate them with external sampling. An import-completion log is not an export-start marker.

For remote import, sample the upload CLI and server PIDs separately. The local Electron server runs inside the main process and shares its PID; measure actual renderer, GPU, and other child processes separately. For a remote connection, measure the server independently, and record a remote-import worker separately as well. Measure a local model independently and avoid double-counting shared pages. This article provides neither a combined peak for these modes nor measured memory numbers. If an observed peak approaches available memory, reduce batches, disable unneeded sources, or schedule summaries separately, then retest with headroom for the OS, other processes, and sampling error.

Batch and recover deliberately

Enable one source at a time in settings, complete its scan, and confirm a successful database save before adding another. The local CLI currently does not forward --source to the import entry point, so that flag does not guarantee a single-source scan. For Cursor or Trae, organize measurements by workspace in isolated copies that retain required metadata; for long JSONL, identify unusually large files first. Avoid multiple processes sharing one data directory during import.

Keep each scan’s JSON/SSE result and server logs to reconcile imported, replaced, skipped, and error counts. crystal status only gives current totals, and note listings cannot reconstruct those four counts for an import pass. After an interruption, use these records and conversation details to decide whether to rerun. Imported records with the same file size and modification time are skipped; changed content is hashed, and replacement can enqueue vector cleanup for old notes. An empty queue does not prove that all summaries and embeddings are complete.

Restart normally when you need to release process-local objects:

crystal serve stop
crystal serve --daemon
crystal status

After recovery, validate one known conversation and one long conversation by checking the source, note state, and vector state. A performance conclusion is meaningful only with its sample, configuration, and measurement procedure.