--- title: "Building an Embedding Service: Text, Chunks, and Commits" description: Read ChatCrystal’s actual embedding pipeline, separate text selection, soft chunking, vector validation, and commits, and verify boundaries offline. image: "https://zengliangyi.com/og/blog/en/embedding-service-design.png" --- The text an embedding service accepts, its chunking rules, and the point where it replaces the old index affect retrieval and failure recovery. This article separates those responsibilities in ChatCrystal’s pinned implementation, then verifies chunk and vector boundaries without calling a model. ## Decide which text becomes searchable This article follows [embedding.ts](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/services/embedding.ts). The service deduplicates and joins the title, summary, key conclusions, tags, and code-snippet descriptions. For ordinary imported-conversation notes, that does not put the raw conversation and every code body into the vector input. Two memory sources, `agent-writeback` and `manual-note`, have additional rules: they include code bodies and available fields for root causes, resolutions, pitfalls, reusable patterns, decisions, error signatures, and file paths. Each code body has consecutive whitespace collapsed before being limited to 1000 JavaScript character units. This distinction affects which notes can be retrieved by code content. Code bodies cannot be treated as universally excluded from vector input. Write down the actual input before choosing a model. Switching embedding models cannot directly recover information removed from that input. See [embedding configuration](/en/blog/embedding-model-configuration/) for provider capabilities and settings. ## 500 is a merging target, not a hard limit The chunker uses `CHUNK_SIZE = 500`, splits at blank lines, and tries to merge adjacent paragraphs. JavaScript `.length` counts UTF-16 code units rather than tokens. It does not guarantee that every chunk stays below 500: a 700-unit paragraph without blank lines remains one chunk. There is no overlap window either. The following standalone JavaScript version preserves the current algorithm’s decision order while simplifying variable names. Save it as `lesson.mjs` in a temporary directory and run with Node 24+. It needs no dependencies and accesses no knowledge-base data. ```js import assert from 'node:assert/strict' ; function chunkText ( text ) { const size = 500 ; if (text. length <= size) return [text]; const chunks = []; let current = '' ; for ( const paragraph of text. split ( / \n\n + / )) { if (current. length + paragraph. length + 2 > size && current. length > 0 ) { chunks. push (current. trim ()); current = paragraph; } else { current += (current ? ' \n\n ' : '' ) + paragraph; } } if (current. trim ()) chunks. push (current. trim ()); return chunks; } assert. deepEqual ( chunkText ( 'a' . repeat ( 700 )). map (( s ) => s. length ), [ 700 ] ); assert. deepEqual ( chunkText ( 'a' . repeat ( 300 ) + ' \n\n ' + 'b' . repeat ( 300 )). map (( s ) => s. length ), [ 300 , 300 ] ); assert. equal ( '😀' . length , 2 ); console. log ( 'chunk boundaries pass' ); ``` Change the first expectation to `[500, 200]` and the program fails; restore `[700]` and it passes. This establishes that the implementation does not hard-split long paragraphs. It does not establish compliance with any model’s token limit. A strict limit would need model-specific counting, long-paragraph splitting, and corresponding counterexamples; these are proposed improvements. ## Generate every vector before starting the update The service calls AI SDK `embed` sequentially and prepares every chunk’s result before opening a Vectra update. If a model call fails beforehand, this replacement does not begin. The factories in [providers.ts](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/services/providers.ts) create models through `embeddingModel(model)`; generation and embedding settings are separate. When designing a service, you can add an explicit vector contract before committing. Append the following to the same file. It is a **teaching design, not an existing application validator**. `embedOne` is an injected fake model and `commit` is a callback. ```js async function stageVectors ( texts , embedOne , commit ) { const vectors = []; let dimension; for ( const text of texts) { const vector = await embedOne (text); const norm = Array. isArray (vector) ? Math. hypot ( ... vector) : NaN ; if ( ! Array. isArray (vector) || vector. length === 0 || ! vector. every (Number.isFinite) || ! Number. isFinite (norm) || norm === 0 ) { throw new Error ( 'invalid vector' ); } dimension ??= vector. length ; if (vector. length !== dimension) throw new Error ( 'dimension mismatch' ); vectors. push (vector); } await commit (vectors); } let commits = 0 ; await assert. rejects ( stageVectors ( [ 'first' , 'second' ], async ( text ) => (text === 'first' ? [ 1 , 0 ] : [ 1 , 0 , 0 ]), async () => { commits ++ ; } ), / dimension mismatch / ); assert. equal (commits, 0 ); await assert. rejects ( stageVectors ( [ 'zero' ], async () => [ 0 , 0 ], async () => { commits ++ ; } ), / invalid vector / ); await stageVectors ( [ 'valid' ], async () => [ 1 , 0 ], async () => { commits ++ ; } ); assert. equal (commits, 1 ); console. log ( 'staging boundaries pass' ); ``` These fixed two- and three-dimensional vectors verify callback timing and validation order; they represent no language semantics. The lesson has not defined empty input, model-identity records, concurrent writes, or transactions across files. A real service needs those contracts too. ## Matching dimensions are insufficient Two models with the same output dimension can still have incompatible vector spaces. The current implementation reads the configured model but does not store a complete model fingerprint in each index record for compatibility validation. When switching models, treat old-index compatibility as a separate concern rather than comparing array lengths alone. The actual commit crosses SQLite state and a Vectra file: a database transaction replaces chunk mappings and marks the note `syncing`, the database is saved, old vectors are deleted, the Vectra update ends, and the note is marked `done` and saved again. There is no atomic transaction spanning both stores. See [index recovery](/en/blog/vectra-index-recovery/) for failure cleanup and recovery. Preparing before committing describes only the model-call stage’s position. Continue with the [fixed-vector local-search exercise](/en/blog/vectra-local-search/) to observe persistence, filters, and scores, or read the official [embed reference](https://ai-sdk.dev/docs/reference/ai-sdk-core/embed). Check input coverage, chunk boundaries, vector validity, model compatibility, and commit failures separately; one successful search does not establish the whole pipeline’s behavior. Share: ## Related Posts [## Structured JSON Prompts: Field Design & Verifiable Evidence Use ChatCrystal’s summaries and relations to define fields, visible evidence, and counterexamples, then distinguish shape and evidence checks with offline Zod. 曾亮懿 October 6, 2026 4 min read #ChatCrystal #AI #Node.js](/en/blog/structured-json-prompts) [## Vectra local search: fixed vectors, filters, and persistence Run a deterministic Vectra 0.15.0 exercise in a temporary directory to check cosine scores, metadata filters, reopening, and canceled updates. 曾亮懿 October 6, 2026 4 min read #ChatCrystal #AI #Node.js](/en/blog/vectra-local-search) [## AI SDK structured notes: adapters, schemas, and business checks Read ChatCrystal’s AI SDK 7 calls, distinguish provider adaptation from output shape and factual validity, and test description limits with offline Zod. 曾亮懿 October 5, 2026 4 min read #ChatCrystal #AI #Node.js](/en/blog/ai-sdk-structured-notes) ## From writing to code Explore ChatCrystal, developer tools and daily learning notes on my GitHub. Questions about the implementation are welcome. [Visit my GitHub](https://github.com/ZengLiangYi) ```json { "@context": "https://schema.org", "@type": "WebSite", "name": "ZengLiangYi Blog", "alternateName": [ "Zeng Liangyi", "ZengLiangYi", "zengliangyi.com" ], "url": "https://zengliangyi.com", "description": "曾亮懿的个人网站。记录 AI Native 探索、开发者工具与全栈产品实践。" } { "@context": "https://schema.org", "@type": "BlogPosting", "headline": "Building an Embedding Service: Text, Chunks, and Commits", "description": "Read ChatCrystal’s actual embedding pipeline, separate text selection, soft chunking, vector validation, and commits, and verify boundaries offline.", "url": "https://zengliangyi.com/en/blog/embedding-service-design/", "image": "https://zengliangyi.com/images/blog/embedding-service-design.svg", "datePublished": "2026-10-06T00:00:00.000Z", "dateModified": "2026-10-23T00:00:00.000Z", "author": { "@type": "Person", "name": "曾亮懿", "url": "https://zengliangyi.com/en/about" }, "publisher": { "@type": "Organization", "name": "ZengLiangYi Blog", "logo": { "@type": "ImageObject", "url": "https://zengliangyi.com/favicon-96x96.png" } }, "mainEntityOfPage": { "@type": "WebPage", "@id": "https://zengliangyi.com/en/blog/embedding-service-design/" } } { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "position": 1, "name": "Home", "item": "https://zengliangyi.com/en" }, { "@type": "ListItem", "position": 2, "name": "Blog", "item": "https://zengliangyi.com/en/blog" }, { "@type": "ListItem", "position": 3, "name": "Building an Embedding Service: Text, Chunks, and Commits", "item": "https://zengliangyi.com/en/blog/embedding-service-design/" } ] } ```