Comparing nomic-embed-text with text-embedding-3-small requires more than dimensions or one search score. This article contains no measured benchmark. Its method evaluates whether identical notes yield evidence-supported results for identical queries.
Freeze notes so summarization is not a variable
Use the same verified snapshot for each model, including titles, summaries, conclusions, tags, source types, and memory fields. Record the application commit, full model identifier, input-text hashes, note count, and actual chunk count. Regenerating summaries separately would change the input.
Embedding text construction combines note fields. Memory notes also include causes, resolutions, errors, files, and up to 1000 characters of evidence per code snippet. Chunking groups paragraphs toward 500 characters; an oversized paragraph is not hard-split at 500. Comparable input requires identical content and source types.
Check current specifications, supported inputs, and fees with the provider when running, and record the date. More dimensions do not guarantee better retrieval on these queries. See embedding configuration.
Label queries before seeing results
Include exact terms, natural-language descriptions, mixed-language queries, near-but-different issues, and questions with no answer in the library. Label acceptable note IDs and source evidence for each query, retaining the annotation version. These are constructed categories, not measured hits:
| Query category | Constructed query | Labeling task |
|---|---|---|
| Specific mechanism | SQLite transaction rollback boundaries | Distinguish transaction failure from ordinary saving |
| Paraphrase | Connections remain occupied after request failure | Locate connection-lifetime evidence |
| No answer | A deployment approach absent from the library | Mark no answer and inspect false matches |
For answerable queries, Precision@5 is correct results in the first five positions divided by 5; missing positions are not correct. Recall@5 is retrieved labeled notes divided by all acceptable notes for that query. Exclude no-answer queries from this recall calculation and track false matches and unsupported actions separately. Set metrics before inspecting results.
Separate indexes and verify rebuilding
Use independent test copies and indexes for each configuration. When a changed embedding provider or model is confirmed, the fixed configuration route clears old vectors and sets notes to pending. It does not finish rebuilding. This request only submits eligible work:
curl --fail --request POST 'http://localhost:3721/api/embeddings/batch'queued is an enqueue count, not a success count. Inspect individual note states, task errors, and known queries rather than an empty queue alone. Startup can set status from embedding rows even when index files are missing. See Vectra recovery.
First disable relation expansion and save raw responses at the same k:
curl --fail --get 'http://localhost:3721/api/search' \
--data-urlencode 'q=SQLite 事务回滚边界' \
--data-urlencode 'limit=5' \
--data-urlencode 'expand=false' > result.jsonThe fixed search route returns snake_case IDs and application-adjusted scores. Term boosts, thresholds, and deduplication mean score is not a cross-model accuracy metric. Compare IDs against human labels. Evaluate relation expansion in a separate run rather than mixing retrieval settings.
Report reviewable evidence
Retain ordered results, human judgments, no-answer false matches, errors, latency, and resource records for every query. Separate indexing, query calls, and end-to-end timing; use actual token usage or invoices. Account for local hardware, energy, and maintenance too. Without these records, provide a method instead of a winner or a library-size threshold.
Summary models remain independently configurable; see summary evaluation. Changing summarization, chunking, and embedding together cannot establish which caused the difference.