A model fits a note library when repeatable inputs produce traceable results. This article provides a method, not measured model rankings, latency, or price tables. Local inference still has hardware, energy, and maintenance costs.
Choose the pipeline being compared
The fixed summary implementation prepares a transcript and runs an experience gate before calling the summarizer. An existing note returns its ID; switching the LLM does not overwrite it. After persistence, embedding and relation discovery can fail without failing the summary. Separate end-to-end completion time from summary-model generation time.
For model-only comparison, freeze one visible transcript and the same SYSTEM_PROMPT and SummarizeSchema, then submit identical material through an independent evaluation program. Clicking an existing conversation again is not regeneration. For application comparison, use separate test data directories for each configuration, import identical source copies, and record gate acceptance/filtering, summary errors, vector state, and final notes. DATA_DIR does not isolate sources or select the client’s server. See the measurement and isolation guide.
Freeze samples and actual inputs
Include authorized Chinese, English, debugging, tradeoff, long-conversation, and no-durable-lesson samples. Record sample ID, source hash, application commit, complete model identifier, prompt version, and actual-input hash. Remove credentials and retain the redaction rules; send only authorized material to the selected service.
Transcript preparation uses a configured character budget, not an exact token count. It selects some assistant messages per turn. For long conversations it retains the first and final two turns and compresses the middle to budget, but fixed turns can still make the final text exceed that budget. Save the actual transcript instead of claiming every model saw the full original.
sha256sum sample.jsonl transcript.txt system-prompt.txtThese filenames are evaluation conventions, not application exports. Save the material through an isolated evaluation program before hashing it.
Review five fields against the source
| Field | Human review question |
|---|---|
| title | Does it name the specific issue without overstating the outcome? |
| summary | Are decisions, their reasons, and conditions retained without added facts? |
| key_conclusions | Can each conclusion be traced to visible evidence, without repetition? |
| code_snippets | Do code, versions, and explanations come from the source, and are snippets understandable on their own? |
| tags | Are they useful for retrieval and consistent with team conventions? |
The fixed implementation uses generateText with Output.object, an output limit of 4096 tokens, and SDK retry configuration 3. The schema constrains field types; title length and requested item counts are descriptions and prompt instructions without matching hard length/count validators. Parseable JSON is not a truth proof, and calls can fail.
Define “meets / partly meets / does not meet” for each dimension before running. Hide model names and have two reviewers locate evidence, resolving disagreement against the source. Evaluate gate-only samples separately; do not assign summary-quality scores when no summary call occurred. Establish objectives, data, and metrics first; see OpenAI evaluation guidance.
Record quality, failure, and resources separately
Keep outputs, field reviews, call start/end times, retries, errors, and invoices or available token usage. Character counts do not establish cost. Record hardware, quantization, and cold/warm startup for local models, and region/network conditions for cloud calls. Repeat runs and report distributions and failures instead of extrapolating from one success.
The queue limits task starts, not every individual model request. One task can include gating, summarization, other calls, and retries; it does not guarantee free quotas or immunity to rate limits. Verify connectivity through the LLM configuration guide, then choose a configuration from the evidence rather than the provider name.