---
title: "Comparing LLM Summaries: Fixed Inputs and Verifiable Evidence"
description: Evaluate local vs cloud LLM summarization on identical transcripts, field review, and failure logs, separating model quality from app pipelines.
image: "https://zengliangyi.com/og/blog/en/llm-summary-evaluation.png"
---

A model fits a note library when repeatable inputs produce traceable results. This article provides a method, not measured model rankings, latency, or price tables. Local inference still has hardware, energy, and maintenance costs.

## Choose the pipeline being compared

The fixed [summary implementation](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/services/summarize.ts) prepares a transcript and runs an experience gate before calling the summarizer. An existing note returns its ID; switching the LLM does not overwrite it. After persistence, embedding and relation discovery can fail without failing the summary. Separate end-to-end completion time from summary-model generation time.

For model-only comparison, freeze one visible transcript and the same SYSTEM_PROMPT and SummarizeSchema, then submit identical material through an independent evaluation program. Clicking an existing conversation again is not regeneration. For application comparison, use separate test data directories for each configuration, import identical source copies, and record gate acceptance/filtering, summary errors, vector state, and final notes. `DATA_DIR` does not isolate sources or select the client’s server. See the [measurement and isolation guide](/en/blog/conversation-import-memory-budget).

## Freeze samples and actual inputs

Include authorized Chinese, English, debugging, tradeoff, long-conversation, and no-durable-lesson samples. Record sample ID, source hash, application commit, complete model identifier, prompt version, and actual-input hash. Remove credentials and retain the redaction rules; send only authorized material to the selected service.

[Transcript preparation](https://github.com/ZengLiangYi/ChatCrystal/blob/764276814e5c5a33ecbfeae63754f74b9261a3a8/server/src/services/transcript.ts) uses a configured character budget, not an exact token count. It selects some assistant messages per turn. For long conversations it retains the first and final two turns and compresses the middle to budget, but fixed turns can still make the final text exceed that budget. Save the actual transcript instead of claiming every model saw the full original.



```bash
sha256sum   sample.jsonl   transcript.txt   system-prompt.txt
```



These filenames are evaluation conventions, not application exports. Save the material through an isolated evaluation program before hashing it.

## Review five fields against the source

| Field | Human review question |

| --- | --- |

| title | Does it name the specific issue without overstating the outcome? |

| summary | Are decisions, their reasons, and conditions retained without added facts? |

| key_conclusions | Can each conclusion be traced to visible evidence, without repetition? |

| code_snippets | Do code, versions, and explanations come from the source, and are snippets understandable on their own? |

| tags | Are they useful for retrieval and consistent with team conventions? |

The fixed implementation uses `generateText` with `Output.object`, an output limit of 4096 tokens, and SDK retry configuration 3. The schema constrains field types; title length and requested item counts are descriptions and prompt instructions without matching hard length/count validators. Parseable JSON is not a truth proof, and calls can fail.

Define “meets / partly meets / does not meet” for each dimension before running. Hide model names and have two reviewers locate evidence, resolving disagreement against the source. Evaluate gate-only samples separately; do not assign summary-quality scores when no summary call occurred. Establish objectives, data, and metrics first; see [OpenAI evaluation guidance](https://developers.openai.com/api/docs/guides/evaluation-best-practices).

## Record quality, failure, and resources separately

Keep outputs, field reviews, call start/end times, retries, errors, and invoices or available token usage. Character counts do not establish cost. Record hardware, quantization, and cold/warm startup for local models, and region/network conditions for cloud calls. Repeat runs and report distributions and failures instead of extrapolating from one success.

The queue limits task starts, not every individual model request. One task can include gating, summarization, other calls, and retries; it does not guarantee free quotas or immunity to rate limits. Verify connectivity through the [LLM configuration guide](/en/blog/cloud-llm-configuration), then choose a configuration from the evidence rather than the provider name.

Share:

##  Related Posts

[##  Review AI coding conversations: denominators, quality, and reuse

Audit source and length denominators, false accepts and rejects, and empty samples; separate current filtering, summaries, and measured reuse.          曾亮懿       October 7, 2026      5 min read

#ChatCrystal

#Knowledge management

#Evaluation](/en/blog/conversation-knowledge-retrospective)    [##  Evaluate embedding retrieval with fixed notes and labeled queries

Compare local and cloud embeddings with labeled queries and separate indexes, checking rebuild completion, retrieval quality, and resource costs.          曾亮懿       October 4, 2026      3 min read

#ChatCrystal

#Embedding

#Evaluation](/en/blog/embedding-retrieval-evaluation)    [##  Configuring Cloud LLMs: Providers, Endpoints, and Summaries

Follow ChatCrystal provider registrations to configure remote generation and check native APIs, gateways, persisted settings, credentials, and test boundaries.          曾亮懿       October 3, 2026      4 min read

#ChatCrystal

#LLM

#Configuration](/en/blog/cloud-llm-configuration)

##  From writing to code

Explore ChatCrystal, developer tools and daily learning notes on my GitHub. Questions about the implementation are welcome.

[Visit my GitHub](https://github.com/ZengLiangYi)

```json
{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "name": "ZengLiangYi Blog",
  "alternateName": [
    "Zeng Liangyi",
    "ZengLiangYi",
    "zengliangyi.com"
  ],
  "url": "https://zengliangyi.com",
  "description": "曾亮懿的个人网站。记录 AI Native 探索、开发者工具与全栈产品实践。"
}
{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "Comparing LLM Summaries: Fixed Inputs and Verifiable Evidence",
  "description": "Evaluate local vs cloud LLM summarization on identical transcripts, field review, and failure logs, separating model quality from app pipelines.",
  "url": "https://zengliangyi.com/en/blog/llm-summary-evaluation/",
  "image": "https://zengliangyi.com/images/blog/llm-summary-evaluation.svg",
  "datePublished": "2026-10-04T00:00:00.000Z",
  "dateModified": "2026-10-23T00:00:00.000Z",
  "author": {
    "@type": "Person",
    "name": "曾亮懿",
    "url": "https://zengliangyi.com/en/about"
  },
  "publisher": {
    "@type": "Organization",
    "name": "ZengLiangYi Blog",
    "logo": {
      "@type": "ImageObject",
      "url": "https://zengliangyi.com/favicon-96x96.png"
    }
  },
  "mainEntityOfPage": {
    "@type": "WebPage",
    "@id": "https://zengliangyi.com/en/blog/llm-summary-evaluation/"
  }
}
{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    {
      "@type": "ListItem",
      "position": 1,
      "name": "Home",
      "item": "https://zengliangyi.com/en"
    },
    {
      "@type": "ListItem",
      "position": 2,
      "name": "Blog",
      "item": "https://zengliangyi.com/en/blog"
    },
    {
      "@type": "ListItem",
      "position": 3,
      "name": "Comparing LLM Summaries: Fixed Inputs and Verifiable Evidence",
      "item": "https://zengliangyi.com/en/blog/llm-summary-evaluation/"
    }
  ]
}
```