Review AI coding conversations: denominators, quality, and reuse

Normalizing many AI coding conversations can support a more systematic review of sources, tasks, investigations, decisions, and reuse. Volume alone does not establish productivity. The original 5247 conversations, 3811 notes, tag rankings, and personal eight-month story have no auditable dataset here. This article replaces them with executable counting and quality-sampling methods.

Define denominators and missing coverage first

server/src/routes/status.ts counts every row in conversations, notes, and tags for stats, including demo data in the library. The non-demo realConvCount only determines isSeeded; the conversation join only supplies recentNotes. Neither condition filters the totals. These are current-library snapshots, not every original history in every tool. CLI conversations and notes list return paginated items/total; search returns an array. First-page counts, search hits, and tag usage are different measures. Record the target instance, pinned version, date, filters, pagination coverage, import errors, and unsupported sources. If your cohort excludes demo data, apply and document that rule explicitly.

Question	Suitable unit/denominator	Common confusion
Source and project distribution	Normalized conversations in one inclusion scope	Raw file count, repeated imports, missing sources
Length distribution	Per-conversation normalized messages with disjoint bins	Messages versus conversations, parser omissions
Tag distribution	Usage within a specified note cohort	Multi-tag totals need not equal note count
Summary coverage	Notes and filtering outcomes counted separately over included conversations	Filtered is not summary failure or an existing note
Reuse outcomes	Traceable acceptance on the same task	A tag hit or summary is not a solved problem

Source copies, normalized messages, model summaries, and readable exports are separate layers; see conversation management options. Missing thinking/tool records and constructed timestamps affect later statistics. A normalized library is not a lossless record of original behavior.

Ask falsifiable questions about four patterns

Evaluate short exchanges through the problem, process, outcome, and reuse conditions; a short complete fix can be valuable. Debugging notes should retain rejected hypotheses and validation, rather than only the final answer. Architecture discussions need options, constraints, tradeoffs, and actual checks, potentially in an ADR. For apparently repeated questions, compare project, version, cause, and outcome. Semantic similarity does not establish the same problem or a measured amount of work saved.

server/src/services/experience/gate.ts already implements signal prefiltering, model dimensions, and combination rules. Filtering is not merely a future plan based on fewer than five messages without code. server/src/services/summarize.ts stores filtered states and reasons for unaccepted conversations, with separate existing-note and user-rejection paths. Inspect summary, embedding, and relation outcomes separately; queue completion is not knowledge quality.

Demonstrate denominators and error lists with synthetic labels

Save this as retrospective.mjs and run it with Node 24. All six entries are hand-written synthetic cases. expected is an example human label; selected is a prewritten decision. No current gate, model, CLI, database, or real export executes. The program checks source totals, disjoint length bins, confusion counts, and empty denominators, retaining false-accept and false-reject IDs. Its precision and recall belong only to these six cases, not user data.

import assert from 'node:assert/strict';

// Hand-labelled synthetic cohort, not exported conversations or model judgments.
const cohort = [
  {
    source: 'fixture-a',
    id: 'short-fix',
    messages: 4,
    project: 'one',
    expected: true,
    selected: true,
  },
  {
    source: 'fixture-a',
    id: 'long-chat',
    messages: 100,
    project: 'one',
    expected: false,
    selected: true,
  },
  {
    source: 'fixture-b',
    id: 'quick-info',
    messages: 2,
    project: 'two',
    expected: false,
    selected: false,
  },
  {
    source: 'fixture-b',
    id: 'missed-fix',
    messages: 24,
    project: 'two',
    expected: true,
    selected: false,
  },
  {
    source: 'fixture-c',
    id: 'decision',
    messages: 45,
    project: 'three',
    expected: true,
    selected: true,
  },
  {
    source: 'fixture-c',
    id: 'unresolved',
    messages: 80,
    project: 'three',
    expected: false,
    selected: false,
  },
];
function audit(items) {
  const seen = new Set();
  const sources = {};
  const bins = [0, 0, 0, 0]; // <10, 10–29, 30–79, >=80
  let tp = 0,
    fp = 0,
    fn = 0,
    tn = 0;
  const falseAccepts = [],
    falseRejects = [];
  for (const item of items) {
    const key = `${item.source}:${item.id}`;
    assert.equal(seen.has(key), false);
    seen.add(key);
    assert.ok(Number.isInteger(item.messages) && item.messages >= 0);
    sources[item.source] = (sources[item.source] ?? 0) + 1;
    bins[item.messages < 10 ? 0 : item.messages < 30 ? 1 : item.messages < 80 ? 2 : 3]++;
    if (item.selected && item.expected) tp++;
    else if (item.selected) {
      fp++;
      falseAccepts.push(key);
    } else if (item.expected) {
      fn++;
      falseRejects.push(key);
    } else tn++;
  }
  const ratio = (a, b) => (b === 0 ? null : a / b);
  assert.equal(
    Object.values(sources).reduce((a, b) => a + b, 0),
    items.length
  );
  assert.equal(
    bins.reduce((a, b) => a + b, 0),
    items.length
  );
  assert.equal(tp + fp + fn + tn, items.length);
  return {
    total: items.length,
    sources,
    bins,
    tp,
    fp,
    fn,
    tn,
    precision: ratio(tp, tp + fp),
    recall: ratio(tp, tp + fn),
    falseAccepts,
    falseRejects,
  };
}
const result = audit(cohort);
assert.deepEqual(result.bins, [2, 1, 1, 2]);
assert.equal(result.precision, 2 / 3);
assert.equal(result.recall, 2 / 3);
assert.deepEqual(result.falseAccepts, ['fixture-a:long-chat']);
assert.deepEqual(result.falseRejects, ['fixture-b:missed-fix']);
assert.equal(audit([]).precision, null);
assert.throws(() => audit([...cohort, cohort[0]]));
const lengthOnly = cohort.map((item) => ({ ...item, selected: item.messages >= 5 }));
assert.ok(audit(lengthOnly).falseRejects.includes('fixture-a:short-fix'));
console.log(JSON.stringify({ scope: 'six synthetic labelled fixtures only', result }, null, 2));

Make counting errors detectable

Remove duplicate source:id rejection: the duplicate-sample assert.throws must fail. Change an empty-denominator ratio to 0: the precision=null assertion must fail. Restore and rerun. The length rule missing short-fix is an example countercase, not a model or product ranking.

For real review, retain all included and excluded IDs, sample by source, task, length, and decision state, and independently label omissions, inventions, misattribution, and reuse conditions against the same originals. Retain disagreements and adjudication. Separate training/rule changes from held-out acceptance; report concrete false accepts, false rejects, and denominators rather than one aggregate pass rate. Fixed samples in server/src/services/experience/eval-samples.json declare synthetic_calibration_cases and contains_real_user_data=false. Rule-calibration results cannot establish quality on a personal library or real model.

Turn question answering into traceable reuse

Before a task, search history using actual symptoms and project context. During work, give the assistant the original evidence, version, and applicability of retrieved notes. Afterwards, check whether problem, process, outcome, and provenance are complete. Record reasons and acceptance when correcting content. Deletion feedback in server/src/services/experience/reviews.ts records quality reasons; it is not a complete version history for every edit.

Tags, summaries, and relations serve different purposes. Claims that tags always matter more or that approximate summaries with greater volume are better require same-task comparison. Counts cannot establish insufficient testing or a doubled coding speed. Fix task-success criteria first, compare usable evidence retrieval, execution outcomes, and manual correction, then discuss benefits.

Richer decision records, code-dependency validation, member permissions, or cross-library retrieval can be future design goals. Separate them from existing filtering, writeback receipts, and remote import. Use graph-assisted refactoring to turn relational clues into fault checks, and team-sharing boundaries to retain approval and identity responsibilities. The series closes with reviewable usage methods, not unverified growth predictions.