Girl Chess · Coach model evaluation

Coach Eval v3: Sonnet vs Opus

Two AI models competed for one job: powering an in-game coaching assistant. The call is to keep the shipped model; this shows why, and the controls that make it trustworthy. The second tab covers how the evaluation was built, challenged, and hardened.

Read this first · noted 2026-08-18

This page truthfully records what was measured on 2026-07-23. Three things have changed since. The 45-word length budget was retired on 07-28; the cap is now 150 words and the sentence limit is gone. The 45-second timeout is now 180 seconds, so the threshold line on the latency chart sits four times too low. And the verdict rested on that length axis: a later run reverses it for live-board questions, giving those to Opus on p90 latency.

Sonnet (claude-sonnet-5, current default) Opus (claude-opus, challenger) Large dot = middle of 3 repeats · line = lowest to highest repeat

Headline recommendation

The winner was picked by a decision rule written down before any data existed, applied to the measures in a fixed priority order. The choice is a rule outcome, not taste.

Recommendation · decided by

The decision rule, evaluated in order (pre-registered as "D4" before any data existed)

Plain language. If one model leaks internal engineering terms clearly less often (gap over 3 points, no overlap across repeats), it wins here. Tied at 100%, falls through.
Answer length. If one model stays inside the coach's word budget clearly more often (gap over 5 points, no overlap), it wins here. this step fired: 33-point gap, no overlap → sonnet
Responds to her held move. Counts only because its automated scorer passed a 95% human audit first. Never reached.
Tie. Keep the current default, sonnet. Not needed.

Reported, not suppressed The scorecard is not one-sided. Sonnet wins on the measure weighted highest, keeping answers short, but reliability runs the other way: sonnet fails over to a canned reply on 8.2% of turns against opus's 0.5%, and sonnet's first run (12.3%) even breached the 10% health gate both models were expected to clear. Section 5 treats that as a real product finding against the winner, not a footnote. Both models scored 100% on finishing every reply cleanly.

Cost to adopt The winner is the current default (AGENT_SDK_MODEL = claude-sonnet-5), so it confirms the shipped model at no cost.

TakeawayKeep sonnet, the shipped default: plain language tied, so length decided it: 55% inside budget vs opus's 22%, no overlap. The cost is reliability (section 5); the pick holds until that flips.

Plain language: does the coach sound like a person?

Share of replies with zero internal engineering vocabulary. The coach should sound like a person, never the machinery behind it. (Internal metric name: jargon-clean rate.)

Replies free of engineering terms (0-100%)

Why the earlier gap vanished, and the lesson it taught An earlier comparison showed 85% vs 38% and looked like one model naturally "talks like an engine." The real cause was the coach's own instruction text, which mentioned the engine. The models were quoting their instructions back. With that instruction text fixed, both models score 100%. The lesson this rerun cemented: fix the structure before comparing models. A broken prompt can make two models look different when they are not, and a bigger model cannot fix a broken prompt.

Technical detail: baseline ran against persona commit ef8aa98 ("the engine's read"); current persona (10619ce) contains zero "engine" language, one un-branched prompt path, plus a runtime voice guard. The checker behind this number was hand-audited at 100% agreement (50/50 rows, both audit rounds).

TakeawayBoth models used plain language in 100% of replies across all six runs. The earlier 85%-vs-38% gap was a prompt bug, not a model difference.

Brevity: does the coach keep answers short?

this decided the pick

Share of replies inside the coach's length budget (45 words / 3 sentences; 20 words for quick confirmations). Short answers matter because the coach speaks in the middle of a live game.

Replies inside the length budget (0-100%)

Honest frame, and a fairness concern checked and cleared Neither model is good at this yet: sonnet misses the budget almost half the time. But sonnet is decisively closer to the coach the spec describes, in every one of six runs. Separately, the stakeholder suspected failed replies (timeouts served as canned fallbacks) were being counted against this score. That was verified directly in the scoring code and in the raw counts: failed replies never enter this measure. The concern did not reproduce, so nothing was changed, and the rule is printed below so the check stays visible.

"template/timeout/error rows are pipeline failures and are excluded from this denominator, verified in harness code and in baseline denominators 55/65". Exclusion rule, quoted verbatim from the pre-registered plan. Per-run sample sizes (57/62/60 and 64/65/65) equal 65 minus each run's failed replies, so the exclusion is visible on inspection.

TakeawaySonnet stays inside the 45-word budget 55% of the time (47% to 56% across repeats) against opus's 22% (19% to 28%). The ranges never overlap, and that decided the pick.

Staying on topic: does the coach address the move she is weighing?

When the player is holding a candidate move and asks about it, the reply should engage that specific move, judged by a human-audited reading, not keyword matching. Small sample: 14 such prompts per run, so a single miss moves this score 7 points. (Internal metric name: pending-awareness.)

Replies that engage her held move (0-100%)

This number survived a human audit The first automated scorer only gave credit when a reply named the exact square. It failed replies that plainly described her move in words ("steps your knight away from the bishop's attack"), and in an earlier version could pass replies that mentioned a square by coincidence. It was rebuilt and re-audited to 28-out-of-28 agreement with human labels before this number was accepted.

TakeawayBoth models address her held move nearly every time (medians 100%; sonnet missed once in 42 chances), so this measure never had to decide the pick.

Reliability: how often live coaching fails over to a canned reply

opus's strongest case

When a reply takes longer than 45 seconds, the player sees a canned fallback message instead of live coaching. This is where opus is genuinely stronger, and the gap survived every control applied to the test.

Canned-fallback rate per run (0-15%)

Timing-bias control: sonnet's fallback rate, before vs after

Each run began with 3 discarded warm-up calls, timing the runner after the warm-up laps, not during them. The models alternated which went first, so time-of-day effects could not quietly favor one side. Uncontrolled baseline: sonnet 10/65 failures front-loaded, opus 0/65.

Response time per run: typical (median) and slowest-tenth (p90), against the 45s limit

Test artifact vs product finding, kept separate Part of the earlier failure rate was the test's fault: cold processes penalized whichever model was timed first, and warm-up roughly halved sonnet's measured failures (15.4% before controls, 8.2% after). What remains is a real product finding, not explained away: even warm and alternated, sonnet serves the canned fallback on roughly 1 turn in 12, with zero hard errors. The pattern is always the same: its slowest replies ride the 45-second limit, on the same two hard open-ended questions in all three of its runs. This finding argues against the recommended model and is reported anyway; it lost only because the decision rule weights brevity above reliability. If that weighting ever flips, the options start here: a longer time limit, an automatic retry, or revisiting the model choice.

TakeawayEven with warm-up and alternated ordering, sonnet fell back to a canned reply on 16 of 195 turns while opus did once: roughly 1 turn in 12 versus 1 in 195, the strongest argument on opus's side.

Can the numbers be trusted? The scoring system was itself audited

Every score on this page comes from an automated checker. Before any of them were believed, a human hand-labeled hundreds of replies and compared their judgment to the checker's, requiring 95% agreement on every measure. Bars show final human-vs-checker agreement.

Automated checker vs human judgment, per measure (0-100%)

What the audit caught Two audit rounds (473 then 466 hand-labeled replies). Round 1 found two real scoring bugs, both in the same direction: the checker failed replies a human would pass. A text rule flagged the ordinary word "lets" as if it were the banned casual phrase "let's", and the on-topic scorer refused credit to replies that described her move without naming its square. Both were fixed as exact, testable rules with regression tests, not adjusted until the numbers agreed, and round 2 confirmed the fixes on fresh samples with no new failure modes. One known 1-in-285 word-count quirk (a dash counted as a word) was documented and accepted rather than patched. Honest scope note: one rare reply pattern the fix could theoretically mis-credit never occurred in either sample, so its behavior there is unmeasured.

Technical detail: fixes landed as commits e2d56b8 and 0b58498; full disagreement-by-disagreement ledger in audit/iter-1.md and audit/iter-2.md.

TakeawayAfter two audit rounds, the automated checker agrees with human judgment at 99.6% or better on every measure. The numbers on this page come from a verified instrument, not a trusted script.

How the comparison was controlled

The controls that make the pick defensible, and the limitations stated in the same breath.

Run order: six runs, alternating which model went first

Limitations, stated plainly Three repeats in a single night: the 33-point gap that decided the pick is robust to that, but the small numbers (failure rates, the single on-topic miss) carry real run-to-run noise. Response times ran on a shared subscription backend whose speed varies by the hour. The same question ranged from 20 to 80 seconds, so these timings describe this backend on this night, not the models in the abstract. The alternation was imperfect: sonnet still went first in 2 of 3 pairs, so warm-up is the primary control and ordering the backup. The pick is a mechanical rule standing in for the owner's blinded side-by-side read; if her blind preference contradicts it, that is a finding to investigate, not an error to suppress. Two judgment calls are flagged for her review: her "over 95% accuracy" bar was interpreted as checker-vs-human agreement, and the tie rule defaults to the incumbent model.

TakeawayThree repeats per model, alternated ordering, warmed processes, a database verified untouched after every run, a decision rule written before the data existed, and a blinded copy waiting for a human read.

Why this tab exists

The Results tab says which model won. This tab is about whether the process behind that answer deserves trust.

This evaluation did not start out trustworthy. The first attempt had measurement gaps, and the second produced a result the project's stakeholder explicitly did not believe. She named three specific concerns. What follows is the record of what happened next: each concern investigated on its merits rather than defended against, the measuring stick itself audited before any number was reported, and one finding that argues against the winning model promoted to the headline instead of buried. The domain here is a game-coaching assistant, but nothing below is specific to it. This is how any comparison of two AI systems should earn a stakeholder's trust.

Three versions to a defensible answer

Each round existed because the previous one had a flaw someone was willing to name.

Round 1 · first attempt

Numbers, but not evidence

Basic execution bugs, including replies cut off by the display before they were scored, plus a single measurement per test case and no controls for when each system ran. Its numbers could not separate real differences from noise, and were not used.

Round 2 · baseline

A real comparison, distrusted

Fixed the execution bugs and produced the first usable head-to-head. The stakeholder still did not trust the result, and said exactly why, with three specific concerns (below). That pushback, not the data, is what shaped the final design.

Round 3 · this run

A controlled experiment

Rebuilt with the controls of a proper A/B test: three repeats per system, a warm-up pass before anything was timed, alternated run order, a decision rule written before the data existed, and a human audit of the scoring system before any number was believed.

TakeawayThe evaluation took three rounds to become trustworthy, and the improvements came from taking a stakeholder's distrust seriously rather than defending the first result.

The stakeholder pushback: three concerns, three different outcomes

Each concern got its own investigation and its own answer: one real flaw fixed with controls, one artifact traced to its cause, one complaint that did not reproduce and was deliberately left alone.

1. "One system was timed cold."real, fixed with controls

The two systems had run under different conditions: whichever went first paid a start-up penalty the other never faced, inflating its failure count.

What was done: the rerun added a warm-up pass: several discarded practice calls before anything counted, the same idea as timing a runner after their warm-up laps, not during them. It also alternated which system ran first, so time-of-day effects could not quietly favor one side. The penalized system's measured failure rate fell from 15.4% to 8.2% (chart in section 05).

2. "The test looks biased against one system."stale-data artifact, verified not a bias

One system appeared to leak internal engineering terms far more often (85% vs 38% clean), which read as either a test bias or a damning model trait.

What was done: traced, not assumed. The gap came from a stale snapshot: the earlier test had run against an old version of the assistant's instruction text that itself mentioned the engine. The models were quoting their instructions back. The instruction fix already existed in the product; the rerun against current instructions showed both systems at 100%. The test was not biased, and neither model "talks like an engine."

3. "The scoring rule punishes one side unfairly."checked, does not reproduce

Failed replies (timeouts served as canned fallbacks) were suspected of also counting against that system's quality scores: a double penalty.

What was done: checked directly in the scoring code and re-derived from the raw counts: failed replies never enter any quality score. The concern does not reproduce in the current code, so nothing was changed: no silent patch to something that was not broken. Instead, the exclusion rule is now printed on the results page itself, so anyone can verify it.

TakeawayOf three stakeholder concerns, one was a real flaw fixed with experimental controls, one was a stale-data artifact traced to its cause, and one did not reproduce and was cleared in the open rather than silently "fixed."

Auditing the measuring stick before trusting it

All scores come from an automated grader. Before believing any of them, a human hand-labeled samples of replies and compared their judgment to the grader's, requiring 95% agreement on every measure, with a hard cap of 10 audit rounds so the loop could not run forever.

Human-vs-grader agreement by measure: audit round 1 vs round 2, against the 95% bar

The audit found real bugs, not noise Round 1 failed two of five measures, and both failures were genuine scoring bugs: cases where the grader failed replies a human would pass. One: a text rule meant to catch the banned casual phrase "let's" also flagged the ordinary verb "lets", as in "this lets you take the bishop." Two: the on-topic grader only gave credit when a reply named an exact board square, so a reply that clearly described the player's move in words was scored as off-topic. Both fixes were exact, describable rules with regression tests: require the apostrophe, accept a described move. Neither was "adjust until the numbers agree." Round 2 re-checked on fresh samples: every measure at or above 95%, no new failure modes, and one known 1-in-285 word-counting quirk documented and accepted rather than patched.

TakeawayThe audit caught two real scoring bugs on its first pass and reached 95%+ human agreement on every measure by round two of the ten allowed.

What the timing controls changed, and what they didn't

The same reliability measure, before and after the warm-up and alternated-ordering controls. Failure here means the player saw a canned fallback message instead of live coaching.

Canned-fallback rate, uncontrolled baseline vs controlled rerun

Half artifact, half finding, and the finding cuts against the winner Roughly half of the recommended system's measured failure rate was the test's fault, and vanished once the test was fair. The half that remains is real: it recurs on the same hard questions in every repeat, and it means the recommended system still shows a canned fallback about 1 turn in 12 while the challenger almost never does. That weakness is reported in the headline section of the Results tab, not hidden. A losing signal for the winner is a finding, not an embarrassment.

TakeawayFair timing cut the recommended system's measured failure rate roughly in half (15.4% to 8.2%); the remaining gap against the challenger (8.2% vs 0.5%) is real and is reported even though it argues against the winner.

Retrospective: what went well

The habits worth keeping, each tied to a concrete moment in this evaluation.

Checked, not defended, and not over-corrected. When the stakeholder said a scoring rule was unfair, the answer was a code-level verification that the rule was already correct, stated plainly on the results page. Silently "fixing" something that is not broken would have been easier and wrong twice over.
The instrument was audited before the measurements were believed. The automated grader turned out to be broken in two places, both of which would have gone unnoticed under "trust the script." No number was reported until the grader agreed with human judgment at 95% or better on every measure.
An inconvenient finding was promoted, not buried. The winning system carries a real reliability weakness: a canned fallback roughly 1 turn in 12. It appears in the headline section of the results, with the conditions under which it should reverse the decision spelled out.

Retrospective: what didn't go well

The honest half of the ledger: what a stronger process would have prevented.

The first round wasted time on basic execution bugs. Replies were being truncated by the display before they were scored, among other mechanical errors a simple smoke test would have caught before any full run.
The original design had no repeats and no timing controls. One measurement per test case cannot distinguish a real difference from noise, and uncontrolled ordering let a start-up penalty masquerade as a quality gap. Both are standard experimental hygiene, and both were added only after the stakeholder pushed back. They should have been in the design from day one, which is the direct source of the recommendations below.

Forward-looking recommendations

Action items adapted for a template rather than a team: what becomes default practice for every future evaluation.

Repeats are the default, not a retrofit. Every future comparison runs multiple repeats per system and reports a range, never a single point. A one-shot measurement cannot tell signal from noise, so it should not be possible to run one by accident.
Timing-bias controls ship in the harness template. Warm-up passes and alternated run order become standard scaffolding for any latency-sensitive comparison, not a fix applied after a stakeholder notices the confound.
The instrument audit is a standing gate. No automated grading system's numbers get reported until a human hand-label check confirms 95%+ agreement, with a fixed iteration cap and the commitment to report a failure to converge rather than hide it.

TakeawayTwo of the three stakeholder concerns pointed at real problems, and the third, once checked in the open, made the whole report easier to believe. Distrust taken seriously is how a measurement earns trust.