A read on the chess coach's answer speed and reliability, in plain terms. Each chart makes one point and nothing else. Numbers are computed from the eval runs and your real games; sample sizes are shown on every chart.
The slow answers aren't the network or our code: the assistant is silently "thinking" before it replies.
On hard questions it spends 20-50 seconds reasoning privately before writing the first word. A 3-minute budget alone fixed the timeouts: retries now rescue every stalled answer (10 of 10), proving retrying was never broken, just starved. Capping the thinking on top makes answers ~3× faster, but costs correctness (6.2% of first drafts failed fact-checks vs 0%), so the verified-facts work should ship before or alongside it. One subjective quality read stays sealed.
Superseded · noted 2026-08-26
This page ends on an open question, and that question has since been answered. The sealed quality read closed the next day: attempt-0 = low shipped to every intent, postgame review included, with no mode gate. See the thinking-arm dashboard for the three-repeat run that decided it. The measurements below are unaffected; only the "still sealed" framing is out of date.
CHART 1
Where a slow answer's time actually goes
One fully-measured 20-second answer, broken into its four stages. What is the reader waiting on?
TAKEAWAY
The invisible thinking is the entire tail. Starting up, reading the board, and writing the reply take about 4 seconds combined; the private thinking takes 16. On the hardest questions that one block stretches to 20-50 seconds while the other three stay flat.
Source: per-event timeline of one instrumented answer, corroborated by 26 controlled backend calls on your real questions (root-cause investigation, 2026-08-02).
CHART 2
How often she gets an answer vs. an apology
Every question sorted into one of three outcomes, across four versions. The bars are share-of-questions, so they always add to 100%.
TAKEAWAY
Give-ups roughly doubled since July, and the change is failures, not slowness. The share of fast answers fell and the "gave up" slice grew from ~8% to ~18% in testing and ~23% in your real play. The answers that used to arrive slowly now arrive not at all.
⚠ One bar carries a caveat. The pre-fix test ran on a machine that became busy partway through; only its board questions (shown here) are usable, and its give-up rate is directional, not certified. July and after-wave-1 are clean.
Testing bars: 65 board questions each (July = average of 3 repeats). Your real play: 91 chat questions from the wave-1 playtest save.
CHART 3
Do retries actually rescue answers?
You said "I've never seen the regen work." When an answer runs long, the system retries once. Here is what those retries produced: at the old 45-second limit, and at the new 3-minute budget.
TAKEAWAY
Retrying was never broken: the 45-second limit starved it. Give a retry room to finish and, at normal thinking, every stalled answer was rescued (10 of 10), up from 9 of 21 at the old limit. The regen you never saw work now works.
SURPRISE
At a 3-minute budget even normal thinking had zero give-ups. Raising the budget alone erased the timeout failures, so the thinking limit is a speed lever, not a rescue lever. (The two bounded arms show 4 give-ups each, but those are the fact-check failures of Chart 5, not timeouts.)
Source: the three overnight 3-minute runs (65 questions each) vs the 45-second baseline. Retries counted as regens that fired; "rescued" = ended as a real answer, "starved" = ended as an apology.
CHART 4
Did our last round make it worse?
An honest before/after on the wave-1 change. Three speed measures, plus the one thing that genuinely changed.
TAKEAWAY
Wave-1 changed what you see, not how fast the answer arrives. Give-ups, slow answers, and typical wait were unchanged. What it added: an instant "thinking" signal (visible in a fraction of a second) so the screen no longer looks frozen. The real regression came earlier, a 22.5-second cut-off added on 08-01 that turned slow-but-delivered answers into guaranteed give-ups.
Source: the two 65-question runs before and after wave-1+caching. "Instant signal" = time to first on-screen response; "the answer" = time to the first real word.
RESULTS ARE INThe decisive test finished overnight. Three versions of the coach: normal, limited, and off thinking, each ran the full 65 questions at a 3-minute budget. Speed and fact-check failures below are now real, measured data. One piece remains sealed: the subjective blinded read of whether the faster answers still teach well.
CHART 5
What the fix costs
Capping the thinking makes answers faster, but does a faster answer become a wrong answer? Both speed and fact-check failures are now measured.
TAKEAWAY
The speed win is about 3× at the median, and it tames the 64-second tail to 15, but it has a real cost. Bounding the thinking made roughly 1 in 16 first drafts (6.2%) fail the coach's fact-checks and turn into apologies, versus none at normal thinking. That argues for shipping the verified-facts work before or alongside the thinking limit, so the faster answers don't trade speed for being wrong.
STILL SEALED
The subjective, blinded quality read (is a limited-thinking answer actually as helpful to learn from?) is built but its answer key is unopened. Until it is read, "cap the thinking" is a strong speed lead, not a shipping decision.
Source: the three overnight 3-minute runs, 65 questions each. Speed = wall time to the answer; fact-check failures = first drafts rejected by the coach's own validators. Subjective grade: blinded worksheet built, key sealed.
IN HER WORDS
What the failures felt like
Verbatim feedback attached to give-ups and slow answers in the wave-1 playtest, the human side of Chart 2.
Source: your thumbs-down notes in the wave-1 playtest save (advice_traces).