Experiments
New experimentAccuracy with confidence intervals
each bar is one bundle · whiskers are the 95% intervalThis replaces a trend line on purpose. With five runs the interval is roughly ±2 points, so 0.10.0 and 0.11.0 sit inside the baseline band and mean nothing — a line chart would have drawn them as progress. Only 0.12.0 clears the band, and it needed 12 runs to get there.
Graded against sampled
the lab number is consistently optimisticThe gap runs 1.2 to 1.7 points and hasn’t closed. Live files are messier than the corpus, so a graded gain should be discounted before anyone promises it to a stakeholder. If this gap ever widened sharply it would mean the corpus had drifted away from production.
All experiments
append-only| bundle | hypothesis | accuracy | runs | cost | separates | verdict |
|---|---|---|---|---|---|---|
| 0.12.0 | step-7 title review; checker sees obligation lines | 96.4% ±1.2 | 12 | $26.28 | yes, above | keep |
| 0.11.1 | same, first build | — | 2 | $3.10 | — | crash |
| 0.11.0 | Opus 4.8 to Opus 5, reasoning effort xhigh | 93.8% ±1.9 | 5 | $10.60 | no, overlaps | inconclusive |
| 0.10.1 | drop page citations from document-reader | 88.9% ±2.4 | 5 | $11.40 | yes, below | discard |
| 0.10.0 | split step-4 into obligations and qualifying payment | 93.1% ±2.1 | 5 | $11.55 | no, overlaps | inconclusive |
| 0.9.2 | HELOC File Review v1, Opus 4.8 | 92.4% ±2.0 | 5 | $12.25 | — | baseline |
Field failures across bundles
out of 150 fields per run| field | 0.9.2 | 0.10.0 | 0.11.0 | 0.12.0 | Δ | reading |
|---|---|---|---|---|---|---|
| DebtToIncomeRatio | 8 | 7 | 6 | 1 | -7 | fixed by giving the checker obligation lines |
| InvestmentAccounts | 3 | 3 | 3 | 3 | — | flat — a missing-document problem, not reasoning |
| HousingHistory | 2 | 2 | 1 | 0 | -2 | resolved incidentally |
| FloodZone | 1 | 1 | 1 | 1 | — | single ambiguous parcel in the corpus |
InvestmentAccounts hasn’t moved across four bundles because no bundle change can fix it — the statement isn’t in the folder. That row is an intake problem wearing an accuracy costume, and it’s the clearest case for spending the next attempt somewhere else.
Run one
Estimated $26.28. At 12 runs the interval lands near ±1.2 and can separate from baseline.
What this doesn’t tell you
Every accuracy figure on this page is graded against a labelled corpus, which means it is a lab measurement and optimistic by the 1.2 to 1.7 points the gap chart shows. None of it is evidence about production; that only comes from blind review.
Four of the six bundles rest on five runs, which buys an interval near ±2 — wide enough that a real two-point improvement would still read inconclusive. Reading those rows as “no effect” is as wrong as reading them as progress. They mean the experiment was too small to say.