Three new frontier models, one month later. Same 25 questions.
Abstract. In August, four models answered 25 CPA-adjudicated questions on tax judgment, GAAP interpretation, and audit reasoning. In September, three of the four vendors shipped a successor model. We asked the successors the identical 25 questions under the identical protocol: closed book, maximum reasoning, three fresh runs each, every response logged. One model answered 75 of 75. The August leader lost its perfect score to a single one-letter citation. The one unchanged model reproduced its August result exactly, down to the same eleven misses. And one model could not finish two of the questions, or one run of a third, in nine minutes of trying.
| Model | Current law | Court holdings | Caught-error set | Detection memos | Confident-wrong | Flips | Aug 2026 |
|---|---|---|---|---|---|---|---|
| Claude claude-fable-5-1 | 30/30 | 15/15 | 15/15 | 15/15 | 0 | 0 | 93.3% claude-fable-5 |
| ChatGPT gpt-6-astra | 30/30 | 15/15 | 13/15 | 15/15 | 2 | 1 | 100.0% gpt-5.6-sol |
| Grok grok-4.7 | 15/30 | 15/15 | 6/8 | 15/15 | 7 | 0 | 69.9% grok-4.6 |
| Gemini gemini-3.1-pro-preview | 6/30 | 15/15 | 6/15 | 15/15 | 32 | 0 | 56.0% gemini-3.1-pro-preview |
All runs September 22, 2026, via each provider's API, no browsing, no tools. Grok 4.7 was released the day before. Its 7 non-answers (C2 and C5 on all three runs and C1 on one, three attempts each at a 3-minute limit, repeated on a second pass with the same result) are excluded from its denominator and reported in section 04; counting them as wrong, Grok scores 51/75, or 68.0%. A flip means the substantive answer changed between identical runs. The last column is the same vendor's model in round 1. Reading guide, one line: the Confident-wrong column is still the one that matters.
The market leader cited the wrong subparagraph, twice, and the second time fabricated a quote from the tax code.
Question C3 / published in round 1, asked again / all responses in the exhibit file
Which subparagraph of IRC Section 642(b)(2) provides the $100 deduction for personal exemption allowed to a complex trust?
KEY: subparagraph (A), the $100 general rule. (B) is the $300 rule for trusts required to distribute all income currently.
Nobody memorizes pinpoint subparagraphs; C3 is here because models misplace or invent citations with confidence, and that is a real risk in a client memo. Grading note: "complex trust" here means a trust not required to distribute all income currently. Treas. Reg. §1.642(b)-1 gives the $300 amount under (B) to any trust required to distribute all income currently, simple or not; an answer citing (A) and noting that exception grades correct. Every miss so far swapped the two subparagraphs outright.
Nine minutes, twice, and no answer.
Question C2 / private / a §1231(c) lookback ordered against unrecaptured §1250 gain, one dollar answer
The stem stays private. It is a five-line fact pattern from the caught-error archive with a single numeric answer, adjudicated to Notice 97-59.
The key has been public since August. Two models still say the law expired.
Question A7 / published in round 1, asked again / all responses in the exhibit file
Does Section 199A continue to apply to taxable years beginning in 2026, and what percentage is used in its general deduction formula?
KEY: Yes; 20%. P.L. 119-21 (2025) repealed the scheduled sunset.
| Vendor | Round 1 model | Round 2 model | Released | Round 1 | Round 2 |
|---|---|---|---|---|---|
| Anthropic | claude-fable-5 | claude-fable-5-1 | Sep 1 | 93.3% | 100.0% |
| OpenAI | gpt-5.6-sol | gpt-6-astra | Sep 3 | 100.0% | 97.3% |
| xAI | grok-4.6 | grok-4.7 | Sep 21 | 69.9% | 75.0% |
| gemini-3.1-pro-preview | same | no release | 56.0% | 56.0% |
Four questions were written down before the run, in a file published with the results. The answers: Anthropic's update fixed every August miss; all five were refusals, and there were none this time. OpenAI's update is not a downgrade on substance, but it is no longer perfect, and its two misses are the same one-letter citation with a fabricated quotation on the second. xAI's update did not leave the second tier: 51 correct in August, 51 in September, and it now times out where 4.6 answered wrong. Google, unchanged, reproduced August exactly: 42 of 75, the same eleven items, every one of them a matter of law that changed in 2025. That last row is the reproducibility check for the whole method. Three of the eleven were published in full in August with their keys. Publishing them changed nothing.
OCR, document extraction, coding, and retrieval-equipped deployments. Models are good at those, and vendors benchmark them constantly. This report is about the chat-box question a partner actually fears: typed question, confident answer, no warning.
Methodology in 60 seconds
25 items, four buckets: current enacted law (10), disguised Supreme Court holdings (5), an archive of errors we have documented frontier models making in our own production logs (5), and staff memos containing exactly one planted defect (5). Every key adjudicated by a licensed CPA against the named primary authority before any model saw anything. The item files, the prompt, and the key are byte-identical to round 1; the hashes below are the August hashes.
Three runs per model per item, each in a fresh conversation, maximum reasoning, no tools, graded independently, no majority voting, no best-of. Binary grading, no partial credit. A call that returns no answer (an API error or a timeout) is recorded with its error text, left out of the accuracy figure and reported separately. A refusal is graded wrong, because the model did respond; that rule, and the four questions this round was designed to answer, were written into a pre-registration file before the first call. Any model answer that disputed our key went back to the primary source before a verdict was entered.
Our interest, disclosed: we sell a CPA test bank that AI writes and a licensed CPA verifies. If models were useless our bank would be worthless; if models were flawless our verification layer would be pointless. We profit only in the middle. This round we published a perfect score by one vendor and a fabricated statutory quotation by another. That is what the middle looks like.
Commitment hashes (SHA-256), unchanged from round 1
Frozen core (15 items):7EAD6FAC8C47D54AE041AA71DDFDAD5DB7A71D17858F8620961F09D8119E3147
Current-law tranche (10 items):629A78030C2B744E1D15B9C5E63A860697A28B0386D87A0A53E212D6E4FAABDD
Answer key with grading rules:7A38224F08212276DF6ED35827956A86E734BE89DA618F3B31B348AC4E65F5DA
Round 2 pre-registration (four questions, no-answer rule, written 22 Sep 2026 before the first call):F6931A2D34699466963B0D5D308DA9EC32778D72BB5723245098927AB7056BDC
Every one of the 300 calls: model, snapshot, run number, timestamp, self-reported confidence, verdict, citation verdict, adjudication note, and the 7 non-answers with their error text. Response text is included for the three published items (A4, A7, C3, public since August) and withheld for the other 22. No new question is published this round.
WAVE2-LOG-PUBLIC.CSV WAVE2-EXHIBITS-FULL.CSVwave2-log-public.csv SHA-256: 457E3D6F58E02A9C9312F70D7DA38EBB06BF50F141037951155449753996CCA7
wave2-exhibits-full.csv SHA-256: 634C28012A92290F9257271ED578DDFF0FC1191EF08CAA5E3DA23FC83C3881FA
Run by Nicholas Miller, CPA (Oregon #14907). Next round: within two weeks of the next frontier release, same frozen core, same protocol. Standings across all rounds, and every published question with its key so you can try them in your own chatbot, are on the series page. Corrections to this report, including ones that embarrass us, are posted here, dated, with the original text preserved. Corrections log: none to date.