Gemini climbed 25 points, and still gave different answers to the same question.
Abstract. Anthropic released Claude Fable 5.2 and Google released Gemini 4 Argon, so all four vendors sat the same 25 CPA-adjudicated questions again: closed book, maximum reasoning, three fresh runs each, every response logged. Claude answered 75 of 75 for the second round running. GPT-6 Astra got nothing wrong, but on one memo question it thought for longer than our 3-minute limit on all three runs. Gemini 4 Argon jumped from 56% to 81%, yet on three questions it knew the 2025 tax law on one run and not on another. Grok 4.7, unchanged, still says the 2025 law never passed, and failed to answer within 3 minutes on 11 of 75 attempts.
| Model | Current law | Court holdings | Caught-error set | Detection memos | Confident-wrong | Flips | Sep 2026 |
|---|---|---|---|---|---|---|---|
| Claude claude-fable-5-2 | 30/30 | 15/15 | 15/15 | 15/15 | 0 | 0 | 100.0% claude-fable-5-1 |
| ChatGPT gpt-6-astra | 30/30 | 15/15 | 15/15 | 12/12 | 0 | 0 | 97.3% gpt-6-astra |
| Gemini gemini-4-argon | 16/30 | 15/15 | 15/15 | 15/15 | 5 | 3 | 56.0% gemini-3.1-pro-preview |
| Grok grok-4.7 | 14/29 | 15/15 | 6/6 | 14/14 | 7 | 1 | 75.0% grok-4.7 |
Claude and GPT calls ran November 9 to 10, Gemini on the morning of November 10 (UTC), Grok later that day, each via the provider's API, no browsing, no tools. Accuracy is correct answers divided by answers returned within 3 minutes. GPT's 3 non-answers (D4 on all three runs) and Grok's 11 (C1, C2 and C5 on all three runs, one run each of A8 and D1) are excluded from the denominator and reported in section 03; counting them as wrong, GPT scores 72/75 (96.0%) and Grok 49/75 (65.3%). A flip means the substantive answer changed between identical runs. The last column is the same vendor in round 2.
The new Gemini knew the 2025 tax law on two runs out of three.
Question A4 / published in round 1, asked again / all responses in the exhibit file
For a taxpayer who is not married filing separately, state the 2026 SALT deduction limitation, the MAGI threshold at which it begins to decrease, and the minimum limitation after the phase-down.
KEY: $40,400; phase-down begins at $505,000 MAGI; floor $10,000.
The key has been public since August. Grok still says the law expired. Gemini says it twice.
Question A7 / published in round 1, asked again / all responses in the exhibit file
Does Section 199A continue to apply to taxable years beginning in 2026, and what percentage is used in its general deduction formula?
KEY: Yes; 20%. P.L. 119-21 (2025) repealed the scheduled sunset.
C3: all four models now cite the right subparagraph.
Question C3 / published in round 1, asked again / all responses in the exhibit file
Which subparagraph of IRC Section 642(b)(2) provides the $100 deduction for personal exemption allowed to a complex trust?
KEY: subparagraph (A), the $100 general rule. (B) is the $300 rule for trusts required to distribute all income currently.
| Vendor | Round 2 model | Round 3 model | Released | Round 2 | Round 3 |
|---|---|---|---|---|---|
| Anthropic | claude-fable-5-1 | claude-fable-5-2 | Nov 9 | 100.0% | 100.0% |
| OpenAI | gpt-6-astra | same | no release | 97.3% | 100.0% |
| gemini-3.1-pro-preview | gemini-4-argon | Sep 30 | 56.0% | 81.3% | |
| xAI | grok-4.7 | same | no release | 75.0% | 76.6% |
Each vendor is represented by the model its own documentation presents as its most capable on the run date. Anthropic's release of Fable 5.2 on November 9 triggered this round; Google had also replaced Gemini 3.1 Pro with Gemini 4 Argon since September. OpenAI's and xAI's top models did not change, which makes their rows a reproducibility check. Grok reproduced itself closely: the same five current-law misses (plus one run of A2), the same two unfinished questions. GPT-6 Astra improved on the same model: its two September misses (C3) are gone. Next round: Google released Gemini 4 Carbon on November 10, the day of this round's Gemini calls, and describes it as its strongest model. That makes Carbon Google's entry in round 4.
OCR, document extraction, coding, and retrieval-equipped deployments. Models are good at those, and vendors benchmark them constantly. This report is about the chat-box question a partner actually fears: typed question, confident answer, no warning.
Methodology in 60 seconds
25 items, four buckets: current enacted law (10), disguised Supreme Court holdings (5), an archive of errors we have documented frontier models making in our own production logs (5), and staff memos containing exactly one planted defect (5). Every key adjudicated by a licensed CPA against the named primary authority before any model saw anything. The item files, the prompt, and the key are byte-identical to rounds 1 and 2; the hashes below are the August hashes.
Three runs per model per item, each in a fresh conversation, maximum reasoning, no tools, graded independently, no majority voting, no best-of. Binary grading, no partial credit. A call that returns no answer within 3 minutes is recorded with its error text or timing, left out of the accuracy figure and reported separately. A refusal is graded wrong, because the model did respond.
Disclosed, because the protocol is the point: unlike round 2, no separate pre-registration file was written for this round before the first call. Nothing in it would have changed: the items, prompt, key and grading rules are the round 2 files, unchanged, and the no-answer rule is the one published with round 2. The one judgment made after the run was how to treat the 10-minute runner limit described in section 03; we chose the published 3-minute rule over the more generous one.
Our interest, disclosed: we sell a CPA test bank that AI writes and a licensed CPA verifies. If models were useless our bank would be worthless; if models were flawless our verification layer would be pointless. We profit only in the middle.
Commitment hashes (SHA-256), unchanged from round 1
Frozen core (15 items):7EAD6FAC8C47D54AE041AA71DDFDAD5DB7A71D17858F8620961F09D8119E3147
Current-law tranche (10 items):629A78030C2B744E1D15B9C5E63A860697A28B0386D87A0A53E212D6E4FAABDD
Answer key with grading rules:7A38224F08212276DF6ED35827956A86E734BE89DA618F3B31B348AC4E65F5DA
Every one of the 300 calls: model, snapshot, run number, timestamp, self-reported confidence, verdict, citation verdict, adjudication note, and the 14 non-answers with their timing or error text. Response text is included for the three published items (A4, A7, C3, public since August) and withheld for the other 22. No new question is published this round.
WAVE3-LOG-PUBLIC.CSV WAVE3-EXHIBITS-FULL.CSVwave3-log-public.csv SHA-256: {LOGHASH}
wave3-exhibits-full.csv SHA-256: {EXHASH}
Run by Nicholas Miller, CPA (Oregon #14907). Next round: Gemini 4 Carbon, released November 10, with the other three vendors' top models at that date, same 25 questions, same protocol. Standings across all rounds, and every published question with its key so you can try them in your own chatbot, are on the series page. Corrections to this report, including ones that embarrass us, are posted here, dated, with the original text preserved. Corrections log: none to date.