The same 25 questions. Every frontier release.
Whenever OpenAI, Anthropic, xAI, or Google ships a new frontier model, we retest all four on the same 25 CPA questions on tax, GAAP, and audit. Each model gets three fresh attempts per question, 75 per round, and a licensed CPA grades every returned answer. In November, Claude scored 75 of 75 for the second round running. GPT-6 Astra got nothing wrong, but took longer than 3 minutes on all three runs of one memo question. Gemini 4 Argon, new since September, rose from 56% to 81%, yet on three current-law questions it knew the 2025 tax law on one run and not on another. Grok, unchanged, still answers as if the 2025 law never passed, and returned 64 of 75 attempts in time: 49 right, 77% of returned answers, 65% of attempts.
One page · PDFThe results on a single pageStandings, the key finding, the three public questions with keys. For a class or a partner meeting.DOWNLOAD ↓| Vendor | All rounds | Nov 2026 → | Sep 2026 → | Aug 2026 → | Confidently wrong all rounds |
|---|---|---|---|---|---|
| OpenAIChatGPT | 99.1% | 100.0%+2.7 · 72/72 · gpt-6-astra | 97.3%gpt-6-astra | 100.0%gpt-5.6-sol | 0.9% |
| AnthropicClaude | 97.8% | 100.0%±0 · claude-fable-5-2 | 100.0%claude-fable-5-1 | 93.3%claude-fable-5 | 0.0% |
| xAIGrok | 73.7% | 76.6%+1.6 · 49/64 · grok-4.7 | 75.0%grok-4.7 | 69.9%grok-4.6 | 11.7% |
| GoogleGemini | 64.4% | 81.3%+25.3 · gemini-4-argon | 56.0%gemini-3.1-pro-preview | 56.0%gemini-3.1-pro-preview | 31.1% |
Accuracy is correct answers divided by returned answers, out of 75 attempts per round. Refusals count as wrong, because the model responded. A call that returns no answer (a timeout or API error) is not graded: it is left out of the figure and reported separately. In November Grok returned 64 of 75 in time (49 right) and GPT 72 of 75 (72 right); Grok's all-rounds figure is 151 of 205. All-rounds GPT is 220 of 222, Claude 220 of 225, Gemini 145 of 225.
All rounds follows the vendor across model versions, not one model over time; the model that sat each month is named in the cell. Confidently wrong is the share of returned answers that were wrong and labeled High confidence. Rows are ordered by all-rounds accuracy. We sell a CPA-verified test bank; method, hashes, and conflict of interest are in section 06.
One email when a new round is published: the newest models, the same 25 questions. Nothing else.
A current enacted law, 2026 figures. B Supreme Court holdings with the facts disguised. C known traps: errors we have seen frontier models make in production. D staff memos with exactly one planted defect.
Read down a column for a model, across a row for a question. A1 to A8 are the law that changed in 2025 (A9 and A10 are GAAP items): Grok has missed most of them in all three rounds; Google's new model gets several right on some runs and wrong on others. C3 is a one-letter citation three different models have gotten wrong; in November all four got it right. B and D are nearly solid black: models reason well about facts they are given and fail on facts that changed after they were trained.
Ranked by cost per correct answer, not by accuracy. Same round as the standings. This round the cheapest right answer came from Gemini, now in the second tier rather than the bottom.
| Model | Correct / answered | Spend, calls returned | Output tokens per call | Cost per correct answer |
|---|---|---|---|---|
| Geminigemini-4-argon | 61 / 75 | $1.42 | 1,841 | $0.023 |
| Grokgrok-4.7 | 49 / 64 | $1.96 | 4,121 | $0.040 |
| Claudeclaude-fable-5-2 | 75 / 75 | $4.85 | 1,553 | $0.065 |
| ChatGPTgpt-6-astra | 72 / 72 | $8.70 | 2,277 | $0.121 |
Token counts from each provider's own API response, priced at list rates on the day of the run (Claude $8 in / $40 out per million; GPT $10 / $50; Gemini 4 Argon $2 / $10 introductory; Grok $2 / $6). Reasoning tokens are billed as output by all four and are included. Spend covers every call that returned, including the 10 answers that arrived after 3 minutes and are not graded (GPT 75 calls, Grok 71); Grok's 4 calls that never returned are not in it. Claude's price fell 20% with Fable 5.2 and it used fewer output tokens per call, so its cost per right answer fell from $0.093.
Round 3 · November 10, 2026 · claude-fable-5-2, gpt-6-astra, gemini-4-argon, grok-4.7
Gemini climbed 25 points, and still gave different answers to the same question.
Claude Fable 5.2 answered 75 of 75 for the second round running. GPT-6 Astra got nothing wrong but needed more than 3 minutes on all three runs of one memo question. Gemini 4 Argon rose from 56% to 81%, yet on three current-law questions it knew the 2025 tax law on one run and not on another. Grok 4.7 repeated its September misses and again could not finish two questions. Next round: Gemini 4 Carbon, released November 10.
READ ROUND 3 →Round 2 · September 22, 2026 · gpt-6-astra, claude-fable-5-1, grok-4.7, gemini-3.1-pro-preview
Three new frontier models, one month later. Same 25 questions.
Claude Fable 5.1 answered 75 of 75. GPT-6 Astra lost its perfect score to a single one-letter citation, and on the second miss fabricated a quote from the tax code: 2002 wording plus a clause that never existed, presented as current law. Grok 4.7 matched 4.6's 51 correct, timed out on seven attempts, and never finished two of the questions. Gemini, unchanged, reproduced August exactly: same score, same eleven misses.
READ ROUND 2 →Round 1 · August 22, 2026 · gpt-5.6-sol, claude-fable-5, grok-4.6, gemini-3.1-pro-preview
GPT-5.6 went 75 for 75. Grok and Gemini sat the 2026 exam with 2024 law.
GPT-5.6 answered 75 of 75. Claude made zero wrong claims but refused five times. Grok and Gemini answered as if the 2025 tax law had never passed, and 72% of all wrong answers carried high confidence. The round that set the protocol: hashes, three runs, binary grading, refusals and timeouts reported on their own.
READ ROUND 1 →Teaching this? Download the one-page instructor brief (PDF, rounds 1 to 3): standings, the key finding, these three questions with keys, and discussion prompts. Free to use in class.
Three of the 25 have been published in full, with keys, and are still asked every round. Paste one into any chatbot and compare. If it gets A7 or A4 wrong, it is answering from before the 2025 law. Ask more than once: in November one model got both right on some runs and wrong on others. C3 is a pinpoint citation nobody memorizes; it is here because models misplace or invent citations with confidence. ("Complex trust" in C3 means a trust not required to distribute all income currently; citing (A) and noting the (B) exception for trusts that are grades correct.) Every model's verbatim responses to these are in the exhibit files, linked in section 06.
Does Section 199A continue to apply to taxable years beginning in 2026, and what percentage is used in its general deduction formula?
KEY: Yes; 20%. P.L. 119-21 (2025) repealed the scheduled sunset.
For a taxpayer who is not married filing separately, state the 2026 SALT deduction limitation, the MAGI threshold at which it begins to decrease, and the minimum limitation after the phase-down.
KEY: $40,400; phase-down begins at $505,000 MAGI; floor $10,000.
Which subparagraph of IRC Section 642(b)(2) provides the $100 deduction for personal exemption allowed to a complex trust?
KEY: Subparagraph (A), the $100 general rule. (B) is the $300 rule for trusts required to distribute all income currently.
Method, in 60 seconds
25 items, four buckets: current enacted law (10), disguised Supreme Court holdings (5), known traps from our production logs (5), and staff memos containing exactly one planted defect (5). Every key adjudicated by a licensed CPA against the named primary authority before any model saw anything.
The questions stay private because published questions leak into training data and the benchmark dies. Three (A4, A7, C3) are published on purpose, as contamination controls: they stay in the run, and if models start getting these right while still missing the private ones, that is the leak showing. The item set was hash-committed before the first run; the fingerprints below prove it existed, unchanged, before any model answered. Three runs per model per item, each in a fresh conversation, no tools, no majority voting, no best-of, binary grading. A refusal grades as incorrect and is reported separately. A call that returns no answer (a timeout or API error) is not graded; it is left out of the accuracy figure and reported separately. Each round's questions and rules are written down before the run; round 3 reused round 2's files unchanged and did not add a separate pre-registration, which the round 3 page discloses.
Our interest, disclosed: we sell a CPA test bank that AI writes and a licensed CPA verifies. If models were useless our bank would be worthless; if models were flawless our verification layer would be pointless. We profit only in the middle, so our incentives cut both ways.
The data: every call, every response
Every call in every round: model, snapshot, run, timestamp, answer, confidence and verdict. The exhibit files hold the full verbatim responses to the published questions. Check any file against its SHA-256.
Round 3 (Nov 2026)wave3-log-public.csvSHA-256 4CFD6A159128DD402A5E924A85BF1EB47D8EE04E287347B55F55F7D409BB7437wave3-exhibits-full.csvSHA-256 373D9D96BEF7A757482DB711C44A47B8E7E21E561999E0B95AA978DC407F6C65
Round 2 (Sep 2026)wave2-log-public.csvSHA-256 457E3D6F58E02A9C9312F70D7DA38EBB06BF50F141037951155449753996CCA7wave2-exhibits-full.csvSHA-256 634C28012A92290F9257271ED578DDFF0FC1191EF08CAA5E3DA23FC83C3881FA
Round 1 (Aug 2026)wave1-log-public.csvSHA-256 B2B9AB10480D238CDD891B41ECE6388BCA4FEF3689EDCCE1E8F0AAF9F8614434wave1-exhibits-full.csvSHA-256 0A39F02EAFFD3F122956D4D0AE893144B92C05FD199204D40A6A1DE0C5E10D66
Commitment hashes (SHA-256)
Frozen core (15 items):7EAD6FAC8C47D54AE041AA71DDFDAD5DB7A71D17858F8620961F09D8119E3147
Current-law tranche (10 items):629A78030C2B744E1D15B9C5E63A860697A28B0386D87A0A53E212D6E4FAABDD
Answer key with grading rules:7A38224F08212276DF6ED35827956A86E734BE89DA618F3B31B348AC4E65F5DA
Run by Nicholas Miller, CPA (Oregon #14907). The next round adds Gemini 4 Carbon, released November 10, and runs on the same 25 questions. Get it by email. Corrections, including ones that embarrass us, are posted on the round pages, dated, with the original text preserved.

