AI in Practice
What 30 AI models got wrong on ordinary business work
I ran 324 tests across inbox triage, browser invoice entry and a month-end close. The results changed how I think businesses should test and deploy models.
I ran 324 tests across inbox triage, browser invoice entry and a month-end close. The results changed how I think businesses should test and deploy models.
A supplier invoice listed three unit prices. In capital letters, it said every price included GST.
The accounts system asked for prices excluding GST.
Most models read both instructions, completed the form and saved the invoice. The supplier was right. The dates were right. The descriptions and tax codes looked right.
The unit prices were wrong.
Of the 80 runs that saved this invoice, 72 copied all three GST-inclusive prices straight into the ex-GST fields. Most also selected the GST tax code, so the system would add GST again.
The finished record looked normal. That is what made it dangerous.
Businesses ask models to do ordinary work inside existing processes, where a plausible mistake can pass unnoticed. That is what I wanted to test.
I tested 30 model configurations from OpenAI, Anthropic, Google, xAI, DeepSeek, Qwen, Kimi, GLM, MiniMax, Mistral, Meta and Cohere. Eighteen were current models. Twelve older releases joined the inbox and invoice tasks so I could compare generations.
The original benchmark produced 324 runs. Some failures were obvious. The harder problem is deciding whether a plausible result should be trusted.
The parts of this study I found interesting
Models did not follow the written policy
The inbox task included an email with URGENT in the subject. The sender wanted to know whether a brochure came in blue and said there was no deadline.
The supplied policy made that normal-priority work. It was useful, but it was not urgent.
Eighty-five of 90 runs marked it low. Four got it right. One returned no answer.
Most runs marked the email low priority instead of following the supplied rule that useful routine work was normal. The outputs still commonly recommended answering it, so the result shows policy noncompliance rather than why the models made the choice.
This happened across the strongest results. Five model configurations stayed above 98% in all three inbox runs. All five missed this rule at least once. Across their 15 runs, 14 marked the email low.
A good overall score can hide a decision your business cares about. That decision needs its own reporting and its own check.
A good demo tells you very little about reliability
DeepSeek V4 Flash scored 100% on its first month-end close. Then it scored 100% again.
The third run scored 70%.
If I had stopped after either of the first two runs, I could have produced a very convincing demo. The third run changed the deployment decision.
AI is non-deterministic. Three repeats were enough to expose this particular failure, but they are nowhere near enough to establish a production failure rate.
The right validation set depends on the work. A low-volume process may need dozens of reviewed examples. A business processing hundreds of items each day can run hundreds or thousands under supervision and compare the results with real outcomes.
Until the evidence is there, it is still a trial.
Model choice needs to follow the job
Sonnet 5 was the only model to enter all 18 invoice fields correctly in all six browser runs.
Opus 5 produced three perfect month-end close packs. It also refused all six browser invoices through content filtering.
The inbox produced another group of leaders. Five models stayed above 98% across all three runs, while 17 of the 30 fell below 95% at least once.
We normally hire a person who can handle the hardest part of a broad role. That person also spends time on tasks that barely use their experience.
Models let us split a role into smaller jobs. Choose the configuration that clears each job and its controls. Compare cost only after the result is acceptable, then keep the tested configuration in place.
How I tested them
I chose three synthetic but realistic jobs for an Australian business:
- Triage 20 messages in a shared inbox.
- Enter two supplier invoices through a browser into a small accounts system.
- Prepare a month-end close pack from an unadjusted trial balance and seven close notes.
Every model received the same prompt and input for its task. Ground truth was written before the official runs. Scoring was mechanical. Models used provider-default reasoning through OpenRouter, with one attempt per run. Transport failures could be retried. Wrong, malformed or empty model responses stayed in the results.
Each cell ran three times. The invoice task had two cells, one for each document, so each model received six official invoice runs.
I changed how I grade the result
The original research thresholds were too permissive for a business deployment decision. Inbox runs cleared at 90%. Close runs needed 90% and could not omit the duplicate-invoice or bank-reconciliation controls. An invoice cleared at 17/18 fields, or 94.4%.
Those original values remain in the raw records. I have not changed the data or rescored an answer to make a point.
I changed the publication grade after looking at what the old word pass allowed through:
- Good: every official run for the task scored at least 98%.
- Workable: every run scored at least 95%, with at least one below 98%.
- Reject: at least one run scored below 95%.
The worst run sets the grade. An average cannot hide a bad repeat.
| Task | Good | Workable | Reject |
|---|---|---|---|
| Inbox triage | 5 | 8 | 17 |
| Invoice entry | 1 | 0 | 29 |
| Month-end close | 3 | 0 | 15 |
Even a Good grade is only a starting point for more testing. It is not a production guarantee.
Task one: triage the inbox
The inbox belonged to a fictional 45-person building-services consultancy. Its 20 messages covered sales, customer support, accounts payable, operations, HR, security and spam.
Each message had three checks: category, priority and next action. That made 60 checks per run.
The examples included a supplier requesting new bank details before a payment run, two phishing emails, an exposed electrical cable and a warehouse outage.
Five models earned a Good grade:
| Model | Worst run | Mean accuracy | Mean time |
|---|---|---|---|
| GPT-5.6 Terra | 98.3% | 98.3% | 5.9s |
| Qwen3.7-Max | 98.3% | 98.3% | 65.1s |
| Qwen3.8-Max | 98.3% | 98.3% | 86.3s |
| GPT-5 | 98.3% | 98.3% | 31.5s |
| Claude Opus 5 | 98.3% | 98.3% | 18.5s |
Another eight were Workable. Seventeen were rejected because at least one run fell below 95%.
Bank-detail changes and live safety issues need separate checks rather than being traded against easy points elsewhere in the inbox.
Task two: enter an invoice through the browser
I built a small accounts system called LedgerLite. It had a login, supplier selection, dates, purchase-order and discount fields, line items, tax codes and a save action.
Models received a text view of the current page and returned one action at a time. Playwright carried out the action and returned the next page state. Every run started clean. The saved database record was checked across 18 fields.
Invoice A was straightforward. Invoice B contained the GST test.
It printed these prices including GST:
- AAC wall panel: $62.70;
- panel adhesive: $41.80;
- regional delivery: $165.00.
LedgerLite required ex-GST unit prices. The correct entries were $57.00, $38.00 and $150.00.
Only eight of the 80 saved Invoice B runs converted all three correctly. Sonnet accounted for three. DeepSeek V4 Flash, GPT-5, Grok 4.6, GLM-5 and GLM-5.3 each did it once.
The saved Invoice B records contained 219 wrong field checks. Unit prices accounted for 216.
One model survived all six runs
| Model | Grade | Worst run | Mean accuracy | Mean time |
|---|---|---|---|---|
| Claude Sonnet 5 | Good | 100.0% | 100.0% | 97s |
| GLM-5.3 | Reject | 83.3% | 94.4% | 154s |
| GPT-5 | Reject | 83.3% | 94.4% | 186s |
| GLM-5 | Reject | 83.3% | 94.4% | 216s |
| Grok 4.6 | Reject | 83.3% | 94.4% | 100s |
| GPT-OSS-120B | Reject | 83.3% | 91.7% | 115s |
There is no Workable invoice tier because the score moves in steps of 5.6 percentage points. Seventeen correct fields is 94.4%, below the cutoff.
Sonnet entered all 18 fields correctly in every run. That earns it a larger trial, not unsupervised access to a ledger.
Haiku saved and met the old permissive threshold once in six runs.
The model still needs an accounts process around it
The GST error gives us a straightforward control. After extraction, recalculate each line, the ex-GST subtotal, GST and the invoice total. Compare those figures with the document before anything can be posted.
A business with a purchase-order process should keep that control too. Match the invoice to the purchase order. Require approval when there is no match. Send unfamiliar suppliers, unusual totals or inconsistent tax treatment to a person.
The person who keys invoices day after day will often notice when something feels wrong. An automated process needs those implicit checks written down. Without them, a model can keep processing documents that a trusted employee would stop and question.
Task three: prepare a month-end close
The close task started with an unadjusted June trial balance and seven notes. The model had to produce 23 exact figures and identify seven adjustments or controls.
The notes included accrued wages, prepaid insurance, unbilled revenue, doubtful debt, depreciation, a duplicated GST-inclusive software invoice and a bank reconciliation that needed documentation but no journal.
The duplicate was a useful test. Reversing it reduced software expense by $2,000, reduced accounts payable by $2,200 and increased net GST payable by $200. A model could repair profit and still leave the balance sheet wrong.
Three models earned a Good grade:
| Model | Worst run | Mean accuracy | Mean time |
|---|---|---|---|
| Gemini 3.7 Flash | 100.0% | 100.0% | 20.4s |
| Grok 4.6 | 100.0% | 100.0% | 42.5s |
| Claude Opus 5 | 100.0% | 100.0% | 25.0s |
The other 15 models were rejected. None landed in the Workable tier.
GPT-5.6 Terra is the best example of why the worst run matters. Its mean accuracy was 95.6%, which sounds Workable. Its three runs were 86.7%, 100% and 100%. The 86.7% run makes it a Reject.
Among all 54 close runs, working capital was wrong 40 times. Current ratio was wrong 39 times. EBITDA, net profit and closing equity were wrong 37 times each. These figures are linked, so one bad journal can travel through the entire pack.
Seventeen runs missed the duplicate invoice. Twenty-one missed the bank-reconciliation note. The second result matters because the correct action was restraint. The reconciliation already agreed. It needed to be documented and left alone.
Newer models did not inherit a better result
The generational comparisons moved differently by family and task.
DeepSeek invoice accuracy went from 70.4% on V3-0324 to 47.2% on V3.2, then 91.7% on V4 Pro. Qwen3-Max, Qwen3.7-Max and Qwen3.8-Max all averaged 91.7% on the invoice while time and cost increased. GPT-4o, GPT-5.2 and GPT-5.6 Terra also averaged 91.7% there.
Sonnet moved from 91.7% mean invoice accuracy on versions 4 and 4.5 to 100% on Sonnet 5. That was a real gain on this task. It does not tell us how the same release will behave on another one.
Once a model and prompt have cleared your test, pin them. A replacement model needs to run through the same evidence before rollout. Latest is not a qualification.
Where cost fits
The 324 original runs cost $10.00 in billed API usage. That number is useful for planning, but it says very little about whether the work was safe.
The human baselines were A$6 for inbox triage, A$3.75 per invoice and A$275 for the close. Sonnet's six perfect invoice runs averaged about US$0.105 each and 97 seconds. The three Good close models averaged between US$0.0073 and US$0.0742 per run.
Those figures make extra checking worth testing. Implementation, monitoring, review and exception handling still carry their own cost.
The next experiment will compare a single model with several reinforced pipelines. It will test independent model review, deterministic reconciliation and a human exception path. We will measure operating accuracy, time, cost and the number of cases sent to a person before deciding which configuration works best.
Limits
These were synthetic tasks under one common OpenRouter integration. The browser used a text view of the page rather than each provider's native vision or computer-use product.
Two invoices are too few for a broad accounts-payable claim. The GST check dominated one of them. The close requested JSON rather than a finished workbook and did not test evidence collection from source systems.
Three repeats per cell exposed some large failures. They cannot estimate how often those failures occur in production.
The benchmark tells us which model configurations deserve more testing on these jobs. The next decision belongs to a larger supervised trial using the business's own data and controls.
