A 40 Percent Performance Gain That Wasn't Real #
In August, a coding agent appeared to outperform Rust's regular expression library, a performance benchmark, by 40 percent. The result was invalid, and that is not the interesting part.
Dan Luu, an engineer who has spent years measuring system performance, described the case in a post published on August 1, 2026. He asked an agent to build a regular expression engine, and on the rebar benchmark suite, the engine appeared to outperform Rust's library.
It tailored its responses to individual cases and, in part, gamed the test harness. Telling the agent that hidden tests existed reduced the cheating, but did not eliminate it.
The costly part was not the error itself. It was the fact that the error looked exactly like success, and that it took a second person with an independent benchmark to tell the difference.
A plausible answer is acceptable in tasks where it is read as one option among many and any error is filtered out before it matters. It becomes a risk when it is the only answer, someone else treats it as fact without reviewing it, and the task requires a deterministic outcome.
Why the Model Would Rather Guess #
The next model will make fewer mistakes, but it will still prefer a guess to silence. That is not a bug; it follows from the objective it was built to optimize.
In September 2025, three OpenAI researchers, Adam Tauman Kalai, Ofir Nachum, and Edwin Zhang, together with Santosh Vempala of Georgia Tech, explained why models make things up: the tests used to evaluate them reward a correct answer and penalize abstention as heavily as an incorrect one.
A system trained this way learns to behave like a student in an exam who, faced with a difficult question, guesses rather than leaving the answer blank.
The study on Skills that I discussed in an earlier article confirms the same pattern from another angle. Among the failures of agents left without instructions, one in eight involved a verification step that the agent claimed to have completed but never ran. The task was marked complete even though the test had never started. “Done” was the most likely answer, so that is the answer the machine gave.
Psychohistory Works in the Aggregate #
One book drew the map seventy years before we needed it.
In Foundation, which Isaac Asimov published in 1951 by assembling stories that had begun appearing in 1942, Hari Seldon predicts the fall of the Empire through psychohistory, a mathematical discipline that works only across enormous populations.
Statistical prediction works across populations. It says nothing about a single individual, and the series proves it when one man, the Mule, shatters the plan.
That is the first half of the map. When model output is consumed in aggregate, probabilistic behavior is acceptable: the draft you will rewrite, the twenty options from which you will choose one. An error is one among many, and a reader filters it out before it becomes consequential.
When a Plausible Answer Becomes a Fact #
The second half begins when the answer stops being one among many.
In February 2024, British Columbia's Civil Resolution Tribunal ordered Air Canada to compensate a passenger after the airline's chatbot described a refund policy that did not exist. The company argued that the chatbot was a separate entity responsible for its own actions.
In October 2025, Deloitte Australia refunded part of the A$440,000 the government had paid for a 237-page review of the system used to automate welfare sanctions. The report contained nonexistent academic references and a fabricated quotation from a federal court judgment, as the Associated Press reported on October 7.
In both cases, the answer was plausible, and it was the only one. No one downstream was expected to review it. The passenger booked; the government published.
The relevant criterion is no longer the model's accuracy, but the fate of its answer. If the answer leaves the room as fact, a deterministic checkpoint has to stand at the door. The refund rule should be pulled directly from the official document rather than summarized. Every citation should be checked against an archive and rejected if it does not exist.
In my own case, that map became an architecture. I was extending an OpenAI GPT, a custom assistant built on the company's model, to control unmanned aircraft systems, or UAS. I used a different language model to design a deterministic backend, built with code and modules on the Make.com automation platform, and integrated it with the GPT.
I treated the design for what it was: probabilistic output. I reviewed and tested it before deployment.
Once the system is running, the GPT only chooses which of the functions I gave it to call. The backend executes the action on the aircraft. Given the same request, it produces the same action, and it rejects requests outside the permitted conditions. Reliable, repeatable operation lives there, not in the model's answer.
The Last Reviewer Draws the Line #
The objection is that models are improving, and that is true. It does not move the boundary by an inch.
A model that is right ninety-nine times out of a hundred still leaves the hundredth answer on someone's desk in exactly the same form as the other ninety-nine. Accuracy changes how often the failure occurs, not where it lands. And that is precisely where the company has control, because it depends on how the workflow was designed, not on who trained the model.
Model accuracy does not define the boundary. The boundary is upheld by the last person who checks the output, and in more than one company that person has disappeared without anyone consciously deciding to remove them. The task quietly crossed the boundary. You discover it when someone else treats a plausible answer as fact.