An AI will not tell you it is guessing

In Stack Overflow’s developer survey, 84% of developers use or plan to use AI tools. About 3% say they highly trust the accuracy of what comes out, and 46% say they actively distrust it. They use it anyway.

That gap is the whole problem, and it is not really about code quality. It is that a wrong answer and a right answer arrive looking identical.

Why the output cannot warn you

An AI does not know it is guessing. There is no internal state that says “this part I read, that part I inferred” and then leaks into the wording. A confident sentence is generated by the same process whether or not it happens to be true.

There is a good measurement of this. A 2026 Snowflake AI Research paper, Confident and Wrong, ran four frontier models over 1,750 trajectories on SWE-bench Verified and separated two things that usually get treated as one: how often a model submits a patch, and how often the patch actually passes the tests. GPT-5 submitted on 100% of runs and resolved 44%. Llama 4 submitted on 99% and resolved 18%.

The authors call the gap silent semantic failure, and it is not spread evenly across failures, it dominates them: 80% of Llama 4’s failing runs and 68% of GPT-5’s. On 19 of 50 tasks GPT-5 submitted a confident-looking patch on every single run and every one failed. Not a coin flip repeated, the same misinterpretation repeated.

Silent is the important word. Most engineering failures announce themselves. A script throws. A test goes red. A build stops. This one returns a paragraph that reads like a finding.

Four false numbers in one afternoon

I run a lead pipeline where a coordinating agent reports state up to me and several worker agents do the actual reading. One afternoon the coordinator gave me four numbers that were wrong. Every one was caught by a worker or by me. None were caught by the agent that said them.

“The job is running, about 300 rows in.” It had never started. The automation fires on new or changed rows, and the rows were already there, so switching it on did nothing at all. Eighty-one minutes passed with a status that sounded like progress and described zero work.

“About 40% of the enrichment spend is wasted.” The real figure was 12%, of a different population. Same sentence shape, different denominator, three times the alarm.

“That whole branch produced nothing, everything came from the other one.” The branch it dismissed had produced 56% of the output. I was one decision away from throwing out the majority of the next batch on the strength of that line.

Not one of these was a hallucinated fact. Each was a real measurement of the wrong thing, described in the confident register of a real measurement of the right thing.

The fix that does not work

My first instinct was to ask for more care. Be rigorous. Double-check before reporting.

That does nothing, and it is worth being clear about why. An agent generates turn by turn. It has no will that persists between turns, its context gets compacted, and a fresh session starts blank. A promise to be careful is enforced by nothing and does not survive the next reset. You are asking a system for a property it has no mechanism to hold.

The fix that works

Stop asking for confidence. Require method.

Every number has to arrive with a label saying how it was obtained, and the vocabulary is fixed so it cannot be fudged:

• CAPTURED - I read the actual source, every row of the relevant population.
• SAMPLED (n, selected how) - I read a subset, and here is how I chose it.
• EXTRAPOLATED - I computed a rate from a subset, of this size.
• BOUNDED - I cannot measure it exactly, but it is provably at most this, for this reason.
• NOT MEASURED - I do not know.

The last one carries most of the weight. Giving a model an explicit, respected way to say “I have not checked this” removes the pressure that produces the confident guess in the first place. A guess is what you get when not-knowing has no legal output format.

The same paper measures why this matters. In a second experiment the authors handed models a bug that was already fixed, where the correct move is to do nothing. Most models edited the correct code anyway. They call it action bias, and note it is exactly what completion-based metrics reward: an agent scored on whether it produced something will always produce something.

Which is the same trap as a confident number. If your only two available outputs are an answer and a failure, you will get an answer.

So a usable number is not “1,522 pushed”. It is “1,522 pushed, CAPTURED, from the success status on every row of the relevant view”. The method is part of the number. Strip it and the reader cannot tell the number from a hunch.

Name the denominator, out loud

The most dangerous number I produce is not a wrong one. It is one that is 100% measured over the wrong population.

It feels maximally rigorous, because I did read every row. I just read every row of the wrong container - whichever one happened to be in my hand. The tell is that the denominator sits wherever the measuring was easy: the file I already had open, the rows that already had a value, the one campaign I had loaded.

So the question to ask about any number is not “is this right” but “which population did you divide by, and is that the population the question is about?” If nobody can name the denominator, there is no number yet.

Read the primitive, not the summary

The last piece: before stating something, read the thing the claim actually rests on. Not your memory of it, not a summary, not what another agent said about it. And read all of it, not one instance of sixteen.

I have broken this rule in the obvious way and in the embarrassing way. The embarrassing one: I spent hours blaming a deploy failure on an account permissions problem. The real cause was that I had been stamping every commit with the wrong author identity myself. There was no configuration anywhere to blame. One command would have shown me, and I did not run it, because I already had an explanation I liked.

The part that generalises

The reason all of this has to be mechanical rather than moral is that the agent has no skin in the primitive.

A human running a script gets corrected by reality constantly. The script crashes, the count comes back impossible, the file is not there. When an agent is wrong in a sentence, nothing pushes back. A false sentence costs exactly as much to produce as a true one and reads exactly the same.

So the verification has to come from outside the thing being verified. Not a better prompt asking for honesty. Two numbers derived by different methods that have to agree, a reader that is not the writer, a check that can actually fail.

The model is not the risk. The unverified claim is.

Sources: Aman Mehta, Confident and Wrong: Silent Semantic Failures in Coding Agents, Snowflake AI Research, 2026. Stack Overflow Developer Survey.