Why AI cannot check its own code well

Everything has changed, except what matters most

A few years ago the bottleneck in programming was the writing of code itself. Today it is reading, understanding the system and taking responsibility. Large language models generate hundreds of lines in a dozen seconds. No team can read and understand that code at the same pace at which it is produced.

This speed has a price. Generating code with an LLM comes with a certain unpredictability of the final result. The model does not “know” that a solution is correct, it judges that it is probable. And software quality cannot be variable. A banking system, a calendar, an online shop or a food-ordering app either works correctly or it does not.

And here a tempting shortcut appears: since we will not have time to read the code, let AI write the tests too. Green build, two seconds, done.

The trap of the superficial supervisor

This is the heart of the problem. When the “prompt -> result” cycle happens too fast, the human stops being an active participant and becomes a superficial supervisor. Supervising does not engage our cognitive processes. We look at the green build and nod. It works, so let's move on.

If a human only accepts results, they do not build their own understanding of the system. And without understanding we will not ask the question that really matters: Does this do what I think it does?

Where is the human irreplaceable?

If AI writes the code, and AI should not test it on its own, what is left for the human? Three things the model will not do well for us.

Defining fundamental properties

Not “for this example”, but “for all possible cases”. A user never sees a stack trace. Every element of a sorted list is less than or equal to the next. The sum of money after a transfer between accounts does not change. One person cannot be in two meetings at once. Such invariants come from understanding the system, not from syntax. A study from the FSE 2025 conference showed that models can extract correct properties in about 76 percent of cases, and generate code consistent with them in about 52 percent. That is too little to hand this step over to a machine. We cannot count on it working once and the next case, in different conditions, failing.

Understanding counterexamples

Imagine AI wrote a function that calculates a cart price after applying a discount code. Codes can be percentage (“-20%”) or fixed-amount (“-30 PLN”). The model also added example tests: a cart worth 100 PLN with a -20% code gives 80 PLN, a cart worth 100 PLN with a -30 PLN code gives 70 PLN. Both pass. Green build. In the age of working with AI we could conclude that since everything works and the tests pass, there is no need for further verification, but are we sure?
A person who knows the system and its assumptions can overturn our certainty of correctness with one simple assumption: “The price after the discount is never lower than zero and never higher than the price before the discount.”
The simplest raw example, which the AI did not consider at all:

• cart: one product for 15 PLN

• discount code: -30 PLN

• function result: -15 PLN

Now everyone sees the bug. The function subtracts the discount amount from the cart total and does not check whether the discount is larger than the cart itself. A customer with such a code would get a negative price, and depending on the further logic, either a payment error or, worse, a refund for the purchase.

Why did the model not check this, either in the code or in the tests? Because in its “imagination” of the task the discount code was always smaller than the cart. Nobody wrote this in the prompt, but nobody wrote that it could be otherwise. The same gap in the model went into the implementation and into the tests at the same time. The same error twice.

A counterexample does more here than point to a single line to fix. It forces the question: what else could be “larger than I assumed”? A percentage discount above 100%? Two codes used at once? A discount calculated on a cart from which the customer has meanwhile removed products? From one counterexample come three more properties to check, and from them probably more counterexamples. Each such loop deepens an understanding of the system that we did not have before, even though the code “worked”.

Solving the problem, not just writing code

Writing code is only one part of engineering. Understanding the problem, setting the boundaries of correctness and verifying that the system does not cross them is the part that does not get cheaper. For years these two things were glued together, because the same person did both and often discovered a decision to be made only while writing. AI has unglued them. Take the discount code as an example. A programmer who understands the discount module does not have to open the code to answer: “what happens when a customer uses two codes at once?” When they remove a product from the cart after entering the code? When the code expires between adding to the cart and payment? They know the answers not because they remember every line by heart, but because they hold in their head a model of how discounts work: what kinds there are, at what moment they are applied, what takes precedence, where the boundaries are.

Understanding does not scale with code

And here lies the fundamental problem of the AI era. Code can today be produced at a pace no human can keep up with. Understanding arises at the pace of human thinking and cannot be sped up with a prompt. Every generated module that nobody on the team understands is a debt: the system works, but nobody can predict when it will stop and why. This debt does not appear in any tool for measuring code quality. It shows up only during an outage, a change in requirements, or when the only person who “remembered something” leaves. It is not about reading every line, because that is impossible. It is about having places in the process where a human has to formulate what the system should do, predict how it will behave, and confront that with reality. Properties, understanding the problem and counterexamples are tools for creating such places. But the tool is secondary. The goal is primary: A team must know how what it hands to the client works - and that is the value that guides our work.