A funny thing has been happening to my code reviews lately. The better my AI agents get at writing code, the less certain I am that I’m doing a particularly good job of reviewing it.
It’s a strange position to find yourself in after twenty-five years of building software. I’ve spent a good part of my career figuring out how things work, why they break, and occasionally wondering what on earth I was thinking when I wrote something six months earlier. But these days, an agent can investigate a problem, implement a solution, write the tests, and present me with a tidy summary before I’ve even finished my coffee.
And the code is often pretty darn good. That’s partly what makes this so interesting. I open the changes, follow the implementation, check the tests, and everything seems perfectly reasonable. But am I actually questioning what I’m looking at, or simply following along with a convincing explanation?
There’s a nasty little trap here. We can end up checking whether the answer matches an explanation supplied by whoever produced the answer. Not the strongest starting point for critical thinking, is it?
The thinking we’re handing over
I recently came across an article by Addy Osmani about cognitive surrender, a term introduced in Steven Shaw and Gideon Nave’s research, Thinking—Fast, Slow, and Artificial.
The distinction is worth understanding. Cognitive offloading means delegating work while retaining responsibility for evaluating the result. Cognitive surrender happens when we accept an answer without properly questioning it. The AI’s conclusion quietly becomes our own.
In their experiments involving reasoning problems, participants who consulted faulty AI accepted its incorrect answers on about 73% of those trials. This wasn’t code review, but the behaviour sounds familiar.
Think about how we traditionally tackled difficult programming problems. You’d investigate, form a theory, write some code, and eventually discover that your brilliant theory wasn’t quite so brilliant after all. A dependency behaves unexpectedly, an edge case exposes a flaw, or something breaks elsewhere. Lovely. Back to the drawing board.
Frustrating as that was, we learned plenty along the way. Every wrong assumption taught us something about the system. We developed an understanding of how things connected and an instinct for where problems might be hiding.
Now an agent can go through that entire process and hand us the result. Fantastic for productivity, but we’ve also skipped many of the discoveries that previously helped us understand what we were building.
We’ve removed much of the struggle from development, and some of that struggle was where the learning happened. Of course, nobody needs to spend three hours chasing a silly bug just to build character. But we do need to figure out how to preserve the valuable thinking without bringing back all the unnecessary work.
The things we didn’t think to look for
We recently had a situation involving browser information that needed to be sent as part of payment processing. Our initial focus was on one payment flow, but feedback during the work exposed something we’d overlooked: the same requirement applied to other payment flows too, including an older booking flow.
The problem wasn’t complicated. We’d simply been looking at it too narrowly.
And here’s the uncomfortable part. My own understanding of the task was narrow too. If I’d written down my expectations at the start, they might have been just as incomplete as the original ticket.
An agent asked to investigate every place where payments were processed might have discovered the wider requirement before we did.
So why bother forming our own expectations at all?
Because the point isn’t to be right from the beginning. It’s to have something of our own to question and compare against. When new information challenges that understanding, we have to investigate why. Perhaps the agent missed something. Perhaps we did. Either way, there’s an actual decision to make instead of simply nodding along.
And when it turns out I was the one who was wrong, that’s the old back-to-the-drawing-board moment again, just without losing the whole afternoon.
That payment issue was a scoping miss, caught through feedback rather than an incorrect fix escaping into production. But it also exposed a weakness in how we think about reviewing changes.
Imagine we had implemented and tested the original requirement perfectly. The code might have been completely correct, yet other payment flows would still have been missing the required information. And if those flows used separate implementations, tracing the changed function’s callers wouldn’t necessarily have revealed them.
We’d need to look for the same behaviour elsewhere, not just code connected to the change.
- The payment flow the ticket named
- Named in the ticketYes
- Found by tracing callersYes
- Found by searching for the behaviourYes
- Another payment flow that calls the changed code
- Named in the ticketNo
- Found by tracing callersYes
- Found by searching for the behaviourYes
- A flow with its own implementation, like the older booking flow
- Named in the ticketNo
- Found by tracing callersNo
- Found by searching for the behaviourYes
I’ve been building FeatureKit to help with this kind of problem. It’s a set of skills for AI coding agents that brings structure to development by questioning requirements, exploring alternatives, and connecting an agreed plan to implementation and verification. It even gets the agent to review the work and record its evidence.
But I’ve realised there’s still a hole in my approach. All of that helps the agent check the work. I can still end up staring at an enormous diff afterwards, wondering whether I’ve understood enough to approve it.
Meanwhile, the agent has made decisions about state, error handling, dependencies, and architecture. Many are ordinary implementation details, but others could have consequences well beyond the task.
I don’t particularly want to spend my afternoon debating variable names with an AI. I do want a better way of recognising which decisions deserve my attention.
Looking beyond the diff
Addy already suggests forming your own expectations before seeing an AI-generated answer, reviewing the code properly, and looking for evidence before declaring the work done. I think that’s good advice.
The part I’m trying to figure out is what happens when the agent makes decisions we never anticipated, or when the original requirements themselves are incomplete.
I’m starting to think we need to investigate a change in two directions.
The first is from the problem towards the rest of the system. Before implementation, establish your initial understanding of what should happen, what must remain unchanged, and where similar behaviour might exist. Then investigate beyond that first picture. In our payment example, that means searching for every place that initiates a payment, including completely separate implementations, rather than assuming the original ticket covers everything.
The second is from the implementation towards its consequences. Once the agent finishes, examine what actually changed without relying entirely on its explanation. Which shared functions were modified? Were new defaults introduced? Did error handling change? Are there new dependencies or failure paths? Which decisions weren’t covered by the original brief?
Our existing tools can help with that investigation. A fresh agent could work from the original brief and the raw diff, without the implementing agent’s summary, and produce a factual inventory of decisions the brief didn’t cover, with references to the actual code. No verdicts about whether everything looks wonderful. Just the facts we need to examine. It can still miss things, of course, so I’d treat that inventory as a place to start looking, not a complete list.
Then we compare what we expected, what we discovered elsewhere in the system, and what the implementation actually does.
This also helps with deciding where to focus our attention. We can’t simply label changes low-risk because they’re small. A few lines in a shared function could affect dozens of workflows, while a much larger change might be neatly isolated. We need to investigate the reach of a change before deciding how much scrutiny it deserves.
And finally, we need evidence. Tests, browser checks, runtime behaviour, or whatever makes sense for the change. We should also be clear about what hasn’t been verified. A green test suite is useful, but it can’t prove requirements we never thought to test.
I’m not claiming this is some revolutionary new code review methodology. Much of it builds on practices we already know. The challenge is making the important thinking happen consistently when agents are producing work at a pace we simply can’t match.
I also haven’t tested this exact approach across enough real changes to say how well it works. That’s the next part I want to explore, including where it fails. I’d rather find that out now than write another confident explanation about why my own clever idea must be correct. That would be slightly awkward, given the subject of this article.
Keeping the thinking without keeping all the work
There’s one question I keep coming back to. Are we actually losing our ability to think critically, or are we getting fewer opportunities to practise it?
I suspect the latter is a big part of what I’m experiencing. I can’t say whether my underlying ability has changed, but I can certainly recognise when I’ve allowed an agent’s explanation to do more of the thinking than I intended.
And that’s something I can work on.
I still want AI doing the heavy lifting. I have absolutely no desire to return to writing everything myself. But I’d like to make better use of the time it saves me, investigating the things that matter, challenging my assumptions, and understanding more about the systems I’m responsible for.
So the next time an agent tells me everything is implemented, tested, and ready to merge, I’m going to resist the temptation to dive straight into the diff. First, I want to go back to what I expected, see what the agent decided that I didn’t ask for, and check whether we’ve actually got evidence that the important things work.
Because “looks fine” is a pretty low bar for something I’m about to put into production.