The User Story We Never Wrote

The feature is finished. The tests pass, the code has been reviewed, and everyone agrees it does what it’s supposed to do.

Somewhere out there, somebody is asking a completely different question.

What can I make this thing do that it was never supposed to?

And unlike us, they’re not particularly concerned about acceptance criteria, sprint commitments or whether their approach follows our carefully designed user journey.

How inconsiderate.

I’ve always considered security part of building software properly. Authentication, authorisation, validation, protecting sensitive data. These things are hardly new.

But some recent security work has made me question something about my own approach.

How much time do I spend thinking about how to secure what I’m building, compared with actively trying to figure out how somebody could break it?

Because those are two very different ways of looking at the same system.

And with AI changing the way attacks are carried out, I think that distinction deserves considerably more attention.

The other side got AI too

We’ve become rather good at using AI to accelerate development. Agents can explore codebases, implement features, write tests and work through problems that previously required hours of manual effort.

Unfortunately, the people attacking our software have discovered the same technology.

In November 2025, Anthropic reported disrupting an AI-orchestrated cyber-espionage campaign targeting roughly thirty organisations, with a small number of successful compromises.

By September 2026, its threat intelligence team was documenting attackers using coordinated agent swarms to investigate multiple targets, identify weaknesses and conduct operations in parallel.

Think about what that means.

An attacker no longer needs to personally investigate every potential weakness. They can delegate much of that work to agents, explore several possibilities at once and keep going while the human focuses elsewhere.

And many of the vulnerabilities being exploited aren’t particularly exotic. They’re familiar problems involving credentials, permissions, exposed services and assumptions that haven’t been properly challenged.

The scary part isn’t that attackers have suddenly invented entirely new ways to break software. It’s how much faster they can explore the ways we’ve already left open.

We’ve given ourselves an enormous productivity boost with AI. Attackers have done exactly the same.

The question is whether we’re adapting how we think about security quickly enough.

We know how it’s supposed to work. That’s part of the problem.

When building software, we naturally develop a mental model of how everything fits together.

We understand the user, the requirements, the flow of information and the boundaries between different parts of the system. We make decisions based on that understanding.

It’s how good engineering happens.

But it also creates blind spots.

Imagine an application where authenticated users can download their own documents. We check that the user is logged in, retrieve the requested document and return it.

Everything works.

Now someone replaces the document identifier in the request with one belonging to another user.

Does the backend verify ownership? Does every route to that document enforce the same boundary? Or have we trusted something simply because the frontend would never normally send that request?

It’s a simple example, and experienced developers will recognise the problem immediately.

Which makes it all the more interesting that these kinds of vulnerabilities keep appearing. Broken Object Level Authorization, which is exactly this problem, sits at the top of OWASP’s 2023 API Security Top 10.

We understand the security principles. The difficult part is consistently recognising where our assumptions might fail.

An attacker doesn’t share our mental model. They’re actively searching for the gaps in it.

They don’t care whether an operation makes sense in the normal workflow. They’ll happily try requests in unusual sequences, manipulate inputs and explore interactions that nobody thought to include in the requirements.

And sometimes the most useful question isn’t whether the code contains a bug.

It’s whether the system allows something that should never have been possible in the first place.

Thinking like an attacker is a muscle

The security industry has understood this for a long time. Threat modelling, penetration testing, red teaming and abuse cases all involve looking at systems from perspectives beyond their intended use.

But knowing these practices exist doesn’t automatically make us good at that kind of thinking.

I certainly don’t think it comes naturally just because we’ve built enough systems.

We’re trained to solve problems. Given a requirement, we work towards a solution. We become good at recognising patterns and anticipating implementation difficulties.

Thinking like an attacker requires a slightly different instinct.

Instead of following the intended behaviour, we deliberately look for ways around it. We question the assumptions that make the design appear safe. We imagine what someone could gain by crossing a boundary and work backwards to discover how that might happen.

And that takes deliberate, regular practice.

Write the other user story

So here’s the practical part. It’s the habit I’m adding to my own workflow, and it fits into something we already do.

Every feature starts with a user story. So write a second one: what the user must not be able to do.

Take the document example.

It sounds too obvious to bother writing down. That’s exactly why it gets skipped, and why it never gets tested. But it’s just as much a requirement as the first one, and it deserves its own acceptance criteria.

Each of those criteria becomes a test. Usually that’s as simple as creating two users and trying to cross over.

Finding these stories means thinking like an attacker, but it doesn’t need a security background. For any feature that touches someone’s data, money or permissions, I’m starting with four questions:

• Swap it. Change any identifier in the request to one that belongs to someone else.
• Skip it. Call the API directly, without the screen or the step that normally comes first.
• Repeat it. Send the same request twice, or twice at the same time.
• Stretch it. Send values the UI would never allow. Negative amounts, huge numbers, or fields like a role or a price that the user shouldn’t be able to set.

Not every question applies to every feature, and often the answer is “that’s already handled”. That’s fine. Asking them takes ten minutes during refinement. Sometimes what comes out of it changes the design. Either way, the answer is now written down and tested, instead of assumed.

A story that touches someone’s data, money or permissions isn’t done until we’ve tested what it should refuse.

The hard part isn’t the questions. It’s asking them consistently, especially when the implementation looks perfectly reasonable and the sprint is already running late. That’s why I want it in the story, not left to memory.

Give the agents something to check

Here’s where I think AI offers an interesting opportunity.

We’re already using agents to write and review software. Why not deliberately use them to challenge our thinking from an attacker’s perspective?

These stories make this much easier, because now the agent has a specific goal instead of a vague question like “is this secure?” Something like this:

Rule: users can only access documents they own.
Go through every route that reads, changes or deletes a document
and check that the code enforces this rule.
Write tests where a second user tries each route and is refused.
Don't tell me whether it's secure. List any route where a test
fails or the check is missing.

Different agents can get different stories. One goes after ownership, another after repeated or out-of-order requests, a third after fields that shouldn’t be editable. Each works from an independent objective, rather than simply reviewing whether the implementation looks correct.

And instead of a reassuring report saying everything appears secure, we get evidence. Failing tests, or a clear list of the places where the check is missing.

One gotcha worth knowing. The newer frontier models have deliberately broad cybersecurity safeguards. Ask an agent to attack your app or produce working exploits, and it may flag the session, even when it’s your own code. Claude Opus 5.5, for example, hands flagged work to an older model. Reviewing your own source code and writing tests is the work these tools are meant to help with, so frame the job that way: here’s the rule, check the code enforces it, write the tests. That’s the better instruction anyway. A test keeps protecting you after the review is done.

I explored a related idea in The Code Looks Fine. That’s What Worries Me., where an independent AI review became more useful when it had expectations to measure the implementation against.

The same principle applies here.

But there’s a catch.

If I hand all the adversarial thinking to an agent, I’m not developing that skill myself. I’m simply adding another automated step to the process.

So I write the stories first and let the agents check the code against them. And while I’m quite happy for AI to do the repetitive investigation, I still want to understand why a particular weakness exists, which assumption failed and how we could have recognised it earlier.

Otherwise we might become very efficient at fixing vulnerabilities without becoming much better at preventing them.

That’s a slightly awkward place to end up.

Security needs a seat at the table

The more I think about this, the more convinced I am that security has to be treated as part of understanding the problem itself.

By the time we’re reviewing a completed feature, some of the most important security decisions may already be buried in its architecture.

If we discover an overlooked trust boundary early, we can design around it. Discover it after the feature has been built, and we might find ourselves undoing decisions that seemed perfectly sensible at the time.

So my approach going forward is simple. Write the story we usually skip next to the real one, test that the software refuses it, and let the agents check the code against it.

We don’t all need to become professional penetration testers. Their expertise remains incredibly valuable.

But I do believe we need to become much better at approaching our own systems with the curiosity and persistence of someone determined to find a way through them.

AI has made that capability increasingly accessible to attackers. We have every reason to develop it ourselves.

Because somewhere out there, someone is looking at a feature we’re rather proud of and wondering what else they can make it do.

They never agreed to our acceptance criteria.

Perhaps it’s time we started writing a few from their perspective.