The Incident I Rehearse For

The capstone to Security Without a Security Team, a series on the security controls behind live production systems at Mannat AI Labs. This is the fifth piece. I only planned four.

A confession about this piece

The four parts before this one each opened with something that actually happened. The first probe three minutes after DNS resolved. The morning I could not cleanly remove one person's access. The disk that filled at 4pm while I found out at 9pm. Those were real, and telling them honestly was the whole point.

This piece is different, and I want to be straight about how before you read a word of it.

I have not been breached by a supply-chain attack. Nothing in what follows is a war story. It is a fire drill. It is the incident I rehearse for, assembled from the near-misses I have caught, the shapes I actually see in my logs, and the public disasters that hit teams far better resourced than mine. I am walking through it deliberately, start to finish, because the honest measure of a security posture is not how it looks on a calm afternoon. It is how it behaves on the worst day, and the only way to know that before the worst day is to rehearse.

So treat this as a tabletop exercise written down. The scenario is constructed. The controls it leans on are real, and live, and the ones I would actually reach for. Where the drill exposes something I have not solved, I say so, same as every other piece in this series.

I am structuring it in three acts, because that is how an incident actually unfolds: crisis, survival, advancement. The breach, then the fight to contain it, then the changes that get baked in so the same thing cannot happen the same way twice.

Why this attack, of all of them

I picked a compromised dependency on purpose. It is the attack that should frighten a small team the most, because it is the one where doing your own job perfectly is not enough.

You can write clean code. You can review every line you commit. You can scan, sign, and verify everything you build. And then a package five levels deep in your dependency tree, one you have never heard of, gets a new maintainer with bad intentions, or an existing maintainer's account gets phished, and a malicious version ships to the world. You did nothing wrong. The poison arrived inside something you trusted, through the front door you hold open for every routine update.

This has happened, publicly, more than once. A maintainer account taken over and a data-stealing payload slipped into a wildly popular package. A years-long social-engineering campaign to plant a backdoor in a compression library that nearly reached the entire Linux world before one engineer noticed a few milliseconds of latency. Self-propagating worms that harvest developer credentials and use them to publish more poisoned packages. The genre is not hypothetical, and it is not rare. It is the ambient risk of building software the way everyone now builds software.

Every ring in this series was, in some sense, built with this attack in the back of my mind. This is the drill that tests all four at once.

Act 1: Crisis

It starts the way these always start. Not with an alarm, but with an advisory.

A security bulletin lands. A package is compromised. Specific versions are named. And the entire industry, all at once, asks the same three questions in the same order.

Am I affected? Somewhere in my dependency tree, do I have a named version?

Did it ship? If I have it, did it actually make it into something running in production, or is it sitting harmlessly in a lockfile for a service that never deployed?

Is it running right now? This second, is there a workload live on my infrastructure executing that code?

Here is the honest part about the crisis phase: the danger is not the malicious package. The danger is the time between the advisory landing and you being able to answer those three questions with certainty. That gap is where panic lives. That gap is where you start SSH-ing into boxes and grepping lockfiles by hand at midnight, hoping you remember every service you run. A small team that cannot answer "am I affected" quickly does not have a security problem yet. It has an inventory problem, and the security problem is about to exploit it.

So the crisis, properly understood, is a race between the advisory and your own ability to see yourself clearly. Everything in the survival phase is really about having already won that race, before the starting gun.

Act 2: Survival

Survival is three jobs, in order: work out the blast radius, contain it, then eradicate it. Panic wants to do them in the opposite order, or all at once. Discipline is doing them in sequence.

Blast radius: answer the three questions fast.

The reason I can answer "am I affected" in minutes rather than hours is that the pipeline from Part 4 already produces the answer as a side effect. Every image the gate builds is scanned, and the dependency review reads the full lockfile, not the top-level list. The lockfiles are pinned and committed, so the exact version of every transitive dependency is a fact in the repository, not a mystery resolved fresh on each build. "Am I affected" becomes a search across pinned manifests, not an archaeology dig.

"Did it ship" is answered by the signing chain. Only artifacts that cleared the gate get signed and published, and the platform only ever runs signed artifacts. So the set of things that could possibly be running is exactly the set of things the gate approved, which is a known, enumerable list, not the open question it would otherwise be.

"Is it running right now" is answered by the observability plane from Part 3. I can see what is deployed, on which host, since when. The inventory is not in my head, where it would be wrong. It is in a system I can query.

Winning the blast-radius race is not heroics. It is the dividend of boring inventory discipline paid forward.

Containment: assume it did run, and shrink what that means.

Suppose the worst honest answer: yes, a compromised version made it into a running workload. Now the design decisions from the earlier rings stop being architecture diagrams and start being the difference between a scare and a catastrophe.

A supply-chain payload almost always wants two things: credentials, and a way to phone home with them. So containment is about starving it of both.

It reaches for the environment, greps for anything that looks like a secret. In the posture this series describes, application secrets are issued from a dedicated secrets manager rather than sitting in plain environment variables, so the easy harvest comes back thinner than the attacker hoped. What it does find has a short life. The access tokens from Part 2 expire in a coffee break, and the refresh tokens are hashed at rest and revocable as a family in one call. A stolen token is a badge that stops working almost immediately, and I can make it stop working for everyone, everywhere, with the revocation endpoint I built for the day someone leaves.

Then it tries to phone home. This is the control I am least smug about, and I will come back to it in the honest section, but the shape of the defence is egress discipline: a workload that has no legitimate reason to open connections to the wider internet should not be able to, so that exfiltration meets a wall rather than an open door.

Containment, in other words, is not one dramatic action. It is the compounding effect of a dozen quiet decisions that each individually looked like mild inconvenience and collectively mean the payload lands in a room with no windows.

Eradication: roll back to something you can trust, then rotate.

Getting the bad code out is the easy part, precisely because of the signing chain. There is always a last-known-good signed artifact, and rolling back to it is a normal deploy, not a forensic reconstruction. The runtime will run the known-good signed image because it is signed and verified; it would refuse the compromised one anyway if the signature did not match. The property that makes deploys safe on a normal day is the same property that makes recovery boring on the worst day.

Rotation is the part people forget in the relief of having rolled back. Anything the payload might have touched during its window gets treated as burned: rotate the credentials, revoke the sessions, invalidate the tokens. Not because I know they were stolen, but because the entire point of the exercise is to not have to know. You assume exposure and rotate, because certainty is a luxury you do not have and cheap rotation is a luxury you can build.

Act 3: Advancement

Survival gets you back to standing. Advancement is the only part that actually matters in the long run, because it is what makes the next instance of this attack a non-event instead of a repeat performance. An incident you survive but do not learn from is just a rehearsal for the one that finally gets you.

Here is what this class of attack bakes into the architecture, permanently.

Inventory becomes a query, not a panic. The single biggest upgrade is generating a bill of materials for every build, so "am I affected by advisory X" is answered by searching a stored artifact, in seconds, across every service at once. The crisis phase is won or lost on this, so it stops being something I hope I can reconstruct and becomes something the pipeline emits every time, automatically.

Dependencies get pinned to immutable references, everywhere. Not floating version ranges that silently pull in whatever was published most recently, but exact, content-addressed pins, so an update is a deliberate act that goes through the gate, never an ambient thing that happens to you overnight. The gate already caught mutable references in CI actions once, in Part 4. This extends the same principle to the whole tree.

The signing chain gets extended toward provenance. It is not enough that an artifact is signed by me. The stronger claim is that an artifact was built from this exact source, by this exact pipeline, with this exact set of inputs, and can prove it. That is the direction the chain moves: from "this is mine" to "this is mine and here is the verifiable story of how it came to be."

Egress becomes default-deny. The honest gap below becomes the next project. A workload should reach exactly the destinations its job requires and nothing else, so that phoning home is not merely monitored but structurally impossible for a service that has no business talking to the outside world.

The verification script grows another assertion. Consistent with the whole series: anything I learned to check by hand becomes a thing the machine checks on a schedule. After this drill, the script gains a check that the running workloads match the signed, known-good set, and that nothing is deployed whose provenance it cannot verify. The drift I rehearsed for becomes drift the machine notices without me.

Each of these is sequenced behind the controls that came before it, the same way the whole series is. Provenance is only worth proving once signing exists. A bill of materials is only worth generating once the pipeline is the single path to production. Advancement is not a pile of new tools. It is the next layer that the previous layers finally make worth building.

The honest part

Every piece in this series has one, and a fire drill that skipped it would be worse than useless, because the whole reason to rehearse is to find what breaks.

Here is what this drill surfaces that I have not fully solved.

Egress control is the big one. The containment story above leans on a workload being unable to phone home, and my egress discipline is not yet where that sentence implies. It is a real gap between the defence I describe as the shape and the defence I have fully built. I am telling you that plainly because the alternative is to let a rehearsal quietly become a boast, and that is exactly the drift the series is against.

Provenance is aspirational, not done. The signing chain is live. The full "prove how this artifact was built" story is a direction, not a finished control. I would rather name it as the next layer than imply it already stands.

And the deepest limit is the one no small team escapes: the speed of the initial advisory is not mine to control. If a compromise is discovered fast and disclosed responsibly, my inventory discipline wins the race. If it sits undiscovered upstream for months, as the compression-library backdoor nearly did, then no amount of my tooling helps during the window when nobody in the world knows to look. Defence in depth narrows that window's blast radius. It does not close the window. Anyone who tells you their posture closes it is selling something.

What the drill is really for

I do not run this exercise because I expect this exact incident. I run it because rehearsing the worst day is the only way to find out whether the calm-afternoon architecture actually holds under load, and to find it out while the stakes are imaginary instead of real.

That, in the end, is the thread through this whole series. A WAF you do not watch is a checkbox. An auth stack you have not tested against "someone leaves today" is a login screen. Observability you have not proven survives the origin catching fire is a dashboard. A gate you have not run red is a suggestion. And a posture you have never rehearsed against its worst day is a diagram.

The work, all of it, is the difference between believing you are defended and having checked. I check by looking, on a schedule, at controls that fail closed, and by occasionally sitting down and walking through the fire on paper before it happens for real. That is what a small team can actually do. No dedicated security team. No large budget. Just the discipline of assuming the worst day is coming and refusing to meet it unrehearsed.

That is the whole series. Thank you for reading it.

If you have run this drill and found something I missed, that is the comment I most want to read.

A note on context: everything here describes independent projects run under Mannat AI Labs. The incident in this piece is a constructed exercise, not a real event. Views and mistakes are entirely my own.