The Pre-Production Gates

Part 4 of Security Without a Security Team, a series on the security controls behind live production systems at Mannat AI Labs.

The last ring is the first one that acts

The first three parts of this series were about what happens to traffic that arrives: the edge that filters it, the identity layer that challenges it, the nervous system that watches it. This part is about the only ring that acts before anything arrives: the set of gates a change must clear before it is allowed to exist in production at all.

The principle underneath this ring is blunt: a warning you can ignore is a warning you will ignore. Every control described below is wired so that failing it has a consequence: a commit flagged, an image withheld, a deploy that refuses to start, a pod the cluster will not admit. Reports inform. Gates decide. A small team does not have the headcount to chase reports; it needs decisions made by machinery, with humans reserved for the judgement calls the machinery escalates.

The gate, stage by stage

Every application on the platform ships through the same pipeline shape. On each push, before anything is published:

Source scan. The repository is scanned for committed secrets (across the full git history, not just the diff), plus known-vulnerable dependencies read from the lockfiles, and infrastructure-as-code misconfigurations. Secrets are fail-closed with no grace period: there is no "triage the backlog first" for a leaked credential, because the only correct number is zero.

Static analysis. The application source is scanned for insecure code patterns, the class of bug that is invisible in a demo and obvious in an incident report.

Build. The real production image is built, loaded locally, not yet published anywhere.

Image scan. The built image is scanned as a unit: base operating system, bundled tooling, application dependencies. This is the stage that catches what the source never shows you: the vulnerable library that arrived inside the base image, not inside your code.

Sign and publish. Only if every stage passed is the image signed and pushed to a private registry. The signature is the pipeline's testimony: this exact artifact cleared the gate.

Because the platform only ever deploys images by tag, an image that never gets published is an application that simply cannot deploy. The gate does not ask for cooperation. It withholds the thing the deploy needs.

Two details keep the gate itself trustworthy. Every scanner runs from a pinned, immutable reference, so nothing in the pipeline can be silently repointed by a third party. And the threshold is deliberately scoped to fixable high-and-critical findings: a finding with no available fix never blocks, because a gate that wedges people on things they cannot act on gets muted within a week, and a muted gate protects nothing.

The day it earned its keep

The honest test of a gate is the first time it meets a real codebase. When the first full application was onboarded to the platform, the gate went red five times before it let the app through, and none of it was ceremony.

The source scan flagged eight denial-of-service vulnerabilities in the library handling every file upload in the app. Upgrading to the patched major release closed all eight before the application ever served a request; the app's own test suite confirmed nothing broke.

The image scan found the base image carrying vulnerable OpenSSL builds, and sixteen more findings inside package-manager tooling that was bundled into the runtime image but that the application never runs, including a critical decompression flaw. The fix was two-fold: patch the OS, and strip the dead tooling out of the runtime image entirely. Code that is not there cannot be exploited.

Static analysis then turned the gate on its own scaffolding: it caught the CI workflows referencing third-party actions by mutable tags, the exact silently-repointable pattern behind recent real-world pipeline-compromise attacks. Every reference is now pinned to an immutable revision. It also flagged a log statement a crafted input could have used to forge log lines. Fixed the same day.

And one finding was wrong: the scanner read an environment variable ending in _FILE as a leaked secret, when it holds a file path and the secret itself is generated at runtime. That false positive got the only acceptable treatment: a single, justified suppression whose reasoning lives in the diff next to the rule it silences: reviewable, revertible, and specific. Remediate the real ones; justify the false ones in writing. That one rule is what keeps a fail-closed gate honest instead of theatrical.

Five red runs. Four real classes of vulnerability fixed, one false positive documented, zero vulnerable images shipped. The upload-library upgrade was never a ticket for next sprint. It was the price of shipping at all. That is what fail-closed changes: not what you know, but what you are allowed to defer.

One bar, two doors

Not every application deploys by image. One of the systems in this series builds in place, on the server it runs on, straight from source. No registry in the middle. No image to withhold. The chokepoint the gate above relies on does not exist.

The wrong answer would have been to exempt it. An exempted app is where the incident will be. The right answer was to move the door, not the bar: the same scanners (secrets, dependencies, code patterns) run as a preflight that the deploy job depends on. The deploy cannot start until the gate ahead of it comes back green. No clean scan, no rollout. Same sentence as every other app, enforced one link further down the chain. One script runs the identical scan on every push, every pull request, and as the deploy preflight, so a finding flags a commit and blocks a release the same way, with no drift between what the push check enforces and what the deploy enforces.

The policy is uniform across the platform. Only the mechanism bends, because a good gate meets each application where it actually deploys.

The cluster does not take the pipeline's word for it

A signed image is a claim. The platform verifies the claim at the moment it matters: admission. A policy engine sits at the cluster boundary, and an unsigned image, or an image whose signature does not match what the pipeline produced, is not slowed down or flagged. It does not run. Full stop.

The same boundary enforces the platform's deployment hygiene before anything reaches the cluster: manifests are validated in review: no mutable image tags, resource limits declared, health probes present, services private by default, images only from the organisation's own registry. What the review gate checks as a required status, the admission layer enforces again at deploy time. Belt, then braces, because the two catch different failure modes: review catches the honest mistake; admission catches everything else, including the change that never went through review.

This closes the loop the earlier stages opened. The source was scanned. The artifact was scanned and signed. The signature is verified where the artifact runs. The chain from commit to running pod has no unverified link, which is the actual meaning of supply-chain security, as opposed to the checklist meaning.

The attacker's-eye view, on a schedule

Everything above inspects the system from the inside. One control deliberately does not: a dynamic scanner probes the running application the way an attacker would: from outside, against a staging environment, with no knowledge of the source. It runs on a schedule rather than blocking deploys, and that is a design decision, not a compromise: dynamic findings are noisier and slower to triage than static ones, and a gate that blocks on noise trains people to route around it. Its findings feed the same triage discipline as everything else: remediate the real, justify the exceptions in writing.

The runtime completes the picture: secrets are issued to workloads from a dedicated secrets manager rather than living in configuration, and public API traffic passes through a gateway that owns the cross-cutting concerns: one place to reason about exposure instead of a dozen.

From one app to every app

The most important thing the gate produced was not a fixed CVE. It was a standard. Once the pipeline proved itself on the first application, the whole thing was lifted out of that repository into a copy-paste template and a short handbook: the pipeline stages, the fail-closed policy, and a triage playbook for each finding class. The next application inherits the entire gate by copying a single file.

This is the platform-thinking point of the whole series, so I will make it explicit: hard-won lessons must become the paved road, or they remain tribal knowledge that leaves when the person does. A control that exists once is a fix. A control that every future application inherits by default is a posture. The second is the actual job.

The verification script: trusting nothing, weekly

Every control above shares a weakness: each one asserts something at a moment in time. The scan was clean when it ran. The manifest was valid when it merged. But production drifts: certificates expire, containers get rebuilt from base images missing an assumed module, a configuration file quietly stops being loaded. Part 0 called this the real work, and this is where the work lives.

So the final gate is a hand-written verification script that opens the running production system and asserts that every security-relevant claim in this series is still true, right now. Not "was configured." Is configured, and I just proved it.

The generic shape — free to lift into your own stack:

#!/usr/bin/env bash
# verify-posture.sh : assert the running system still matches its claims.
# Every check hits the LIVE system. Exit non-zero on the first broken claim.
set -euo pipefail

fail() { echo "DRIFT: $1" >&2; exit 1; }

# 1. Transport: certificate valid and not near expiry
days_left=$(check_cert_days_remaining "$PUBLIC_ORIGIN")
[ "$days_left" -gt "$CERT_MIN_DAYS" ] || fail "TLS cert expires in ${days_left}d"

# 2. Edge: security headers actually present on a real response
for h in strict-transport-security x-content-type-options; do
  response_has_header "$PUBLIC_ORIGIN" "$h" || fail "missing header: $h"
done

# 3. Edge: the WAF still inspects (a benign canary probe must be acted on)
waf_reacts_to_canary "$PUBLIC_ORIGIN" || fail "WAF did not react to canary"

# 4. Identity: auth endpoints still rate-limited (canary burst gets throttled)
login_burst_is_throttled "$PUBLIC_ORIGIN" || fail "login rate limit inactive"

# 5. Runtime: every expected workload present, none unexpected
diff <(list_running_workloads) expected-workloads.txt || fail "workload drift"

# 6. Observability: log pipeline intact (recent entries carry expected labels)
logs_have_expected_labels "$SINCE_MINUTES" || fail "log labels missing"

# 7. Supply chain: running workloads trace to signed artifacts only
all_workloads_signature_verified || fail "unsigned workload running"

echo "OK: all claims verified against the live system"

The function bodies are yours to write. They are where your coordinates live, which is exactly why they stay out of a blog post. The shape is the point: every check interrogates the live system, the script fails loudly on the first broken claim, and it runs on a schedule whose output lands somewhere it stares at you every week. The only defence against drift is a machine you can point at production and ask is it still true? And the only defence against forgetting to ask is making the asking automatic.

Why this order, and not another

A closing note on sequencing, because a security posture is not a shopping list. It is a dependency graph.

Secrets scanning came first because a leaked credential invalidates every other control. Scanning came before signing because a signature on an unscanned artifact is testimony to nothing. Signing came before admission verification because the cluster can only verify what the pipeline attests. Dynamic testing came after observability, because findings you would never see and could never act on are not a control. They are a liability with a dashboard. Each layer earns its keep only because the layer beneath it is already true.

That is the shape of the whole series, and the reason it was written as four parts and not one: not a stack of tools, but an argument, each ring justified by what the previous ring cannot do alone.

If you have built this differently, or better, the comments are the review. That offer from Part 0 stands.

A note on context: everything here describes independent projects run under Mannat AI Labs. Views and mistakes are entirely my own.