The Overview

Part 0, the overview. A 4-part series on the security controls behind live production systems at Mannat AI Labs.

Why this series exists

Mannat AI Labs is the umbrella I run my independent projects under. It exists so that ideas take shape as running systems instead of dying on a whiteboard. A few of those projects are live, in production. When one of them goes down, work stops. When one gets compromised, trust is gone. The code being open source does not lower those stakes; it raises them, because anyone can read exactly how each system is built.

There is no dedicated security team behind any of this. There is just me, one engineer with production systems carrying real data and real consequences.

Most writing about production security assumes a team: a person whose whole job is security, another whose whole job is compliance, a third who runs the tooling. Useful, but not the situation I am actually in. The situation I am actually in is the one most small engineering teams are in: one or two engineers stretched across product, infrastructure, incident response, and, somewhere in there, the security posture.

I finished my Master's in Cyber Security at IIT Kanpur in 2025, followed by a six-month AI fellowship at OutSkill, and this lab is where that education continues. A live production system teaches things no course or certification does. It means owning the full stack end to end (security, platform, operations) and having complete freedom to test ideas against production reality, at my own risk, on my own infrastructure. This series is partly a record of that education.

So this is the honest write-up of what I have actually built, kept running, and would defend. Four pieces, one layer each. Not a manifesto. Not a checklist. The shape of a small-team defence, told from the inside.

Before I go deep into any one layer, I want to lay the whole thing out on one page.

The four layers, in one glance

Think of the system as four concentric rings. Traffic enters from the outside, works its way in, and every ring does exactly one job.

Ring 1: The edge. A reverse proxy fronts everything public-facing. In front of that sits a CDN and WAF provider that terminates TLS at the network edge, absorbs volumetric attacks, and rewrites the client IP into a header the origin can trust. On the origin itself, a second WAF (an OWASP Core Rule Set instance running inside the same proxy container) inspects every request against roughly 1,700 published rules. A separate monitoring dashboard, built into the application, tails the WAF's event log in near real time, so I can see what is happening instead of guessing.

Ring 2: The identity layer. Nothing behind the edge trusts you until you have proven who you are. Short-lived access tokens paired with rotating refresh tokens, both hashed at rest. Time-based one-time-password 2FA with hashed backup codes. Email-match-only OAuth for the two identity providers we support. A one-call revocation endpoint that kicks any user out within minutes. Rate-limited login. Audit logging on every authentication event.

Ring 3: The nervous system. A second physical server exists whose entire job is to watch the first one. Between them is a private encrypted tunnel; the observability plane does not touch the public internet. Every container's logs are shipped, every host and container metric is scraped, dashboards live as code in the repo, and alerts fire to a chat channel that reaches my phone. When something starts to go wrong, I know within minutes, not five hours later from a user's message.

Ring 4: The pre-production gates. Before anything ships, it clears six checks. Static analysis. Secret scanning. Dependency review. Type check. Build. And then the one most people skip: a hand-written verification script that opens the running production system and asserts that every security-relevant claim I care about is still true. The script is the difference between "I meant for this to be configured" and "this is configured, right now, and I just proved it." The gate does not end at the build either. What the pipeline approves gets signed, and the runtime refuses to run anything that does not carry a valid signature, so the chain from commit to running workload has no unverified link. And the gate is not unique to one app: every application on the platform clears the same bar, even though they do not all deploy the same way. Part 4 comes back to that point.

That is the whole posture. Four rings, each with a specific job, each with its own failure modes, each capable of doing its job when the other three are having a bad day.

The principles underneath

The specific components are less interesting than the principles they encode. If you are building the same thing with different tools, these are the pieces worth stealing.

Defense in depth, not defense in perfection. No single ring is expected to be airtight. The edge misses things; that is why the identity layer exists. The identity layer misses things; that is why observability exists. Observability misses things; that is why the pre-production gates exist. Any one ring failing on its own is an incident. Two failing at once is a bad day. All four failing at once is the sort of scenario a small team accepts as a risk, in exchange for building a system a small team can actually run.

One bar, many doors. The security policy is identical across every application on the platform: secrets, vulnerable dependencies, and insecure code do not ship, anywhere. The mechanism is allowed to differ, because the apps do not deploy the same way. An app that ships as a container image gets gated at the image: no clean scan, no published image, nothing to deploy. An app that builds in place on its own server has no image to withhold, so the same scanners run as a preflight that the deploy waits on: no clean scan, no rollout. The bar does not move between apps. Only the door it stands in front of does. A gate that insists on one mechanism everywhere either blocks apps from existing or quietly exempts the ones that do not fit, and an exempted app is where the incident will be.

Boring, self-hosted, open-source tools wherever possible. Every component in this stack has been running in serious production for years, is documented to death, and does not disappear if a vendor changes their pricing page. The exotic option is almost never the right one for a small team. Boring compounds.

Visibility first, enforcement second. Every new control gets shipped in observation mode first. The WAF starts in detection-only mode: it flags but does not block. The alert rules start noisy and get tuned down. This is deliberate. Turning something on in blocking mode without watching it in shadow mode first is how you spend a Saturday night rolling back a security control that broke real users.

The real work is fighting drift. Grouping alerts. Tuning false positives. Making sure the log-shipping pipeline still has all its expected labels a week after a version swap. Everything drifts: certificates expire, a backup file with a .conf extension gets loaded by mistake, a container gets rebuilt from a base image that no longer has the module you assumed was there. The only defense is a machine you can point at production and ask one question: is it still true? And the only defense against forgetting to run that machine is putting its output somewhere it stares at you every week. This unglamorous work is 80 percent of the actual job. The other 20 percent is picking the tools.

Publish the shape, not the coordinates. The series that follows describes the pattern of every control I have built: the shape, the reasoning, the tradeoffs. It does not publish the specifics. No exact rate limits, no cookie names, no endpoint paths, no thresholds, no hostnames, no IPs. Those are the details that help an attacker map an environment and help nobody who is trying to learn from what I have done. If you want to build the same thing, the shapes are enough.

What each part covers

Part 1: The WAF Monitor. How I built a live view of every attack the system absorbs, what the numbers looked like once I had them, and the drift audit that came out of actually watching the traffic for the first time. The story of the two-WAF layer cake at the edge, why one is not enough, and why a monitor is the piece that turns a WAF from a checkbox into a control.

Part 2: The Identity Layer. The morning I realized the door had no lock. Why access tokens without refresh tokens are theatre. Why 2FA that stores backup codes in plaintext is worse than no 2FA. How email-match-only OAuth removes the entire "someone signed up with a new account" attack. The design decisions I would defend, and the ones I honestly would not.

Part 3: The Nervous System. The night the disk filled up at 4pm and I found out at 9pm. What real observability actually looks like when it is running: two hosts, a private tunnel, dashboards as code, and alerts that reach the phone before users do. Why "just check the logs" was never a strategy, and why silent regressions in the observability plane are a uniquely painful class of bug.

Part 4: The Pre-Production Gates. The checks every change clears before it is allowed into production, and the story of the first time the gate earned its keep: five red runs on the day a new app was onboarded, every one of them a real vulnerability fixed before a single container shipped. Why fail-closed changes the incentive in a way a report never does. How the same bar gets enforced across apps that deploy in completely different ways, and how the chain continues past the build with signed artifacts, verified at the moment they run. Plus a generic verification-script template, free to lift into your own stack, that plugs the drift-detection hole this whole series keeps circling back to.

Read them in order or jump to the one you care about. Each part stands alone.

What this is not

This is not a tutorial. There are no copy-paste able configuration files, no exact package versions, no annotated screenshots of my dashboards with the interesting bits visible. Those documents exist and I keep them internal, for reasons that will be obvious by the end of Part 1.

This is not an academic survey of security best practice. There are three or four decisions in this stack that a bigger team, or an infosec purist, would make differently. I say so where it matters.

This is not a claim that the setup is finished. It is not. The point of the series is not "look at this perfect system." The point is "look at what a small team can actually build and defend."

Who this is for

- Solo engineers or two-person teams running a production system that matters. If the system going down means real work stops, this is written for you.

- Founders whose engineer is also the security person and who want to know what "good enough for now" looks like from the inside.

- Larger-team security engineers who are curious how a serious posture looks when it has to be built and maintained by one person.

- Anyone building a multi-tenant or customer-facing product and wondering where to start on security without hiring for it first.

- Engineers with a serious independent project. If you are using a real system, run on your own terms, as your hands-on education in security and operations, this series is one long field note from someone doing exactly that.

If you have done something better, or seen me miss something obvious, I would rather hear that than a compliment. The series is the write-up. The comments underneath it are the review.

Part 1 is next. It starts with the three minutes it took, after DNS first pointed at my server, before the first automated probe showed up in the access log.

A note on context: everything here describes independent projects run under Mannat AI Labs. Views and mistakes are entirely my own.