Ernie
← Back to articles
/

Passed every test. Still took down prod.

aitypescriptsoftware-engineeringstate-managementfeature-flagstypesformal-methodspostgresformal-verificationrails

The combination no test ever ran

On August 1, 2012, Knight Capital pushed a deploy to eight servers. One of them didn't take the new code. That single mismatch — eight servers, new code or old, 256 possible configurations and exactly one of them ever tested — woke a decade-dead code path in production and lost the firm $440 million in 45 minutes.

256 possible configurations. They'd tested exactly one. $440 million, gone in 45 minutes.

No single line of that code was wrong. The bug was a state nobody designed, a combination no test ever ran — and you've shipped the small version of it. The spinner keeps spinning over content that's already on screen. The error toast slides in over a page that loaded fine. The empty state flashes for a frame before the data it already had paints.

Three fields behind it — isLoading, isError, and data that's either there or not. Eight combinations between them; four mean something — idle, loading, loaded, failed. The other four are the bug: the spinner still turning while the data sits right there, loading and errored at once, the render that's somehow none of them. Your code has to survive all eight, because the type says all eight exist.

That shape has a name: it's useQuery's old return type — and the footgun was common enough that React Query itself replaced those booleans with a single status field (pending | error | success).

State explosion

useQuery · 3 fetch booleans

8 representable4 valid4 nonsense

Those three fields are tangled — most of the eight states are nonsense you can delete outright. Feature flags are the opposite: independent by design, so every combination is a real timeline your code runs in production. Open your dashboard, count the flags older than six months you "meant to clean up." I'll wait.

Now multiply them. Each flag forks the timeline — on or off. One flag, two timelines. Two, four. Three, eight. It doubles every single time.

Find your team's number — 20flags1,048,576timelines — and every one of those is a path your code can take in production.

Nobody tests a million timelines. QA signs off on the dozen you flip on purpose; the suite goes green. The rest just exist — in production, never enumerated by anyone, waiting for the right user to load the right page in the right order. That's Knight Capital's 256 configurations, one layer up — the same trap, now with your feature flags instead of eight servers.

And it rarely looks like a bug in the code — no crash, no stack trace, no failing test. It passed everything you wrote; it broke the one combination you didn't. It looks like "customer X is seeing the wrong price." You chase it as a race condition until someone pulls the flag history and sees this exact three-flag combo has never happened in the system's life, never will again after Friday's release, and is breaking precisely one invoice right now. Nobody designed that state. Every flag did what it said on the toggle. The combination was a timeline no human ever imagined.

Nobody told me this in school. That bug you couldn't reproduce usually isn't a logic error — it's a state you forgot existed, a branch of the timeline nobody meant to grow.

And it isn't just flags. One invisible thing explodes the state space in a way nothing else does — and you've spent more of your career than you'd like debugging what it causes: the NullPointerException, the undefined is not a function, the one check you forgot on the one path that mattered.

It's null. Its own inventor, Tony Hoare, calls it his billion-dollar mistake — he slipped it into a type system in 1965 "simply because it was so easy to implement," and spent the next fifty years watching it cause "innumerable errors, vulnerabilities, and system crashes."

Here's why it's the worst offender. A boolean you add on purpose, and you can see it coming. null the language adds for you — quietly, to almost every field at once, turning each T into T | null. It's a flag stapled to every value you have: it doubles that field's states, and multiplies across all of them. A form with 15 optional fields is 32,768 presence-and-absence states before you've written a line of logic. (Your largest form — you know the one.) A user model with isAdmin, isVerified, isBanned, isDeleted has 16 states, about four of which make sense — and at least one database row somewhere has all four set to true. Out-of-order webhooks, a retry firing through a half-finished write, a message that arrives twice or never — same shape every time. The set of states your code can be in dwarfs the set it actually handles.

And you can't test your way out — the math is against you. Knight was the cheap version; a 2015 study of production failures in major distributed systems found that about a quarter trace to configuration combinations nobody tested — the same trap at the infrastructure layer. The state space expands by default; your ability to verify it grows linearly at best. The gap widens every time someone ships a feature.

TL;DR — the two cheapest cuts. Most prod bugs aren't bad lines; they're states nobody enumerated. You can't test your way out — the combinations explode — but you can shrink the state space so the bad states can't exist:

  1. Discriminated union — a type that's always exactly one of a few shapes, never a mix; impossible states won't even compile.
  2. Database constraint — enforce what the type can't, where nothing can bypass it.

Start today: take one boolean-soup model and collapse it into a union. For covering arrays, model checking, and why the spec is now cheap to write — Part 2.

Now the AI is branching it too

An LLM writes a growing share of your code now, faster than review can keep up. Generation scales; careful reading doesn't. The branches multiply faster than anyone can check them by hand, so I've stopped betting on the check. You can make the check cheaper — tighten the feedback loop so mistakes surface fast, turn your conventions into lint rules the CI enforces — and you should. But the cheapest branch to review is the one that can't exist. So prune the branches that should never exist — make them impossible to write down. A branch the AI can't spell is a bug it can't ship.

Pruning means the bad branch can't be spelled

Almost everything a type allows is invalid. The legal states are a small island; what the type actually permits is an ocean. It took me embarrassingly long to see my own defensive code for what it was — the guard clauses, the "this shouldn't happen" comments, the same validation in five different places. All of it babysitting combinations that should never have been representable in the first place.

The fix is to shrink the ocean until it matches the island. Two essays taught me the moves, from different angles:

Make illegal states unrepresentable is Wlaschin's: design the shape so bad states have no spelling. If a state can't be constructed, it can't be passed in, returned, or stored. Parse, don't validate is King's: check incoming data once at the boundary, hand back a type with the invariant baked in, and the compiler carries it from there.

Both cut at the door. The toolkit runs deeper from there.

A huge circle packed with bare dead twigs wraps a small, clearly bounded circle of living leaves at its center; the blob prunes the dead twigs with scissors beside a tidy pile of cut deadwood — almost everything you can spell is invalid, only the small inner set is valid.

Kill the booleans

The classic example. Nobody designs this. Three boolean lifecycle fields accrete one feature at a time: isDraft this quarter, isArchived two quarters later, each added by someone who never looked at the other two (and who has since moved to another team):

interface Order {
  isDraft: boolean
  isPublished: boolean
  isArchived: boolean
}

Three booleans. Eight possible states, and ✗ nonsense. The other five — published and draft at once, archived and draft and published — are nonsense your code still has to handle, because the type says they exist.

You don't have to swear off booleans. The test I use is Matt Pocock's: bad booleans store state; good booleans are derived from it. A stored isPublished you set by hand is the disease; const isPublished = status === "published" is fine: it's computed, so it can't contradict anything.

The fix is a discriminated union — a type that says "this value is exactly one of these shapes, never a mix." Every major language has one; what changes is how hard the compiler makes it to get wrong. (No Sorbet in your Rails project? The DB constraint section is the version that enforces the same rule on any stack.)

// TypeScript
type Order =
  | { status: "draft"; content: string }
  | { status: "published"; content: string; publishedAt: Date }
  | { status: "archived"; content: string; archivedAt: Date }

Eight states → three, and the nulls went with them. The boolean version needed a nullable publishedAt: a real date on published orders, null on drafts, and nothing stopping a draft from carrying a stray timestamp or a published order from carrying none. The union deletes the field from every variant that shouldn't have it, so publishedAt exists only on a Published order. No nullable column to forget, no illegal combination, no billion-dollar mistake left to check for. And "published and draft at once" never had a shape to live in. (No union handy? The one-level-down version is a language that makes null opt-in — Rust's Option, Kotlin's ?, TypeScript's strictNullChecks — so absence becomes a case the compiler forces you to name.)

TypeScript, Python, and Rust make this a compile-time hard stop. Go and C# enforce the structure but only refuse an illegal value at runtime; there's no exhaustiveness proof at compile time. That difference matters when the union grows:

It's the same cut for the spinner from the top: model the fetch as one of idle | loading | error | data and "loading and errored at once" has no shape to live in — not caught at runtime, just unbuildable.

And when you switch on this, the compiler makes you handle every case:

// TypeScript — miss a case and it won't compile
function render(order: Order) {
  switch (order.status) {
    case "draft":     return renderDraft(order.content)
    case "published": return renderPublished(order.content, order.publishedAt)
    case "archived":  return renderArchived(order.content, order.archivedAt)
  }
}

Add a fourth status and this stops compiling until you handle it. Rust enforces the same exhaustiveness as a hard error; Python's assert_never and Sorbet's T.absurd catch it at type-check time; Go and C# only warn or throw at runtime (neither proves a class hierarchy exhaustive).

And the same idea works at the database level. SQL is weaker than a type system, but it's enforced on every write, by every client, forever:

▶The same rule as a Postgres table
-- PostgreSQL — the database refuses to store illegal states
CREATE TABLE orders (
  id           UUID PRIMARY KEY,
  status       TEXT NOT NULL CHECK (status IN ('draft', 'published', 'archived')),
  content      TEXT NOT NULL,
  published_at TIMESTAMPTZ,
  archived_at  TIMESTAMPTZ,
 
  -- Same invariant as the discriminated union, at the persistence layer.
  -- Archived drops `published_at` because archive is reachable from draft
  -- (never published) — so the union can't carry it on `archived` either; the
  -- two layers agree. If archive only ever followed publish, keep it on both.
  CHECK (
    (status = 'draft'     AND published_at IS NULL     AND archived_at IS NULL) OR
    (status = 'published' AND published_at IS NOT NULL AND archived_at IS NULL) OR
    (status = 'archived'  AND published_at IS NULL     AND archived_at IS NOT NULL)
  )
);

Your TypeScript types stop at the network boundary. Your database doesn't, and neither does CHECK. Even if some service in another language forgets the rule, the constraint catches it. We'll come back to this.

Add a fourth status? The compiler points at every place that needs updating, and the code won't build until you've handled them all.

A union collapses one field's states. But some rules span two independent fields — a voided invoice whose payment_status is somehow still succeeded — and no single type can spell that combination away, because each field is legal on its own. That needs a cut that sees both fields at once.

The cut no app can bypass

Every cut so far lives in your code, which means it stops the moment data crosses a boundary you don't own: the JSON the API just received, the row another service wrote, the message off Kafka. Your Order type guarantees nothing about any of them.

There's a word for what all these cuts enforce: an invariant, the one sentence your data must never contradict. "A deleted row never comes back." "A refunded order can't reopen." Bugs are violated invariants. The space the fence needs to protect can be astronomical; the fence itself stays small — a 2021 proof of Paxos searched billions of candidate invariants and found the rule that pins it fits in a handful of terms. The only question is where you enforce it, from weakest to strongest:

Four barriers of rising strength — a paper note, a low fence, a taller fence, a brick wall — a dead twig blows past the first three and shatters on the wall.

▶The full spectrum, as a table
MechanismWhen checkedWhat happens on violation
Comments / docsNeverNothing
Runtime assertAt call site, sometimesCrash, hopefully in dev
TestsAt CI timeBuild fails (for cases you wrote)
LintersAt lint timePR fails (for patterns you encoded)
TypeScript typesAt compile time, erased at runtimeBuild fails, but can be bypassed with as any and stops at the network boundary
Strong types (Rust, Haskell)At compile time, harder to bypassBuild fails earlier and unsafe is opt-in rather than the escape hatch
Schema validation (Zod, Pydantic) — a library that checks incoming data against a declared shapeAt system boundary, at runtimeReject input; on success, the type carries the invariant forward
Database constraintsAt every write, by every client, foreverINSERT/UPDATE rejected — the only layer no application can bypass

Most teams underuse the database here. I did too, for years, and it's probably the strongest invariant layer you have.

At a brick doorway labelled "DB", the blob snips the bare dead twigs off a branch with scissors so they drop into a pile outside, then sends the clean leafy branch through the door — the database only takes a state once the illegal parts are cut off, and no app can bypass the cut.

Each constraint deletes a whole class of bad state. And unlike your types, they hold on every write, from every client, forever, no matter which service forgot:

  • NOT NULL eliminates a state.
  • UNIQUE eliminates a class of duplicate states.
  • FOREIGN KEY eliminates orphan-reference states.
  • CHECK (status IN ('active', 'done', 'archived')) collapses an unbounded text field to three legal values.
▶Why CHECK and not a native ENUM?

Evolving the set stays an ordinary migration. A Postgres ENUM only lets you ADD VALUE; dropping or reordering one means recreating the whole type and every column built on it. text + CHECK is the trade most teams at scale make: GitLab's schema alone carries 35 enum-whitelist checks doing exactly an ENUM's job.

And the teams at serious scale already live this way:

GitLab's schema declares 2,419 CHECK constraints — 82 of them exclusive-arc checks ("exactly one of these columns is set," the database half of a discriminated union).

It's institutionalized, too: a first-class migration helper, add_multi_column_not_null_constraint, makes pushing "exactly one of these is set" into the schema a one-liner any engineer reaches for. That's what a codebase looks like once it takes the database seriously as an enforcement layer.

The part that got under my skin once I started counting: your model already has a spec for which combinations are legal. It's just split across two layers that don't talk to each other.

One side is the pile of conditional validations every non-trivial model accumulates (validates … if:, validate, conditional callbacks). GitLab's app/models carries 173 of them, Mastodon 72. That's literally a decision table — the predicates are the parameters, the rules fire on combinations nobody enumerates. But those validations drift: they're code that has to be remembered, and update_column / upsert_all / insert_all skip them entirely.

The other side is the CHECKs and partial indexes above, which can't drift, because the constraint is the enforcement. So the legal state space ends up half-declared in a layer that rots and half-declared in a layer that can't. The bugs live in the gap between them.

▶Two more constraints worth knowing: partial unique indexes and EXCLUDE

A partial unique index is the idiomatic way to enforce state-machine cardinality at the persistence layer:

CREATE UNIQUE INDEX one_active_subscription_per_customer
  ON subscriptions (customer_id)
  WHERE status = 'active';

After this, the database itself refuses to ever store two simultaneously active subscriptions for the same customer. No race condition, no application bug, no service in another language can produce that state. This isn't a clever trick either: GitLab's schema carries 163 of these state-gated partial unique indexes (WHERE status = ...), enforcing "at most one row in this state" across the codebase.

An EXCLUDE constraint generalizes the same idea to ranges and overlaps. Imagine room bookings:

-- the `=` operator class needs the btree_gist extension:
CREATE EXTENSION IF NOT EXISTS btree_gist;
 
ALTER TABLE bookings ADD CONSTRAINT no_overlap
  EXCLUDE USING gist (
    room_id WITH =,
    during  WITH &&
  );

Every booking system reimplements "no double-booked rooms" in application code, badly, with race conditions the database just refused to permit.

And if you want to know what a missing constraint costs, the cleanest case is Robinhood, 2020–21. The app showed customers false negative balances and fired 84,100 erroneous margin calls off them, drawing FINRA's record $70 million penalty for the "significant harm" it caused. Knight was the combinatorial kind — 256 configs, one tested. Robinhood is the missing-invariant kind: "never show a user a balance the ledger doesn't support" is a rule, and no layer of the stack enforced it.

That's the cut a type can't make: it held even when every layer of application code — the validation, the guard, the recompute — was wrong. Schema validation at the boundary helps; a database constraint helps more, because it doesn't trust the application to be right.

So I've stopped trying to pick the one right mechanism. Stack them. Each layer in that table cuts the state space a little; together they pin the system down to something close to "only valid states are reachable."

The cut at the door

A blob standing in a doorway like a gatekeeper: a chaotic pile of tangled dead branches outside, a tidy living leafy tree inside — parse at the door, trust the type within.

It's what Alexis King named parse, don't validate: check once at the boundary and return a type with the invariant built in, not a boolean everything downstream has to re-check:

▶Parsing vs. validating, in code
// Validation: check, hope, repeat everywhere
function isValidOrder(input: unknown): boolean {
  /* ... */
}
function processOrder(input: unknown) {
  if (!isValidOrder(input)) throw new Error("bad")
  // input is still `unknown`. You'll check again. And again.
}
 
// Parsing: check once, type carries it forever
const result = OrderSchema.safeParse(rawJson)
if (result.success) {
  const order: Order = result.data // ← invariant now lives in the type
  processOrder(order) // No checks downstream. Compiler enforces.
}

Zod, Pydantic, io-ts, Valibot — same trick: a runtime invariant goes in, a type-level invariant comes out. Pay the cost once, at the door; the compiler enforces it for the rest of the program's life.

Start cutting

Two cuts. Make one today.

Today. Find one model with three or more status booleans, or a status string with no CHECK behind it. Replace the booleans with a discriminated union, or add the constraint. One bad state that can no longer be spelled. It's the cheapest cut there is, and you'll feel it.

This week. Take the cross-field invariant your app assumes but never enforces — "a voided invoice is never paid," "an active subscription has a customer" — and push it into the database: a CHECK, a partial unique index, an EXCLUDE. The cut no service can route around.

These two cuts delete most of the bad states outright. For the combinations that survive — the space too large to type away, the sequence nobody can enumerate, the 3am bug that passed every test — that's the verification layer: Part 2 — covering arrays, model checking, and the bug that survives every test.

Prune the timeline. One cut at a time.

References

▶Sources & further reading