Ernie · Engineer and independent researcher, AI coding agents

I find where AI coding agents break, and build the checks that catch it.

Agents now write much of the code that ships, and passing tests is not the same as working. My work measures that gap and turns it into checks a team can run every night.

Background

Earlier · Full-stack: Shipped products for North American startups — React and TypeScript over Rails — and led the teams that built them.
2022–26 · archive.com: Built the Claude Code and Codex harness the whole engineering team works through, and LLM features with evals behind them.
2026 · Research: Agents on a real codebase raised questions nobody had measured. Measuring them became the work.

Paper · peer-reviewed · open access

Measuring the Wrong Number

AgenticDev 2026, a workshop of the IEEE/ACM ASE 2026 conference

It tested the two most-starred tools that promise a cheaper coding agent by having the model write less. What the model writes is only about a fifth of the bill, so even the advertised 65% cut would take only about 13% off it. What actually happened:

Advertised: 65%; token savings the tools claim
Measured: ≈0; change in the bill, 140 correctness-gated runs

Read the paper (ACM) · the blog post

Open source · the checks

vigiles: Checks a coding agent's setup (CLAUDE.md, AGENTS.md, skills, hooks) against the repo it runs in: every linter rule it cites exists and is turned on, every file and script is real.
paperlint: A linter for LaTeX papers that catches what gets a submission sent back: page limits per venue, a silently swapped font.

Blog

Where AI autonomy breaks

2026·07·11I checked 57 repos whose CLAUDE.md bans the any type. One in eight turns the rule off.Of 57 public repos whose CLAUDE.md or AGENTS.md tells the coding agent never to use the any type, only 46% back the ban with a hard lint error; one in eight…2026·07·06Verification Is All You NeedAn AI agent runs unattended exactly as far as you can verify its output cheaply, deterministically, and many times a night.2026·06·05Passed every test. Still took down prod.It passed every test and still took down prod — the bug wasn't a bad line, it was a state nobody enumerated.

All writing →

Contact

Hiring for agent reliability, evals or developer tooling? Write to me.

Or cut your team's coding-agent bill — free first look, details here.

Open to research and applied-AI engineering roles in the US, with visa sponsorship.

contact@zernie.com

LinkedIn GitHub Telegram

Appendix — the full stack, if you're curious

ShipReact · TypeScript · Next.js · Redux Toolkit · Ant Design · Shadcn · Ruby on Rails · PostgreSQL · Redis · Node.js · GraphQL · Elasticsearch · Docker · GitHub Actions · Vercel — ten years, mostly for North American startups.

VerifyState machines · covering arrays · model checking · property tests · database constraints

AgenticClaude Code · OpenAI Codex · Model Context Protocol · GitHub Spec Kit · Vercel AI SDK · Braintrust

NoteListed because they're real, not because they're the point. The point is upstairs.