A month ago I wrote about the workflow I use to ship software with AI agents. The part readers kept asking about was the gate in the middle, the thing that sits between "every task is ticked off" and "it shipped". People asked if they could use it too. So last week I pulled that stage out of my private setup, generalized it until it runs on any software project, and put it up as a free, open-source tool called Gatekeeper.
Releasing it made me more nervous than building it. A tool you run on yourself only has to work. A tool strangers install also has to explain itself, and part of explaining myself is being straight about where the ideas came from. Some of the best parts of this gate are not mine. Other people published the methods first and I adapted them, and their links are all through this post. If you take one thing from it, make it those links. The originals are worth more than my version of them.
The lie it catches
An AI builder works through a task list and ticks the boxes. The list says the feature is done. The code says otherwise. Twice I have caught a real fix sitting uncommitted while its checkbox said done. Twice that I noticed, anyway. I have no idea how many times it happened and nobody looked.
Nothing malicious was going on. The builder was optimizing for "list complete" because the list was the only thing it could see. If your whole world is a checklist, done is whatever the checklist says.
A release may not claim what is not true.
So before Gatekeeper believes a checkbox, it grades the list. Every claimed task gets checked against the code that is about to ship, and each one comes back as true, partly true, or false. A false claim stops the release on its own, and no reviewer vote can talk it out of that. I can't promise the grading is perfect. It doesn't have to be. A verdict computed from evidence beats one someone argued their way into, in my experience so far.
Six phases, in order
The gate runs as a fixed sequence, and the order matters.
The machines go first. Tests and the build run for real, and the tooling itself records the numbers. An AI never reports its own test results, because "tests passed" from the thing being tested is not evidence, it is a claim. Anything red here and the review never starts.
Freeze the work. The exact version of the code gets locked, the same idea as the seal in my workflow. Every reviewer judges that frozen copy and nothing else. If the code changes mid-review, the whole review is thrown out and starts again from the top.
Grade the claims. A fresh conversation that had no part in the work reads the task list and checks each box against the frozen code, with file and line evidence for every grade. It checks the bragging numbers too: the test counts, the pass rates, the references to runs.
Wake the fleet. Seven reviewers, each a brand-new conversation that has never seen the builder's reasoning, summaries, or excuses. They run in parallel, so seven reviews cost about as long as one. Why seven? Partly risk-based, and the rest of my thinking is below.
Re-test the findings. Reviewers produce signal and noise, so the gate takes every serious finding and reproduces it personally before it counts. A demonstrated defect stands even if every other reviewer said the code looked fine. A finding nobody can reproduce stays visible in the report but does not block the release.
The verdict and the receipt. The outcome follows from the evidence, not anyone's mood. A proven critical defect fails it. A false claim fails it. An unresolvable disagreement stops the line and calls a human. A pass earns a receipt, a one-page record of every check that ran, who reviewed, and what was found, bound to the frozen copy it judged. The repo states the rule that holds the whole system up: no receipt, no push.
The seven reviewers
Depth follows risk. A documentation change wakes two reviewers. A change that touches security, permissions, or payments wakes all seven. The exact number seven is a judgment call on my part. Fewer felt thin, more felt like paying for overlap, and I reserve the right to be wrong about it.
| Reviewer | The one-line job | What it catches |
|---|---|---|
| Red Team | Break in on purpose, and find what is simply wrong | Holes nobody thought to guard |
| Code Reader | Judge the code, never the summaries | Logic errors, broken tests |
| Security Reviewer | Sweep for known weaknesses | The vulnerability checklist |
| The Outsider | Use the product through its front door, docs only | Instructions that don't work |
| Auditor | Read across the whole estate, never inside one file | Duplicate logic, broken hand-offs, claims that outran the code |
| Ponytail | Find what to cut | Over-engineering, wasted complexity |
| Architect | Question whether the thing built was the right thing | The right fix to the wrong problem |
The product fit review
Six of the reviewers ask whether the work was done right. The Auditor asks a different question: is the whole product fit to ship? It is the one reviewer that ignores single files on purpose, and its method started life as an audit I wrote for my own projects before the gate borrowed a slice of it for every release.
The idea is simple to state and expensive to fake. The worst problems don't live inside any one file. They live between the parts. Two live versions of the same rule that have quietly drifted apart, so the website rounds money one way and the refund code rounds it another. A security check bolted onto one door and forgotten on its identical twin. A button wired up in the interface that connects to nothing on the server. Read each file alone and it looks fine. The product is broken anyway.
So the Auditor reads the space between. Every implementation of the same rule, gathered and compared. The hand-offs between parts, traced end to end, because checks go missing exactly where one component hands work to another. Anything reachable on one side but not the other, listed as an orphan. And the promises get audited too: the version notes, the docs, the README. Every load-bearing claim is checked against the frozen code, because the false-checkbox problem loves to hide in exactly those pages.
Findings are graded against a bar, not against perfection. The same gap can be a shrug for an internal tool and a release-stopper for something customers touch, so the reviewer has to name the bar it is judging against. Every finding carries its evidence, the file and the line, and the report ends with an honest coverage statement: what got read closely, what got scanned, what was never opened. That last part matters more than it sounds. A review that cannot tell you what it did not look at isn't a review, it's a vibe.
Where the ideas came from
This is the part of the post I most want you to read. A lot of the best thinking in this gate is borrowed from people who published their methods openly. Pretending otherwise would just be theft with better manners. The repo ships untouched copies of the original skills in a resources/ folder, next to a README that maps every reviewer to its source, so the lineage is something you can check rather than something you're asked to believe.
Three reviewers are my own designs: the Red Team, the Security Reviewer, and the Auditor. The Auditor's method is the biggest piece of my own thinking in the gate; the product fit review above began as a standalone audit skill, and the repo ships the original of it in resources/ next to the borrowed ones, because a method should be checkable the same way its sources are. The other four stand on other people's published work. If one of these methods is useful to you, the original author is the one who earned your star.
- Code Reader runs on the CodeRabbit CLI, which does the actual code-review engine work over the sealed range of changes. I did the wiring. The engine is theirs.
- The Outsider takes its stance from the
scrutinizeskill in thananon's 9arm-skills: intent first, then trace and verify. I retargeted it from code paths to the documented user path, so it reviews the product the way the docs promise it, not the way the code happens to behave. - The Architect leans on evidence discipline from Jacob Rothfield's Repo Review Agent prompts: numbered evidence for every claim, a defect only counts when two independent methods confirm it, and an active hunt for contradictions. That toolkit is why the Architect demands proof instead of opinions.
- Ponytail is adapted from Dietrich Gebert's ponytail (MIT license), a skill whose entire worldview is that most code should be shorter. I kept the name because I like it.
Two details in the fleet are mine, and I'll admit I'm fond of both. Ponytail writes one line per finding, and if there is nothing to cut it replies "Lean already. Ship." and stops. Its best possible outcome is that your diff gets shorter. And when the Security Reviewer reports nothing, the gate doesn't take its word for it. The gate plants a known-bad pattern in the code and runs the reviewer again. If the clean bill misses the planted bug, the bill gets discarded and the pass re-runs. I don't know whether that trick survives a truly adversarial reviewer, and I'd like to find out. So far it has caught every lazy pass I've thrown at it.
Why fresh conversations matter
This piece sounds mystical and is actually mechanical. When the same conversation writes code and then reviews it, it reviews its own reasoning, and it agrees with itself. Every time I've tested that, the model favored its own output. The fix costs nothing. Start every reviewer in a brand-new conversation that receives only the frozen code, the task list, and its written job description. The builder's confident summary never reaches them, so it can't infect them.
That is the whole trick in the title. Audit firms have run a version of the same separation-of-duties rule for about a century, which I find reassuring. The people who invented it were worried about money. I'm worried about an eager machine that honestly believes its own checklist. What changed is the price. A second opinion used to mean a second salary. Now it's a second conversation that onboards in thirty seconds.
Fitting it to your project
Gatekeeper adapts through a short list of settings: where your task lists live, what commands run your tests and build, what package it is releasing. There is a setup guide for humans and another one written for agents, because the realistic way this gets installed is you hand your coding agent the instructions and it wires the gate itself.
The release cut is part of the gate too. After a pass, the tooling bumps the version, drafts the notes, tags, and pushes, and the hosting pipeline does the actual publishing. If publishing fails halfway, the fix is to re-run the failed job on the same version, never to release a new version over the wound. Repairs get two attempts, then the gate calls the owner instead of spinning. Unbounded retries were tempting. They are also how a tool burns an afternoon making the same mistake with confidence.
What I'd tell a business reader
If your team ships software built with AI help and someone else signs off on it, you probably already have this problem, whether or not my gate is the answer. I can't claim it's the right fit for everyone. Whichever tool you end up with, I'd look for:
- The checker is never the writer.
- Claims get graded against the actual code, not against a confident summary.
- The verdict comes from rules, so it can't be argued away at 6pm on a Friday.
- The release leaves a paper trail you can show a client.
I run every Seepient release through this gate, the same assistant I use human approval gates for. The gate is free and open source, and wiring it into a project takes an afternoon, maybe two the first time. Book a call if you'd rather I did that part for you.
The repo: github.com/hashangit/gatekeeper. The originals by CodeRabbit, thananon, Jacob Rothfield, and Dietrich Gebert are linked above and credited in the repo's resources/README.md. Read theirs first if you want the ideas without my layer on top.
