How do you verify an AI coding agent actually did its job?
The agent’s job is a requirement, not a diff. Verifying the job means checking the result against what the feature owes, written down before the agent started, with proof that the check would fail if the behavior were wrong. “The tests pass” counts only when the tests were not the agent’s to define.
Three claims, one word.
When an agent says done and shows a green run, the output demonstrates two things and asserts a third.
A test written in the same session as the implementation takes its expected values from that implementation. The test confirms the code agrees with itself. A review of the diff can confirm what is there looks right; it cannot see what is missing, and done is mostly a claim about absence.
- The code ran.shown by the run
- The tests the agent wrote agree with the code.shown by the run
- The behavior you asked for exists.asserted, not shown
The three checks that make it real.
Each one removes the agent from a seat it should not hold: author of the requirement, judge of the test, and keeper of the verdict.
Closure Engine is a local tool that does exactly these three things. The rest of this page shows each on the small Next.js and Postgres app the engine ships with. Everything shown is recorded output.
- 1A requirement the agent did not write
What the feature owes, in checkable sentences, from a source the agent cannot edit.
- 2Proof the test would fail
Break the behavior in a scratch copy and run the one test that claims to protect it.
- 3A verdict pinned to the commit
Every requirement graded proven, unproven or stale at one revision. Done is nothing required left unproven.
One passing test, recorded.
The tutorial starts from a test an agent might reasonably write. It asserts an administrator is allowed, and it passes.
The requirement is that administrator actions are protected, so the engine removes the protection in an isolated copy and reruns the same test. It still passes, which means the green run was never evidence of the job. After the guard checks the role and a second assertion covers a guest, the same control fails under the break and the release check opens.
Why a green suite can lie- as written
protects administrator actionspassesexport const allowed = role => Boolean(role); - perturbed · unconditional-access
- export const allowed = role => Boolean(role); + export const allowed = role => true; - rerun
protects administrator actionsstill passesNOT VERIFIED · vacuous
- repaired guard and test
+ export const allowed = role => role === "admin"; + assert.equal(allowed("guest"), false); - rerun, perturbed
protects administrator actionsfailsHARNESS VERIFIED
- release-check
RELEASE(closure-tutorial) = BLOCKEDRELEASE(closure-tutorial) = READY
closure demo as shipped, then with its solution applied, engine 0.2.2. Full outputWhat the agent receives instead of a prompt.
On the larger fixture, selecting one sentence, “teams can invite people”, produces 58 obligations.
The agent is handed one at a time as a task envelope it cannot rewrite: the requirement, why it is owed, the proof that will be accepted, and how it will be graded. Two of those sections, verbatim.
REQUIREMENT
For every actor A and tenant T where A is not a member of T, no operation performed by A returns or mutates any audit_event owned by T.
PROOF THAT WILL BE ACCEPTED
evidence kind test.property (primary), test.example (fallback)
negative control REQUIRED. This obligation is critical, so a passing
test is not admissible on its own. You must also show the test FAILS when the
implementation is perturbed. Suggested perturbation: drop.authz
Record which perturbation you used and that the test failed under it.
name the test so that it contains the literal string [a3bb0264]
(that is how a passing test binds back to this obligation)closure next on fixtures/example-saas, the fixture Closure ships with. The whole envelopeA verdict is a list of clauses.
Done is not a summary. It is a property of the repository at one commit, and the failing clauses name what is still owed.
The test named protects administrator actions above and the 58 obligations here are graded the same way: proven, unproven, or stale because the code they were proven on has changed since.
- every obligation required at the declared targets is proven32 unproven, of which 0 inadmissible and 2 stale
- every capability reaches its target levelAUDIT.EVENTS T0/T1, BILLING.SEAT_ACCOUNTING T0/T1, COLLAB.INVITATION T0/T3, EMAIL.DELIVERY T0/T3, IDENTITY.ACCOUNT T0/T4, ORG.MEMBERSHIP T0/T1, SCENARIOS T0/T1
- every required scenario traverses end to end0 of 3 traversable
- advisory: proofs at full strength3 proven at fallback strength (weaker kind, partial surface, or self-attested control)
- 7 clauses satisfiedsomething is owed · no obligation rests on unverified evidence · no capability is targeted deeper than the catalog specifies · no unresolved gap objects · no contested obligations · no blocking provisional obligations · no pending weakening or exclusion requests
This verdict is a property of the repository at this commit, not of this run.
closure done; only the fold is editorial. The full ledgerWhat this does not establish
A requirement list is only as good as its author. Closure grades against a catalog of what a feature owes; a requirement nobody wrote is not checked, and the engine says so rather than guessing. A negative control proves a test is load-bearing for one named failure, not for every failure. And a verdict is about the declared scope at one commit. None of this replaces a person deciding what the product should do. It replaces trusting the agent’s word for whether it did.