closure.fast

The science
behind Closure.

Why a green suite stops meaning done, what the engine does about it, and which of its claims are tested rather than argued.

Background, not instructions. The field guide covers the workflow. Where this page cites published work, the figures are as the papers report them; they were read, not replicated. Where it cites Closure’s own results, the runs are the engine’s, on real repositories, and the last section says what they do and do not establish.

Why done stopped meaning done

The stopping condition was economic, and it is gone. Software used to finish because the next change cost something: a person’s afternoon, a review, a deploy. Coding agents drove the marginal cost of one more change toward zero, and a project that can always afford one more improvement never reaches a last page. The agent is not wrong to keep going. Nothing in its loop says stop.

Tests certify what was written down. A passing suite is evidence that every assertion someone wrote still holds. It says nothing about the behavior nobody wrote an assertion for: the recovery flow, the expired invitation, the second request that creates a duplicate. Dijkstra’s remark that testing shows the presence of bugs and never their absence is usually quoted about correctness. It applies with more force to completeness, because an unwritten requirement has no test to fail.

Coverage does not close the gap. Inozemtseva and Holmes (ICSE 2014) report, across five large open-source Java systems, a low to moderate correlation between coverage and a suite’s ability to detect faults once the number of tests is controlled for, and no extra insight from stronger forms of coverage. A line can execute under a test that would pass no matter what the line did. Coverage measures reach, not sensitivity, and sensitivity is the property a proof needs.

Sensitivity can be measured directly. Mutation testing, proposed by DeMillo, Lipton and Sayward (1978), plants a small fault and checks whether the suite notices. Just et al. (FSE 2014), on hundreds of real faults from open-source projects, report a statistically significant correlation between mutant detection and real-fault detection, independent of code coverage, and also real faults that none of the studied mutants resembled. The technique is expensive when applied everywhere and blind about which mutants matter. Both problems shrink when a requirement says where to cut.

Requirements do not survive without a home. Gotel and Finkelstein (1994), from studies with over a hundred practitioners, locate most of what gets blamed on poor traceability in the record of where a requirement came from and who asked for it, before any specification existed. In agent-driven projects that record is thinner still, since the person who knew the requirement often never typed it. A requirement that exists only in someone’s memory of a design document is one re-clone away from gone.

Why the mechanisms work

A catalog separates the denominator from the numerator. Closure’s requirements come from a versioned catalog that is written and reviewed by people and shared across projects, never derived from the code being graded. That one rule is what keeps the verdict falsifiable: a project can fail its catalog. A tool that generates requirements by reading the implementation cannot produce a failing grade, because the standard is the thing under test. The catalog also amortizes: an invitation flow has been built thousands of times and still gets expiry, revocation and replay wrong, and the obligations for it need writing once.

Closure is a subtraction, and both terms are mechanical. A selection names what the product owes in the catalog’s terms. Expansion follows dependencies and applies policies to a fixed point, so “teams can invite people” carries its email, identity and tenancy consequences whether or not anyone thought of them. The result is deterministic: the same catalog version, selection and commit yield the same obligation set, byte for byte. What remains is that set minus what the repository proves.

Evidence decays by construction. Every proof records a digest of what it depended on: the files the test executed, the extractor that read the result, the wording of the obligation it answered, and the perturbation that was declared to check it. Any of those changing expires the proof, and the obligation returns to the plan without anyone asking for a re-run. The dependency set is measured, not parsed: for a language with a resolvable import graph the graph is used, and for any language whose toolchain emits coverage the files that actually executed are used, which also catches a spawned service or a dynamic import that no static analysis would.

A negative control is one mutant, aimed by the requirement, graded by someone else. For critical obligations a passing test is not admissible on its own. The control declares the mechanism the test is supposed to protect; the harness copies the repository, runs the one named test, applies the declared perturbation, runs it again, and requires the second run to fail. A control the worker merely reports is recorded as attestation and graded weaker; only what the harness watched happen counts at full strength. Aiming by requirement is what makes the mutation affordable, and grading by an independent harness is what makes it evidence rather than a claim.

Admissibility follows the claim. A route existing and a route behaving correctly are different statements. Static evidence proves structure at full strength and behavior only at a disclosed fallback strength, because a generated surface can satisfy the shape of a behavior it never implements. The distinction is derived from what the obligation asserts, not from what the worker labels the evidence.

The score is a floor, not an average. A product’s level is the worst of its capabilities, and a capability is credited only for tiers the catalog actually populated and everything below them is proven. A polished corner cannot raise the number, and thinning the catalog cannot either. Done is then a predicate rather than a feeling: every obligation required at its target proven, every capability at target, nothing resting on self-attestation, no unresolved gap, and something owed in the first place. It holds for a scope at a commit and moves when the facts do.

Tested versus extrapolated

The mechanisms above are supported separately by the work cited. Closure combines them in ways that work did not test. This is what the engine’s own runs establish so far, and what they do not.

  • Tested: the loop runs on real code, including its own. Three subjects to date: a small fixture, two production applications, and the engine itself, measured against a catalog authored for it. Every proof in those verdicts rests on a control the harness verified in isolation against the one test it names.
  • Tested: controls find defects a green suite missed. Verifying a control on one subject exposed a path-traversal hole in a proxy nothing else was watching. Obligations with no implementation to test forced two product changes on another, including an authentication link that had been reusable for hours. All three were fixed with the tests kept. These are findings, not a rate.
  • Tested: a control refusal is a claim, not a verdict. Across those runs several refusals turned out to be a perturbation aimed at the wrong layer, one of two redundant guards, or a cache serving the unperturbed code. Each was a finding about the control. The harness distinguishes a perturbation that never reached the test from a test that survived it, and cannot yet detect a false red on its own.
  • Extrapolated: efficacy. Whether a closure run on an unfamiliar repository with a green suite reliably surfaces a real gap, at a cost its owner would pay again, is the claim that matters and it is not yet measured. The bar for it is written down before the trials rather than after, and results will be published.
  • Extrapolated: catalog completeness. Done is decidable relative to a catalog version, and the catalog’s own completeness is unfalsifiable from inside it. The engine converts one open question into a closed one plus a slower, human one. The risk moved; it did not leave.
  • Not yet measured: end-to-end scenarios. The model includes business paths that must traverse end to end. The runner that would traverse them is not built, so a selection that declares scenarios is measured by their individual obligations rather than by the path.

What Closure does not claim

That a verified control proves correctness. It proves the named test notices the named perturbation, nothing wider.

That DONE means finished, safe or bug-free. It means the declared scope is satisfied at this commit by evidence the engine could check.

That it knows every requirement your product needs. Coverage is bounded by the catalogs, selections and extractors in play, and what those do not describe is invisible to the verdict. Human review still finds what the inputs do not.

That it writes code. A coding agent or a person does. Closure decides what is owed, what is proven, and what is next.

References

Read the field guide