What this document is. Fourteen checks to run on one specification through a software factory (a pipeline that takes a spec in and produces code, tests and evidence out) before trusting it with a backlog. The checks cover what goes in, what comes out, the controls around the agents, and the measures that tell whether the output ships. They are built from four published sources: Martin Fowler's note of September 8, 2026 on the gap between generating and verifying; LinearB's 2026 benchmarks on 2.7 million pull requests from the first half of 2026 (August 4, 2026) and its full year report on 8.1 million; Kief Morris's article on martinfowler.com of March 4, 2026 on where humans belong in the loop; and the Thoughtworks Technology Radar assessment of spec driven development (November 5, 2025). How we run the checks on our own line is described at the end and marked as ours.
1. The cost of generating fell. The cost of verifying did not.
Martin Fowler's note of September 8, 2026 quotes Christian Catalini's observation that we are vastly reducing the cost of generating things but not the cost of verifying them, and that the new automation boundary is measurable versus non measurable work. Fowler's reading for software: too much of what makes work effective sits behind slow feedback loops or judgment, so teams automate heavily while measuring incompletely, and the short term dashboards go up while the long term goes wrong. Catalini calls those gains counterfeit utility. Fowler's conclusion is about incentives: people have to invest more in verification than in generation, or they are driving a car with a powerful engine and weak brakes. One more line from the same note, from Catalini: build a history of decisions, not a gallery of outputs.
A software factory is the purest case. It maximizes generation by design. Whether it is worth anything depends entirely on the verification built around it.
2. What the data says about factory output
LinearB's 2026 mid year benchmarks (published August 4, 2026; 2.7 million pull requests, 7.5 million commits, 83,000 developers, 253 organizations, first half of 2026) measure the factory directly. Even at elite organizations, agents author at most 4.7% of pull requests; the autonomous part of the pipeline opens one pull request in twenty. Pull requests overall merge within 30 days 90% of the time at elite organizations, 86% at good ones and 81% at fair ones. Agent opened pull requests merge at 79%, 58% and 37%. LinearB's full year report on more than 8.1 million pull requests across more than 4,800 organizations puts the acceptance of AI generated pull requests within 30 days at 32.7% against 84.4% for human written ones, with AI pull requests waiting 4.6 times longer before review and being reviewed twice as fast once picked up.
LinearB's CTO, Yishai Beeri, gave the mechanism on August 14, 2026: an agent that pulls a ticket, implements it and pushes a pull request with no human owner produces three pull requests where one survives, because nobody chases approvals or drives the change to merge. Shipping code is an act of ownership, and detaching the human removes it. The lever that lifts yield in the same data is an automated review pass before a human opens the pull request, worth up to five percentage points across all pull requests and up to seven at the fair tier.
The same post records the case against the factory: one founder ran his company's work through a pipeline with humans at the edges and shut it down after roughly three months of codebase rot, because nothing in how models are trained rewards keeping a codebase maintainable, and the erosion was one no test caught. LinearB's own framing, from a roundtable it hosted on September 4, 2026: every change the factory merges becomes the environment the agents work in next quarter, so the merge is a poor place to stop measuring. Rework rate, review load and how often merged work comes back are the proxies that move when quality slips.
3. Where the humans sit
Kief Morris, on martinfowler.com on March 4, 2026, separates the why loop (turning ideas into outcomes, which humans run) from the how loop (building the software, with its inner loops of specifying, generating and testing). Humans out of the how loop is vibe coding and, in some interpretations, spec driven development. Humans inside the innermost loop, inspecting every generated line, become the bottleneck, and he suspects that is part of why productivity studies show mixed results. His answer is humans on the loop: build and maintain the harness, the collection of specifications, quality checks and workflow guidance that controls each level of the loop, and when the output is wrong, fix the harness rather than the artifact. Agents produce better code when they can gauge the quality of their own output instead of waiting for a person to check it, so long as they are told what to look for.
Two cautions from the Thoughtworks Technology Radar of November 5, 2025, which placed spec driven development in Assess: the workflows remain elaborate and opinionated, some tools generate lengthy spec files that are hard to review, and the industry may be relearning that handcrafting detailed rules for AI does not scale. The spec is the input to the factory; a spec nobody can review is a control that does not exist.
4. The acceptance test: fourteen checks on one specification
Pick one real ticket that fits the shape of a specification. Run it through the line. Score each check yes or no.
| Area | Check | What passes |
|---|---|---|
| Input | 1. The spec states intention, acceptance criteria and scenarios (given, when, then) | Behavior, not implementation; every criterion can be turned into a test |
| Input | 2. The spec can be reviewed by its owner in one sitting | A product owner approved it without reading a generated document longer than the feature |
| Input | 3. Ambiguity produces a question, not an assumption | The line stopped and asked at least once, or the spec had no holes |
| Output | 4. Code lands in the layer of the system where the team would have put it | Architecture and conventions read before writing; no new layer invented |
| Output | 5. Tests are derived from the acceptance criteria and exercise failure paths | Each criterion maps to a test; at least one test fails on purpose when the code is broken |
| Output | 6. Interface changes come with visual evidence | Desktop and mobile captures that a person can inspect |
| Output | 7. A traceability record exists | Which requirement produced which file, and which test covers it |
| Output | 8. A run log exists | What was produced, what was verified, what failed and was corrected |
| Control | 9. The tests were audited by a verifier that did not write them | Assertions loosened, steps skipped or expected values replaced with whatever the code returned are caught before the pull request |
| Control | 10. A human approves every merge | The line cannot merge its own work; the approval is recorded |
| Control | 11. Every pull request has a named human owner | Someone chases the review and drives the change to merge or to close |
| Measure | 12. Merge yield within 30 days, compared with the team's own human baseline | Measured, not assumed; the gap is known |
| Measure | 13. Rework rate on merged changes | Merged work that is rewritten within weeks is counted against the line |
| Measure | 14. Cost per effective pull request | People and tokens, divided by merged pull requests with rework discounted |
Organization by yaab Research. Checks 1 to 11 come from how the line has to be built; checks 12 to 14 come from the measures LinearB proposes (effective pull requests, rework rate, cost per effective pull request), restated.
Fewer than eleven yes is a line that generates more than it verifies. Check 9 is the one most often missing and the one Fowler's note is about: a pipeline that grades its own tests inflates its grade, because the fastest way to make a check pass is to make it check less.
5. Reading the result
If checks 1 to 3 fail, the problem is upstream of the factory: the team does not write specifications, and no pipeline fixes that. If 4 to 8 fail, the line generates code and not the evidence that makes the code reviewable; a reviewer facing a diff with no derived tests and no trace will do what LinearB's data shows, wait 4.6 times longer and then decline. If 9 to 11 fail, the line will pass its own tests and merge its own work, and the three month rot in section 2 is the expected outcome. If 12 to 14 are not measured, nobody will know which of the above happened until finance asks.
6. How we run this on our own line
On our own work, every specification produces the same five things: the code in the layer where it belongs, functional tests derived from the acceptance criteria including failure paths, visual tests with inspectable evidence when the interface changes, a traceability record and a run log. Every test the line produces is reviewed by an independent verifier that did not write it, looking for tests made to pass by verifying less; the same check covers visual baselines, because updating a reference screenshot is the easiest way to hide a broken interface behind a green run. An engineer approves every merge and the line never merges its own work. When a spec is ambiguous, the line produces a precise question instead of a silent assumption. We run the fourteen checks on the first specification of every new codebase before the backlog goes in, and again when the yield moves.
Sources
- Martin Fowler, Fragments: September 8, martinfowler.com, September 8, 2026, quoting Christian Catalini. https://martinfowler.com/fragments/2026-09-08.html
- LinearB, Your software factory needs a context layer (2026 mid year AI benchmarks), August 4, 2026. https://linearb.io/blog/does-your-software-factory-work
- LinearB, Elite AI-driven engineering teams double PR output while others stall (Yishai Beeri), August 14, 2026. https://linearb.io/blog/elite-engineering-teams-double-pr-output-ai-benchmarks
- LinearB, How to keep seeing when your software factory runs in the dark, September 4, 2026, and 2026 Software Engineering Benchmarks Report (public summary page). https://linearb.io/blog/software-factory-night-vision and https://linearb.io/resources/software-engineering-benchmarks-report
- Kief Morris, Humans and Agents in Software Engineering Loops, martinfowler.com, March 4, 2026. https://martinfowler.com/articles/exploring-gen-ai/humans-and-agents.html
- Thoughtworks, Technology Radar, Spec-driven development (Assess), November 5, 2025. https://www.thoughtworks.com/radar/techniques/spec-driven-development
About the sources. The figures and quotations in this document are cited from the sources listed above and restated in our own words beyond the short phrases marked as such; no chart, table or extended passage from those publications is reproduced here. LinearB is a vendor of engineering analytics and its figures come from organizations that use its product; they are cited as such. The people and organizations cited did not take part in this document and do not endorse it; their names identify the sources and nothing more. The fourteen checks are our adaptation.