This document is an acceptance test for a software factory, the pipeline that takes a specification in and produces code, tests and evidence out. It gives you fourteen checks to run on one specification before the line takes your backlog: what goes in, what comes out, the controls around the agents, and the measures that say whether the output ships.
1. The cost of generating fell. The cost of verifying did not.
Martin Fowler's note of September 8, 2026 quotes Christian Catalini's observation that we are vastly reducing the cost of generating things but not the cost of verifying them, and that the new automation boundary is measurable versus non measurable work. Fowler's reading for software: too much of what makes work effective sits behind slow feedback loops or judgment, so teams automate heavily while measuring incompletely, and the short term dashboards go up while the long term goes wrong. Catalini calls those gains counterfeit utility. Fowler's conclusion is about incentives: people have to invest more in verification than in generation, or they are driving a car with a powerful engine and weak brakes. One more line from the same note, from Catalini: build a history of decisions, not a gallery of outputs.
A software factory is the purest case. It maximizes generation by design. Whether it is worth anything depends entirely on the verification built around it.
2. What the data says about factory output
LinearB's 2026 mid year benchmarks (published August 4, 2026; 2.7 million pull requests (code changes), 7.5 million commits, 83,000 developers, 253 organizations, first half of 2026) measure the factory directly. Even at elite organizations, agents author at most 4.7% of pull requests; the autonomous part of the pipeline opens one pull request in twenty. Pull requests overall merge within 30 days 90% of the time at elite organizations, 86% at good ones and 81% at fair ones. Agent opened pull requests merge at 79%, 58% and 37%. LinearB's full year report on more than 8.1 million pull requests across more than 4,800 organizations puts the acceptance of AI generated pull requests within 30 days at 32.7% against 84.4% for human written ones, with AI pull requests waiting 4.6 times longer before review and being reviewed twice as fast once picked up.
LinearB's CTO, Yishai Beeri, gave the mechanism on August 14, 2026: an agent that pulls a ticket, implements it and pushes a pull request with no human owner produces three pull requests where one survives, because nobody chases approvals or drives the change to merge. Shipping code is an act of ownership, and detaching the human removes it. The lever that lifts yield in the same data is an automated review pass before a human opens the pull request, worth up to five percentage points across all pull requests and up to seven at the fair tier.
The same post records the case against the factory: one founder ran his company's work through a pipeline with humans at the edges and shut it down after roughly three months of codebase rot, because nothing in how models are trained rewards keeping a codebase maintainable, and the erosion was one no test caught. LinearB's own framing, from a roundtable it hosted on September 4, 2026: every change the factory merges becomes the environment the agents work in next quarter, so the merge is a poor place to stop measuring. Rework rate, review load and how often merged work comes back are the proxies that move when quality slips.
Our reading, not LinearB's: a low merge rate is not bad on its own. A line that rejects unverified changes is doing its job; the number to watch is the merge rate of what the line was asked to produce, against the team's own baseline.
3. Where the humans sit
Kief Morris, on martinfowler.com on March 4, 2026, separates the why loop (turning ideas into outcomes, which humans run) from the how loop (building the software, with its inner loops of specifying, generating and testing). Humans out of the how loop is vibe coding and, in some interpretations, spec driven development. Humans inside the innermost loop, inspecting every generated line, become the bottleneck, and he suspects that is part of why productivity studies show mixed results. His answer is humans on the loop: build and maintain the harness, the collection of specifications, quality checks and workflow guidance that controls each level of the loop, and when the output is wrong, fix the harness rather than the artifact. Agents produce better code when they can gauge the quality of their own output instead of waiting for a person to check it, so long as they are told what to look for.
Two cautions from the Thoughtworks Technology Radar of November 5, 2025, which placed spec driven development in Assess: the workflows remain elaborate and opinionated, some tools generate lengthy spec files that are hard to review, and the industry may be relearning that handcrafting detailed rules for AI does not scale. The spec is the input to the factory; a spec nobody can review is a control that does not exist.
4. The acceptance test: fourteen checks on one specification
Pick one real ticket that fits the shape of a specification. Run it through the line. Score each check yes or no.
| Area | Check | What passes |
|---|---|---|
| Input | 1. The spec states intention, acceptance criteria and scenarios (given, when, then) | Behavior, not implementation; every criterion can be turned into a test |
| Input | 2. The spec can be reviewed by its owner in one sitting | A product owner approved it without reading a generated document longer than the feature |
| Input | 3. Ambiguity produces a question, not an assumption | The line stopped and asked at least once, or the spec had no holes |
| Output | 4. Code lands in the layer of the system where the team would have put it | Architecture and conventions read before writing; no new layer invented |
| Output | 5. Tests are derived from the acceptance criteria and exercise failure paths | Each criterion maps to a test; at least one test fails on purpose when the code is broken |
| Output | 6. Interface changes come with visual evidence | Desktop and mobile captures that a person can inspect |
| Output | 7. A traceability record exists | Which requirement produced which file, and which test covers it |
| Output | 8. A run log exists | What was produced, what was verified, what failed and was corrected |
| Control | 9. The tests were audited by a verifier that did not write them | Assertions loosened, steps skipped or expected values replaced with whatever the code returned are caught before the pull request |
| Control | 10. A human approves every merge | The line cannot merge its own work; the approval is recorded |
| Control | 11. Every pull request has a named human owner | Someone chases the review and drives the change to merge or to close |
| Measure | 12. Merge yield within 30 days, compared with the team's own human baseline | Measured, not assumed; the gap is known |
| Measure | 13. Rework rate on merged changes | Merged work that is rewritten within weeks is counted against the line |
| Measure | 14. Cost per effective pull request | People and tokens, divided by merged pull requests with rework discounted |
Fewer than eleven yes is a line that generates more than it verifies. Check 9 is the one most often missing and the one Fowler's note is about: a pipeline that grades its own tests inflates its grade, because the fastest way to make a check pass is to make it check less.
5. Reading the result
If checks 1 to 3 fail, the problem is upstream of the factory: the team does not write specifications, and no pipeline fixes that. If 4 to 8 fail, the line generates code and not the evidence that makes the code reviewable; a reviewer facing a diff with no derived tests and no trace will do what LinearB's data shows, wait 4.6 times longer and then decline. If 9 to 11 fail, the line will pass its own tests and merge its own work, and the three month rot in section 2 is the expected outcome. If 12 to 14 are not measured, nobody will know which of the above happened until finance asks.
6. How we run this on our own line
On our own line, which builds a SaaS product of our own, every specification produces the same five things: the code in the layer where it belongs, functional tests derived from the acceptance criteria including failure paths, visual tests with inspectable evidence when the interface changes, a traceability record and a run log. Every test the line produces is reviewed by an independent verifier that did not write it, looking for tests made to pass by verifying less; the same check covers visual baselines, because updating a reference screenshot is the easiest way to hide a broken interface behind a green run. An engineer approves every merge. When a spec is ambiguous, the line produces a precise question instead of a silent assumption. We run the fourteen checks on the first specification of every new codebase before the backlog goes in, and again when the yield moves.
Questions and answers
How do you measure what a software factory really costs?
Per effective pull request, not per line of code. Add the cost of people and tokens, then divide by the pull requests that merged, with rework discounted. It is the last of the fourteen checks, and it only means something next to the merge yield within 30 days.
Why do pull requests opened by agents merge less often?
Because nobody owns them. An agent that pulls a ticket and pushes a pull request with no human owner produces three where one survives, since nobody chases approvals. The lever that lifts yield is an automated review pass before a person opens the pull request, worth up to five percentage points.
Where should people sit in a software factory?
On the loop, not in it. People run the loop that turns ideas into outcomes and maintain the harness, the specifications, checks and workflow guidance that control the line. When the output is wrong, they fix the harness rather than the code. Inspecting every generated line makes people the bottleneck.
How many of the fourteen checks should a line pass?
At least eleven. Fewer means the line generates more than it verifies. The check most often missing is an independent verifier that did not write the tests, because a pipeline that grades its own tests inflates its grade. A low merge rate alone is not bad: a line that rejects unverified changes is doing its job.
Sources
- Martin Fowler, Fragments: September 8, martinfowler.com, September 8, 2026, quoting Christian Catalini. https://martinfowler.com/fragments/2026-09-08.html
- LinearB, Your software factory needs a context layer (2026 mid year AI benchmarks), August 4, 2026. https://linearb.io/blog/does-your-software-factory-work
- LinearB, Elite AI-driven engineering teams double PR output while others stall (Yishai Beeri), August 14, 2026. https://linearb.io/blog/elite-engineering-teams-double-pr-output-ai-benchmarks
- LinearB, How to keep seeing when your software factory runs in the dark, September 4, 2026, and 2026 Software Engineering Benchmarks Report (public summary page). https://linearb.io/blog/software-factory-night-vision and https://linearb.io/resources/software-engineering-benchmarks-report
- Kief Morris, Humans and Agents in Software Engineering Loops, martinfowler.com, March 4, 2026. https://martinfowler.com/articles/exploring-gen-ai/humans-and-agents.html
- Thoughtworks, Technology Radar, Spec-driven development (Assess), November 5, 2025. https://www.thoughtworks.com/radar/techniques/spec-driven-development
Analysis and conclusions by yaab, based on the published reports listed above.