I asked another agent to review several project contracts and documentation systems. It returned a careful report with severities, file references, and a recommended order of attack. Much of it was useful. One of its highest-severity findings was also false.
The report said a documentation index linked to pages that did not exist. All of the pages were there.
That did not make the review worthless. It made it a review: a set of claims to check rather than a verdict to execute.
Agreement Is Not Corroboration
Two agents can agree because they independently reached the same conclusion. They can also agree because they inherited the same stale path, summary, or assumption. The output looks identical from the outside.
This is especially easy with documentation. A required-read list points at an old directory. Every later agent follows the list, fails in the same place, and reports that the documentation is missing. The second report feels like confirmation, but both reports still have one source: the broken list.
An independent reviewer becomes a second source only when it checks something independent. That may be the file on disk, the generated artifact’s source, a running program, a test that can fail, or the external system the document is trying to describe. Another paragraph agreeing with the first paragraph is not new evidence.
The False Finding Had Real Neighbours
The supposedly missing pages existed. The index was not clean.
Two descriptions still referred to the directory structure from before a consolidation. The validation section named a command that did not exist in the repository that now owned the page. Elsewhere, a generated project table was faithfully reproducing a stale value from its manifest.
That distinction changed the fix.
Deleting valid links would have damaged the index. Updating the stale prose, correcting the manifest, and regenerating its output repaired the actual source of drift. The original report had found the right neighbourhood and knocked on the wrong door.
There was another useful reversal. The report argued that pointer checks were missing. One existing validator already checked ordinary Markdown links and found a broken cross-repository link the review had missed. What it did not check was a required path written as inline code rather than as a link. The system had a gate. It had the wrong boundary for that particular failure.
Internal Consistency Can Be Perfectly Stale
Generated files add another trap. A generated project index and its manifest can match exactly while both describe a path removed weeks ago. A check that asks “does output match source?” passes. A check that asks “does the source still name something real?” fails.
Both checks matter, and neither replaces the other.
The same applies to prose metadata. A field named source looked like a current
pointer and was reported as broken because its target no longer existed. The
migration script defined it as provenance: the path the page came from before
cutover. It was meant to outlive the old file. The field was confusing, but it
was not failing its contract.
Reading the name suggested one answer. Reading the code that wrote it supplied the contract.
The Review Loop
The useful process turned out to be small:
- Take the finding seriously enough to inspect it.
- Read the named source, not only the reviewer’s excerpt.
- Run the behavior when the claim is about behavior.
- Separate observed fact from inference.
- Reject the false claim without discarding nearby real defects.
- Fix the first incorrect source, then regenerate anything derived from it.
- Improve a gate only when the failure can be checked without turning every sentence into a brittle path parser.
The reviewer still saved time. It grouped related drift, exposed several real contradictions, and gave the verification pass a map. What it did not provide was permission to skip the verification pass.
A handoff is context, not proof. A review is the same thing with sharper elbows.