One model, one pass, one confident answer
Ask a language model to review a draft and it will tell you the draft is good. Ask it to be critical and it will find three things, phrase them well, and stop. The failure is not that the output looks wrong. The failure is that it looks right: fluent, structured, and quietly missing the claim that has no source behind it.
The specific risk I care about is publishing a number I cannot defend. My articles cite studies, effect sizes, and statistics. A single-pass review catches the obvious problems and leaves the ones that require actually checking a citation against a source.
So the design question was not how to get a better review. It was how to get a review whose agreement means something.
Fan out to seven, converge through one
Why seven reviewers instead of one longer prompt
- One lens each. A reviewer that is only looking for factual errors does not trade a factual finding away against a stylistic one. Bundle seven jobs into one prompt and the model silently budgets across them.
- No shared context. Reviewers never see each other's output. Two reviewers landing on the same issue is therefore evidence, not an echo. This is the property the whole design exists to buy.
- Different tools per role. The fact-checker has web search and is instructed to verify rather than assess. The other six do not have it and cannot substitute plausibility for a source.
- Different models per role. Reviewers run on a cheaper tier; the synthesiser, which has to hold seven documents and an article in context and resolve conflicts, runs on the strongest one.
Three pieces that make it work
1 · a reviewer is a role brief plus a tool grant. The fact-checker is the only one with search.
---
name: blog-pl-fact-checker
description: Verifies factual claims - cited studies, statistics, dates,
names, scientific assertions. Uses WebSearch + WebFetch when needed.
tools: Read, Write, Glob, Grep, WebSearch, WebFetch
model: sonnet
---
# What you verify
Every cited study - does it exist, does the date match, is the
conclusion distorted. Every researcher name - does the person
exist, did they actually work on this. Every institution and
journal - correct spelling, does it exist.
You do not guess. You verify.2 · the orchestrator fans out in a single message, so the seven run concurrently rather than in sequence.
## Step 2 - dispatch all 7 reviewers IN PARALLEL
In a single message, dispatch 7 Agent calls in parallel.
Each agent gets:
- subagent_type: the agent name
- prompt: full path to the article + full path to the reviews
folder + "write your review to <folder>/<role>.md"
## Step 3 - run the synthesizer
Once all 7 reviewers finish, dispatch the synthesizer with the
article path and the reviews folder.3 · the synthesiser's contract. Point 5 is the one that makes the whole thing auditable.
model: opus
# Role
1. Read the 7 review files for this article.
2. Extract a prioritised list of changes (must-fix vs nice-to-have).
3. Write _synthesis.md with that list.
4. Take the original, apply the must-fix changes, save as <slug>-v2.md.
5. NEVER overwrite the original.Because the original survives, every run is reproducible after the fact. The numbers on this page were not logged while the pipeline ran. They were reconstructed months later by parsing the artifacts it left behind, which is only possible because nothing overwrites its own input.
What 30 production runs produced
These are real runs on real articles, not a benchmark built for this page. Every figure below is derived from the artifacts on disk by a script, and spot-checked by hand against the raw files on three randomly chosen runs.
| Measure | Total | Median per run |
|---|---|---|
| Runs with all 7 reviewers plus a synthesis | 30 / 30 | - |
| Individual findings raised by reviewers | 1,673 | 57 |
| Must-fix items after synthesis | 676 | 21.5 |
| Findings raised independently by 2+ reviewers | 377 | 11 |
| Findings raised independently by 4+ of 7 | 241 | - |
| Wall clock for the 7-way fan-out | - | 118s |
Five corrections that reached the reader as facts
An article on sleeping positions, roughly 3,500 words with a reference list. It read fine. Below are five changes the pipeline forced before publication, each verified against the original and the rewritten version, both still on disk.
"About 60% of Poles sleep on their side, 25% on their back, 7% on their stomach."
"According to Polish consumer reports (Pościel.to 2024) about 70% of Poles fall asleep on their side, 15% on their back, 7% on their stomach. In the 65+ group the side-sleeping share rises to 82%."
"Worsens obstructive sleep apnoea by 30 to 80% compared with side sleeping (Cartwright 1984)."
"In the classic Cartwright 1984 study the apnoea index (AHI) in the supine position was roughly 2x higher (~+100%) than lateral. The definition of positional OSA rests on exactly that ratio."
"Lee 2015 suggests the side position is most effective for flushing metabolic waste from the brain."
"May support glymphatic drainage (Lee/Nedergaard 2015: preliminary data in rats; the mechanism is plausible in humans but not clinically confirmed)." The reference list now carries the full author list, the journal, and the caveat.
"Khoury R. et al. (2019). Body position and obstructive sleep apnea. Chest 156(5)."
Replaced with a source that verifies: "Joosten S.A., Hamilton G.S., Naughton M.T. (2014). Supine position related obstructive sleep apnea in adults. Sleep Medicine Reviews 18(1):7-17."
"After 21 to 30 days the new position becomes the default."
"After 21 to 30 days the new position usually becomes the default (the '21 days to a habit' claim is a simplification; Lally 2010 reports a mean of 66 days, range 18 to 254)."
Item 4 is the reason seven reviewers exist. Six of them read that citation and had no reason to doubt it: correct format, plausible journal, plausible year. Only the reviewer whose single job was verification, and who had a search tool, went and looked. Agreement is the signal for most findings. For that one, isolation was.
What a run costs, and how that number was produced
Cheap models for mechanical stages, the strong model for judgement and synthesis. Eight agent calls per run.
| Measure | Median | Range across runs |
|---|---|---|
| Agent calls | 8 | 8 |
| Input tokens | 72k - 93k | 58k - 139k |
| Output tokens | 34k - 44k | 30k - 53k |
| Cost at published API rates | $1.20 - $1.55 | $0.99 - $2.02 |
- It is not an invoice. These runs executed on a subscription, so no per-token bill exists for them. This is a reconstruction from measured artifact sizes, priced at published rates. The method and the script are in the repository.
- The input figure is a floor, not a total. The harness system prompt and tool-call overhead are not on disk and are not counted. Real input tokens are higher than reported here.
Every figure is a band rather than a point, because the source text is Polish and does not tokenise at a fixed rate. The band spans 2.8 to 3.6 characters per token, and the expensive end of the band is the one quoted first.
The reason the cost is worth stating at all: it is the number that decides whether this runs on every article or only on the important ones. At roughly a dollar-something per run it runs on everything, which is why there are 30 runs to report instead of three.
What this system does not do
Written first, not last, because a case study that only lists strengths is a brochure.
- It is not an automated eval harness. There are no numeric pass/fail thresholds and no gate that blocks publication programmatically. It is a panel of reviewers, a synthesiser, and a human who approves. Calling it "evals" would oversell it.
- Agreement is not correctness. Seven reviewers sharing a wrong prior will agree confidently. Agreement narrows the search space for a human; it does not replace one.
- The fact-checker inherits the web's errors. It verifies that a source exists and says what the article claims it says. It does not evaluate whether the study is any good.
- No production monitoring. There is no alerting, no retry policy, and no dashboard. Failures show up as a missing file in the output folder, which I notice because the run is in front of me.
- Measurement was retrofitted. Nothing logged tokens or timings while these runs happened. Everything on this page was reconstructed afterwards from artifacts, which is why timings come from file timestamps and cost comes from a model rather than a meter.
Where else this pattern runs
Claude Code with subagents and MCP servers, orchestrated by a slash command; Python for the artifact analysis; plain HTML for this page. No framework, nothing to install.
The same fan-out-and-synthesise shape now runs three other jobs: a market-research sweep where each agent searches a different source type, a job-market pipeline where every posting is verified against the employer's own applicant-tracking API before it is trusted, and a documentation review where the lenses are correctness, completeness, and the reader who has never seen the product.
The transferable part is not the seven roles. It is the discipline of paying for independence and then measuring whether it bought you anything.