An escalation ladder for LLM judgement
Traceability problems seem to follow me around. Years ago at DFID I spent one of the most enjoyable projects of my career trying to trace aid money through a graph database, and the itch has never really gone away: derived claims on one side, source material on the other, and the question of where exactly each claim came from sitting awkwardly between them.
The latest version of the itch: about eighty paraphrased items in a research catalogue, each derived from one long published document, and a requirement to trace every item back to the exact sentence it came from. Desk research does this constantly – recut a source into your own tidy structure, cite it at whole-document level, and accumulate what I’ve started calling paraphrase debt. Six months later somebody asks where item 9.3 actually comes from, and the answer is a shrug and a 200-page PDF. I have done my share of the shrugging.
Paying the debt back by hand is a day of tedium with a real error rate, because careful reading degrades badly somewhere around the fortieth item. Paying it back with a frontier model in one long context is expensive, and hands you a single opinion nobody has checked. The bit worth writing up is the architecture in between.
The ladder
Each rung cleans up after the one below it, and the expensive attention sits at the top, used as little as possible.
flowchart TD
P[Parse the source into addressable units<br/>heading anchors, numbered items, lettered sub-items] --> M[Deterministic draft: token-overlap ranking<br/>top guess wrong about 1 in 5]
M --> J[8 mapping agents in parallel, 9 items each<br/>mid-tier model, full document in hand]
J --> V[8 adversarial verifiers, fresh context<br/>instructed to refute, not confirm]
V -->|agree · 69 of 71| A[Accepted]
V -->|disagree · 2 of 71| H[Human reads both texts]
H --> A
A --> Q[Quotes and anchors attached by code<br/>never transcribed by a model]
Q --> O[(Overlay file<br/>item → ref · anchor · verbatim quote)]
style P fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
style M fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
style J fill:#2d8cff,stroke:#fff,stroke-width:2px,color:#fff
style V fill:#10b981,stroke:#fff,stroke-width:2px,color:#fff
style H fill:#f59e42,stroke:#fff,stroke-width:2px,color:#fff
style O fill:#6366f1,stroke:#fff,stroke-width:2px,color:#fff
Rung 0 – parse. The source document’s accessible HTML gave every numbered commitment a stable heading anchor, with lettered sub-items beneath – a few hundred addressable units. Twenty minutes of parsing produced a JSON file of ref, anchor, text triples, and everything downstream works against that file rather than the raw document.
Rung 1 – a deterministic draft. Token overlap between each catalogue item and every document unit, top eight candidates kept per item. Its best guess was wrong roughly one time in five, which sounds disqualifying until you remember its job: it only has to make the next rung’s reading cheap. Wrong-but-close candidates are fine when a model with the whole document in hand is about to check them.
Rung 2 – batched judgement. Eight mapping agents ran in parallel, nine items each, on a mid-tier model (Sonnet, in this case – the sort of model that costs a tenth of the one you’d reach for by default). The batch size matters more than it looks: nine items is small enough that the agent reads every candidate properly, and large enough to amortise the per-agent overhead of loading the parsed document and absorbing the instructions. Each agent received the full document file as well as its candidates, plus two standing instructions that carried most of the quality: don’t trust the ranking, and match on the named mechanism – the specific programme, institution or legal instrument – never on shared vocabulary. Outputs came back through a JSON schema, so merging them was code rather than copy-paste.
Rung 3 – the adversarial pass. This is the crux. Eight fresh verifier agents on the same mid-tier model, one per batch, sharing nothing with the mappers’ context. Each got the proposed mappings, the full document, and a prompt that inverts the prior: assume each mapping is wrong until the text convinces you otherwise; if it’s wrong, find the ref that fits better. The verifiers agreed with 69 of 71 mappings and rejected two, both with specific better candidates attached. A second pass told merely to “check” the mappings would have waved all 71 through – the refute-first framing is what earns this rung its place.
Rung 4 – the human. I read two disagreements. Both verifier objections were right, and adjudicating one of them surfaced a commitment in the document that both model passes had missed. Which is the ladder doing its job: the verifier catches the mapper, and the human catches both. Total human reading: two pairs of paragraphs.
The whole run – sixteen agents, 1.4 million mid-tier tokens – took about fifteen minutes of wall-clock time, and the token bill came to less than the coffee I made while it ran.
| Rung | What it is | Error rate observed | What catches its mistakes |
|---|---|---|---|
| Deterministic draft | token overlap | ~20% at top-1 | mapping agents |
| Mapping agents | 8 × 9 items, full doc | ~3% | adversarial verifiers |
| Adversarial verifiers | fresh context, refute-first | missed 1 shared blind spot | human on disagreements |
| Human | reads 2 items | – | – |
Two rules that made it trustworthy
The models never transcribe anything. An agent’s output is a reference like “13g” and nothing else. The verbatim quote and the anchor are attached by code, from the parsed document. An agent that picks the wrong reference gets caught a rung up; an agent that picks the right one can’t misquote it, because it never writes the quote.
The catalogue itself stays frozen. Mappings live in a separate overlay file – item id, source ref, anchor, verbatim quote – validated on load. The same fields work for any source format: heading anchors for HTML, page numbers for PDF. Swap the document and the machinery doesn’t care.
The quote earns its keep in the interface, because modern browsers support URL text fragments – a link that carries its own highlighter:
https://example.gov.uk/long-document#numbered-commitment-heading:~:text=First%20ten%20words%20of%20the%20sentence
Click it and the browser scrolls to the anchor and highlights the exact sentence. Where fragments aren’t supported, the quote sits in the tooltip, ready for ctrl+F.
The best result was a negative one
Three of the eighty items matched nothing. Mapping agents and verifiers searched the document independently and agreed there was no counterpart – the original desk research had quietly invented them. Reasonable inventions, as it happens, but inventions. They’re now labelled “no direct source” in the interface instead of carrying a link that implies otherwise.
I’d rank that above the 76 successful mappings. A checker that can only ever say yes is a rubber stamp. This one turned out to be capable of saying “we made this bit up”, which is the property I actually wanted from it.
The shape generalises
None of this is specific to tracing paraphrases. It’s an attention budget spent from the bottom up: code drafts, cheap models judge and then re-judge with the prior inverted, and a human reads only the conflicts. Anywhere derived claims point lazily at their source material – literature reviews, audit findings against evidence – the same ladder runs at the same rate: an afternoon, most of it unattended. I’ll be running it again the next time somebody asks where item 9.3 came from.