An escalation ladder for LLM judgement
The task: about eighty paraphrased items in a research catalogue, each derived from one long published document, and a requirement to trace every item back to the exact sentence it came from. Desk research does this constantly – recut a source into your own tidy structure, cite it at whole-document level, and accumulate what I’ve started calling paraphrase debt. Six months later somebody asks where item 9.3 actually comes from, and the answer is a shrug and a 200-page PDF.
Paying that back by hand is a day of tedium with a real error rate, because careful reading degrades badly after the fortieth item. Paying it back with a frontier model in one long context is expensive and gives you a single un-checked opinion. The interesting part is the architecture in between, which is what this post is about.
The ladder
The design principle: each rung only handles what the rung below got wrong, and each rung is cheaper than the one above is precious.
flowchart TD
P[Parse the source into addressable units<br/>heading anchors, numbered items, lettered sub-items] --> M[Deterministic draft: token-overlap ranking<br/>top guess wrong about 1 in 5]
M --> J[8 mapping agents in parallel, 9 items each<br/>mid-tier model, full document in hand]
J --> V[8 adversarial verifiers, fresh context<br/>instructed to refute, not confirm]
V -->|agree · 69 of 71| A[Accepted]
V -->|disagree · 2 of 71| H[Human reads both texts]
H --> A
A --> Q[Quotes and anchors attached by code<br/>never transcribed by a model]
Q --> O[(Overlay file<br/>item → ref · anchor · verbatim quote)]
style P fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
style M fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
style J fill:#2d8cff,stroke:#fff,stroke-width:2px,color:#fff
style V fill:#10b981,stroke:#fff,stroke-width:2px,color:#fff
style H fill:#f59e42,stroke:#fff,stroke-width:2px,color:#fff
style O fill:#6366f1,stroke:#fff,stroke-width:2px,color:#fff
Rung 0 – parse. The source document’s accessible HTML gave every numbered commitment a stable heading anchor, with lettered sub-items beneath – a few hundred addressable units. Twenty minutes of parsing produced a JSON file of ref, anchor, text triples, and everything downstream works against that file rather than the raw document.
Rung 1 – a deterministic draft. Token overlap between each catalogue item and every document unit, top eight candidates kept per item. Its best guess was wrong roughly one time in five. That sounds disqualifying until you assign it the right job: it exists to make the next rung’s reading cheap, not to be correct. Wrong-but-close candidates are fine when a model with the whole document is about to check them.
Rung 2 – batched judgement. Eight mapping agents ran in parallel, nine items each, on a mid-tier model (Sonnet, in this case – the sort of model that costs a tenth of the one you’d reach for by default). Batch size is doing real work here: nine items is small enough that the agent reads every candidate properly, and large enough that the per-agent overhead – loading the parsed document, absorbing the instructions – gets amortised. Each agent received the full document file, not just its candidates, plus two standing instructions that carried most of the quality: don’t trust the ranking, and match on the named mechanism – the specific programme, institution or legal instrument – never on shared vocabulary. Outputs came back through a JSON schema, so merging them was code rather than copy-paste.
Rung 3 – the adversarial pass. This is the crux. Eight fresh verifier agents, same mid-tier model, one per batch, each given the proposed mappings and the full document – and a prompt that inverts the prior: assume each mapping is wrong until the text convinces you otherwise; if it’s wrong, find the ref that fits better. Same model, opposite incentive, no shared context with the mapper. The verifiers agreed with 69 of 71 mappings and rejected two, both with specific better candidates attached. A confirmation-biased second pass would have waved all 71 through; the value is entirely in the inversion.
Rung 4 – the human. I read two disagreements. Both verifier objections were right, and adjudicating one of them surfaced a commitment in the document that both model passes had missed. That’s the ladder working as intended: the verifier catches the mapper, the human catches both, and frontier-level attention lands only where the cheap layers already conflict. Total human reading: two pairs of paragraphs.
The whole run – sixteen agents, 1.4 million mid-tier tokens – took about fifteen minutes of wall-clock and cost less than the coffee I drank while it ran.
| Rung | What it is | Error rate observed | What catches its mistakes |
|---|---|---|---|
| Deterministic draft | token overlap | ~20% at top-1 | mapping agents |
| Mapping agents | 8 × 9 items, full doc | ~3% | adversarial verifiers |
| Adversarial verifiers | fresh context, refute-first | missed 1 shared blind spot | human on disagreements |
| Human | reads 2 items | – | – |
Two rules that made it trustworthy
Models judge; they never transcribe. An agent’s output is a reference like “13g” and nothing else. The verbatim quote and the anchor are attached by code, from the parsed document. An agent that picks the wrong reference gets caught upstairs; an agent that picks the right one cannot misquote it, because it never writes the quote.
Traceability is an overlay, never an edit. The catalogue itself stays frozen. Mappings live in a separate file – item id, source ref, anchor, verbatim quote – validated on load. The same three fields work for any source format: heading anchors for HTML, page numbers for PDF. Swap the document and the machinery doesn’t care.
The quote earns its keep in the interface, because modern browsers support URL text fragments – a link that carries its own highlighter:
https://example.gov.uk/long-document#numbered-commitment-heading:~:text=First%20ten%20words%20of%20the%20sentence
Click it and the browser scrolls to the anchor and highlights the exact sentence. Where fragments aren’t supported, the quote sits in the tooltip, ready for ctrl+F.
The best result was a negative one
Three of the eighty items matched nothing. Mapping agents and verifiers searched the document independently and agreed there was no counterpart – the original desk research had quietly invented them. Reasonable inventions, as it happens, but inventions. They’re now labelled “no direct source” in the interface instead of carrying a link that implies otherwise.
I’d rank that above the 76 successful mappings. A checking system that can only confirm is a rubber stamp; one that occasionally reports we made this bit up is earning its tokens.
The shape generalises
None of this is specific to tracing paraphrases. The pattern is an attention budget, spent from the bottom: deterministic code drafts; cheap models judge in parallel batches; the same cheap models verify with the prior inverted; humans read only conflicts; and every verbatim fact stays out of model hands entirely. Anywhere derived claims point lazily at source material – literature reviews, requirements against contracts, audit findings against evidence – the ladder runs at the same rate: an afternoon, most of it unattended, with the expensive judgement reserved for the two places it was actually needed.