Who said what, and was it right

Court interpreting quality is checked by sampling. Roughly one booking in a hundred gets a review, and the review is a person listening back to a recording. Most hearings go unassured. That gap is what a small speculative project of mine has been poking at: transcribe an interpreted hearing, align the interpreter’s rendition against the source, and hand a human reviewer a head start.

Two caveats before anything else. This is speculative work on synthetic data – mock hearings I wrote and voiced, not real cases. HMCTS has had no involvement, and none of it is a product or a proposal to them.

The parts worth writing up are two problems that word-error rate leaves untouched. One is attribution: putting each turn against the right speaker, which in a courtroom carries more weight than any single mis-heard word. The other is false positives: an LLM asked to flag interpretation errors will flag too many, and a reviewer staring at thirty mostly-wrong flags soon stops reading. Everything below runs on one mock Polish–English hearing with eighteen deliberately seeded errors.

Who said what

Speaker diarisation is the step that decides which words belong to which voice. The usual approach clusters the audio acoustically. On the mock hearing it found the right number of speakers – six – and then put the clerk and the defence counsel on the same label, because they sound alike: two similar voices, one courtroom microphone. Turn attribution came out at 85%.

That 85% is the problem, because it reads as a pass while hiding the one confusion that matters. A merged clerk and counsel means an exchange can be filed under the wrong person, and in a hearing that changes who said what.

Voices are hard to tell apart. Roles are not. Who is speaking at any moment is heavily constrained by what is being said: the clerk calls the case and swears the interpreter, counsel announce themselves for the Crown or the defence, the interpreter renders for the defendant. So after the acoustic pass, a second pass reads the transcript and gives each turn a courtroom role from its content. Because it works turn by turn, it can separate two turns that the acoustic step had glued into one voice.

flowchart TD
    A["Continuous single-mic audio"] --> B["Acoustic diarisation<br/>(cluster the voices)"]
    B --> C["Voice clusters · 85% attribution<br/>clerk + defence merged onto one voice"]
    T["Per-turn transcript text"] --> D
    C --> D["Role-attribution pass<br/>give each turn a courtroom role<br/>from what is said"]
    D --> E["100% attribution<br/>merged speakers recovered"]
    E --> F["Speaker-labelled transcript"]

    style A fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
    style B fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
    style T fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
    style F fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
    style C fill:#f59e42,stroke:#fff,stroke-width:2px,color:#fff
    style D fill:#2d8cff,stroke:#fff,stroke-width:2px,color:#fff
    style E fill:#10b981,stroke:#fff,stroke-width:2px,color:#fff

On this hearing that took attribution from 85% to 100%, and the merged clerk-and-counsel came apart. The acoustic result stays on screen next to the fused one, so the failure and its repair are both visible.

A note on measurement. “Percentage of turns attributed correctly” quietly assumes you already know where each turn starts and ends. The more honest figure is diarisation error rate – the share of speaking time handed to the wrong speaker – which here falls from about 9% to zero. With hand-marked turn boundaries that number only measures confusion, and a real recording, with overlaps and a single far-field microphone, is a harder test than this one. Voice enrolment of the fixed cast, or a microphone per seat, would do most of the work in a real courtroom; the role pass mops up the rest.

Was it right

Now the judgement. Line up each source turn with the interpreter’s rendition and ask whether the rendition is faithful. An LLM does this well, with one catch I walked straight into: asking a single model to both find discrepancies and decide whether each one is real pulled its attention in two, and recall of the seeded errors dropped into the forties. Splitting the job – one pass to detect, a separate pass to attribute each flag to the interpreter or to the pipeline’s own transcription and translation – brought recall back to around 94%.

Detection that sensitive over-fires. Precision sat near 74%: legitimate paraphrase, and artefacts of the machine transcription, were coming through as interpreter errors. The reviewer would have spent most of their time dismissing noise.

The fix is a shape I’ve written about before: an independent verifier with the opposite incentive. A separate pass takes each surviving flag and tries to knock it down, upholding only the ones that are legally material – a changed number, a dropped bail condition – and refuting acceptable paraphrase or a wobble introduced by the transcription. Refuted flags leave the score but stay on screen for the reviewer, marked as dismissed rather than deleted.

flowchart LR
    A["Aligned pairs<br/>source ↔ rendition"] --> D["Detect<br/>mode-aware judge<br/>high recall"]
    D --> AT["Attribute<br/>interpreter vs machine artefact<br/>reads the raw source-language ASR"]
    AT --> V["Verify<br/>independent pass<br/>tries to refute each flag"]
    V --> S["Score<br/>refuted flags shown, not counted"]
    TA["Critical-token audit<br/>deterministic number / date diff"] -.->|corroborate + backstop| AT

    style A fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
    style AT fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
    style S fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
    style D fill:#2d8cff,stroke:#fff,stroke-width:2px,color:#fff
    style V fill:#10b981,stroke:#fff,stroke-width:2px,color:#fff
    style TA fill:#6366f1,stroke:#fff,stroke-width:2px,color:#fff

Precision moved from about 74% into the high 80s and low 90s, with recall holding. The same seeded-error scorecard keeps the trade honest: if the verifier wrongly throws out a real error, that shows up as lost recall, so over-eager refusal carries a cost. The numbers do wobble a few points run to run, which is its own small research topic.

The judge, attributor and verifier all run on a mid-tier model – Sonnet-class, on Bedrock, in the London region – rather than the largest thing available. As with the escalation ladder, the value is in the arrangement, not the size of the model.

The part you don’t need a model for

Amounts, dates and numbers are where an LLM judge is most useful and least trustworthy: a fluent model will happily narrate a number error it never actually checked. They are also the one class of error you can verify exactly. So before any judging, a plain regular-expression diff pulls the money, numbers and dates out of the source and the rendition and compares them.

It does two jobs. It corroborates the judge – a flag that a deterministic check and a language model both raise is a stronger thing to put in front of a procurement team than any confidence score – and it backstops recall, catching a changed value the judge missed. On this hearing it caught the seeded £340 rendered as £380. Only changed values become findings on their own; a merely missing number just corroborates, because an absent figure is as likely to be a transcription drop as an interpreter’s omission.

Names, while the hearing is still running

The other half of the tool is a live aid for the interpreter, and it has a latency budget: past about three seconds of lag, a running transcript stops being useful. Numbers and dates come out of a regular expression on the spot. Names are harder, because the ones that matter – counsel, or a witness nobody flagged in advance – are rarely in a glossary, and a plain pattern match finds none of them. A small language-model pass handles those, running off the critical path and surfacing each name with its role a beat after the line lands. It picked up the interpreter’s own name and registration number, which the pattern matcher had walked straight past.

Problem	Before	After
Speaker attribution	85%, two speakers merged	100%, merge recovered
Judge precision	~74%	~89–94%, recall held
Amounts and dates	judge only	corroborated, and backstopped
Live names	none (glossary only)	caught, with roles

Iterating without the cloud

One design choice paid for itself repeatedly. The interface never talks to the pipeline directly; it only renders a stream of telemetry events the pipeline emits. Record that stream once from a real run and you can replay it.

That buys two things. Building the front end becomes a browser refresh against recorded fixtures, in place of a five-minute rebuild-and-deploy for every wording change. And the whole click-through demo can be served as static files that replay in the browser, on a CDN, with no backend running at all – pennies a month, and it stays up perpetually instead of being brought up for a meeting and torn down after.

flowchart LR
    P["Pipeline on real cloud"] -->|emits| EV["Telemetry event stream"]
    EV -->|recorded once| FX["Replay fixtures (JSON)"]
    FX -->|replayed locally| LM["Local dev<br/>no cloud calls"]
    FX -->|replayed in the browser| ST["Static demo<br/>CDN · no backend"]
    LM --> UI["Same renderer"]
    ST --> UI

    style P fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
    style EV fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
    style UI fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
    style FX fill:#6366f1,stroke:#fff,stroke-width:2px,color:#fff
    style LM fill:#2d8cff,stroke:#fff,stroke-width:2px,color:#fff
    style ST fill:#10b981,stroke:#fff,stroke-width:2px,color:#fff

Where this stops

The honest limit of the QA side is that much of it reasons over text. A seeded interpreter error and a garble from the machine transcription can look identical on the page, and no amount of re-reading the text will separate them. The proper answer is to go back to the audio and re-check the flagged moment against the interpreter’s own channel, which is a next step rather than a solved one. The scorecard also leans on knowing the seeded errors, and a real hearing has no answer key, so reference-free quality estimation is the piece a deployment would actually need.

None of which shifts the framing at the top: synthetic data, a speculative build, no involvement from anyone at HMCTS. There is a fuller overview of the project, and a click-through version of the workbench, here.