A classifier that knows when to give up

I’ve used YNAB to run a budget since the summer of 2018, and for most of that time I was the sort of person who categorised every coffee within a day or so. Then came the sabbatical, and with it a decision I’d made fairly deliberately: I wasn’t going to put all of my savings into a giant mortgage, and having decided that, I was going to stop thinking about money for a while. It worked rather well. The trouble with a deliberate break from a habit is that the habit doesn’t always come back on its own.

I did have a go at catching up by hand, which is why the sabbatical months are categorised at all. It was slow and fairly painful, and somewhere in the middle of it I started wondering whether eight years of my own decisions might be able to do most of it for me.

When I finally looked, there were 2,179 transactions sitting in YNAB with no category, going back to November last year. Behind them sat eight years of transactions I had categorised – 12,492 of them across 31 categories – which is the sort of labelled dataset I find hard to leave alone, as anyone who’s seen my rated iTunes library will know.

So this is a write-up of clearing that backlog with Claude Code over a couple of sittings. The shape is the same one I wrote about in an escalation ladder for LLM judgement: cheap, dumb things go first, expensive attention sits at the top, and each rung only sees what the rung below couldn’t settle. What’s different here is that the data is messier, and much of it is private.

Why bank statements are awkward

A bank’s description of a transaction is written for the bank. Some of it is fine – a rent payment is a rent payment – but plenty of it isn’t. PASS RRPK turned out to be a Lime bike. A Tesco could be fuel or groceries, and I decided early that I didn’t much care which. A payment to a person could be my share of a dinner or a holiday deposit, and the string is identical either way.

More to the point, a lot of the meaning isn’t in the string at all. Train tickets are Transportation on a Tuesday in London and Vacation on a Tuesday in Mull. The payee knows nothing about that; my calendar does.

The ladder

flowchart LR
    B([2,179 uncategorised]) --> R[Rules and trip dates<br/>179]
    R --> C[Classifier<br/>trained on 12,492 labelled]
    C -->|confident · 960| A[(Approved)]
    C -->|suggestion · 542| K[Categorised,<br/>checked later]
    C -->|unsure · 450| S[Sonnet batches<br/>history and Monzo context]
    B -->|over £500 · 48| O[Opus<br/>calendar and email context]
    S -->|high · 124| A
    S -->|medium · 235| K
    S -->|low · 91| W[Web research<br/>business names only]
    W -->|43 placed| K
    W --> H[Me]
    O --> A
    O -->|16 were transfers| T[Linked as transfers]

    style B fill:#2d8cff,stroke:#fff,stroke-width:2px,color:#fff
    style R fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
    style C fill:#1e293b,stroke:#fff,stroke-width:1px,color:#fff
    style S fill:#6366f1,stroke:#fff,stroke-width:2px,color:#fff
    style O fill:#6366f1,stroke:#fff,stroke-width:2px,color:#fff
    style W fill:#10b981,stroke:#fff,stroke-width:2px,color:#fff
    style H fill:#f59e42,stroke:#fff,stroke-width:2px,color:#fff

The bottom rung is a handful of regular expressions for things that are always the same (the water bill, the council), plus my own conventions written down as rules. One of those is that transport during a trip counts as Vacation, which needs to know when I was on a trip; more on that below.

Above that sits a classifier, and above that two tiers of model: Sonnet in batches for the ordinary leftovers, Opus for anything over £500, where a wrong answer moves the budget enough to matter. Whatever the models can’t place goes to a web-research rung, and what’s left after that comes to me.

The expensive rung spent a fair amount of its time discovering that money hadn’t been spent at all. Sixteen of Opus’s 48 large items were me moving money between my own accounts, which the transfer matching had missed.

A mediocre classifier with good manners

The classifier is deliberately unambitious: character n-grams of the payee, plus the account, the direction of travel and a rough size band, fed into a logistic regression with recent years weighted up. It trains in a few seconds on a laptop.

To find out how far to trust it, I trained it on everything up to October 2024 and tested it on the following year, which the model hadn’t seen. Its first guess was right 63% of the time, which on its own is useless – you’d spend longer checking its work than doing it yourself. But a classifier also tells you how sure it is, and the confident end is a different animal.

The most confident third of its guesses were right 93% of the time, and about two thirds of them were right 80% of the time. So I set the thresholds from the backtest rather than by feel: above the first line it approves on its own, between the lines it proposes a category but leaves the transaction for me to check, and below that it hands over to a model. All it really has to do is notice when it’s guessing.

What it can and can’t do is mostly a question of what the payee string carries.

Subscriptions and transport are easy, because the string is the answer. Vacation is right a quarter of the time, because a holiday is defined by where I was rather than who I paid, and Gifts barely registers. Both of those are context problems, and no amount of n-gram engineering was going to fix them.

Context the payee doesn’t carry

The trip problem had a pleasingly cheap answer. Monzo is the card I use abroad, and its API records the local currency of every transaction, so any day with foreign-currency spending is a day I was away. That, a day either side for travel, and a short file of UK trips (a weekend in Deal, a week on Mull), gives a list of trip days that the rules use directly.

For the large items, Opus had my calendar and email as well, so it could line a big payment up with a booking or an event before guessing. It still couldn’t place a large transfer to a friend, which I could: our share of a week on Mull.

Doing this with my own finances meant thinking quite carefully about what goes where:

flowchart LR
    subgraph L[On my laptop]
        direction TB
        E[Bank exports<br/>and Monzo API]
        Y[(YNAB)]
        P[Rules about people<br/>gitignored]
        M[Classifier]
    end
    subgraph C[Sent to Claude]
        direction TB
        S[Batches of transactions<br/>payee, date, category history]
    end
    subgraph W[Sent to the open web]
        direction TB
        Q[Business name and town<br/>nothing else]
    end
    E --> M
    Y --> M
    P --> M
    M --> S
    S --> Q

    style L fill:#0f172a,stroke:#2d8cff,stroke-width:2px,color:#fff
    style C fill:#0f172a,stroke:#6366f1,stroke-width:2px,color:#fff
    style W fill:#0f172a,stroke:#10b981,stroke-width:2px,color:#fff

I sign into the banks myself, in an ordinary Chrome window that Claude Code drives only to download exports, and the YNAB and Monzo tokens live in the macOS Keychain. Anything that names a friend stays in a gitignored rules file. The model tiers see transactions, which I’m comfortable with. The open web sees business names and nothing more.

The web research rung

That left 78 transactions no tier could place: small independent businesses, mostly, with payee strings truncated into nonsense. A few Sonnet subagents with web search took a cleaned-up business name and a town each and came back with what the place was, a category, a confidence and a source link. A little over half came back with something usable.

Marketplace orders defeat this completely. An Amazon line could be anything, so those became Gear by fiat. (In my budget, Gear means physical things that aren’t clothes – furniture, luggage, hardware – and nothing unbecoming of a man who works in public services.) A couple of the results pages also contained text addressed to any passing AI agent, telling it what to do, which the workers had been told to ignore and did. I mention it mostly because it was the first time I’d seen one in the wild on a task this mundane.

Potholes

A selection, since they’re most of what the evenings consisted of:

• Monzo leaves some card payments pending more or less indefinitely – Uber, Lime and the odd vending machine – so the totals didn’t match the app until I added those by hand.
• YNAB reserves a few payee names, such as “Starting Balance”, and one in a batch of a few hundred gets the whole batch rejected.
• A rule I wrote for reimbursements from work quietly overrode a choice I’d made by hand years earlier. The backtest didn’t catch it, because the backtest only checks the model; I caught it by reading the diff.
• YNAB’s API allows 200 requests an hour. The month-by-month back-fill afterwards needed about 250, so the script learnt to sleep through rate limits and I went to bed.

Everything that writes to YNAB defaults to a dry run, in the same spirit as the Terraform guardrails, and every change is logged with its before and after values.

What stays with me

About a dozen decisions came to me in the end, and they were the right ones to come to me. Whether a payment to a friend was dinner or a holiday. Which work costs were expensed, which no model could know from the bank’s side, although Claude did notice one cloud bill matching a reimbursement to the penny. Eighteen low-confidence guesses are still sitting unapproved in YNAB, waiting for me to look at them, and I expect to get round to that in a timely fashion.

Throwaway data science

The ladder is the part I’d draw on a whiteboard, but the thing I’d actually take away from this is smaller. The obvious move now is to hand the whole backlog to a model and ask it to sort things out, and I think that would have been the worse option on nearly every count. Two thousand transactions, each with enough history attached to make sense of it, is a great many tokens, and the model would still have been guessing at conventions I’ve built up over eight years: that I don’t care which Tesco it was, or that a train on Mull is a holiday.

The classifier already knew those conventions, because it learnt them from 12,492 of my own decisions. It trained in a few seconds on a laptop and cost nothing to run, and its confidence scores told the models which 450 or so transactions were worth their attention. Nobody will maintain it, and I don’t think anybody needs to. Next time there’s a backlog I’ll train it again on whatever labels exist by then.

I’ve come to think this sort of disposable modelling is one of the more useful things an AI engineer can keep in the toolbox. It’s quick and cheap, but mostly it’s the piece that makes the expensive part behave. I suspect the models did better work on 450 hard cases than they would have done on two thousand easy ones mixed in with them. It was also a good deal more interesting than reading two thousand bank descriptions, which was the alternative I’d already tried.