Historic Project: Using LLMs to get Metrics from Text
Using LLMs to get Metrics from Text
note that this is an old piece of writing. I’ve dated it to reflect when I was doing the work and taking notes, but it reads in the past tense as this is when I’m writing it up. I hope this isn’t too confusing.
the thing I built
below you can see a little embedded dashboard I built over a year ago now (actually more than that, but the code only made it to github then). it takes a youtube video, chunks up the transcript, and the backend (not part of the dashboard) allowed a use to specify a bunch of quantifiable questions to be asked of each chunk - questions which are more complicated than a yes/no, but not open ended like a real value or a free text field.
the reason for this was to try to look at using llms in a creative way which most people around me weren’t at the time. namely: don’t aske them for yet more text, but ask them to answer simple repetitive questions of a corpus of text, and use that to generate metrics
have a look below…
the business need, as it were
you’ll find that the video i looked at here is an incredibly supplier call, run by a very patient team at HMRC. someone at my firm wasn’t able to make the original call and asked me off-handedly something like “can you use AI to get something interesting out of the call or make it easier to know where to look”
learning
when looking at the graph, do so with a pinch of salt - this was several iterations of models ago, and this really was a proof-of-concept. i went through a few iterations, but if i do this again there will be some big improvements to make (not least there’s function calling now, so my trick of manipulating gpt3.5 into cramming outputs into a valid json slug and then post-processing wouldn’t be necessary now!)
in any case, it definitely taught me that you can get genuinely interesting metrics out an body of text you can chunk up or put in an order, providing you have some interesting questions to ask of capable llm.