from Hacker News

Ask HN: Could an LLM predict someone's speech 5 seconds ahead in realtime?

by comova on 12/17/24, 2:14 AM with 3 comments

LLMs are in many ways text predictors. Could you have an LLM that runs in realtime, watching a speech on TV for example, and predicts what the person is going to say 5 seconds in advance based on what they are currently saying? My friend was thinking about this as a test for State of the Union but then we realized that the transcript is published ahead of time so it would not be that impressive. But maybe something like the inauguration speech? Especially if tuned on previous speeches from that person.

  • by gregjor on 12/17/24, 3:04 AM

    > LLMs are in many ways text predictors.

    Actually, in every way. But LLMs don't "predict" in the sense of predicting the future. They predict from probabilities based on training data. They auto-complete based on samples of words used in similar contexts before.

    Since politicians tend to deliver stock messages, using a specific kind of language, and repeat themselves, an LLM may very well successfully auto-complete something as canned and, well, predictable as a state of the union address or inauguration speech. The vocabulary and space of novel thoughts will stay small in such contexts (more or less depending on the speaker). A bot to imitate almost any president since Lincoln would only need slightly more sophistication than a word processor.

    LLMs would have less success "predicting" what I might say in a conversation unless I have written something similar before and that got into the training data.

    Using an LLM to predict or simulate what someone might say introduces the temptation to confuse the prediction with what someone would actually say. Imagine talking to someone who interrupts to finish your sentences. They may get it right, more or less, probably not verbatim.

  • by compressedgas on 12/17/24, 2:52 AM

    I would suggest you read Strobelt and Gehrmann _Catching a Unicorn with GLTR: A tool to detect automatically generated text_ (2019) http://gltr.io/

  • by joegibbs on 12/17/24, 2:27 AM

    You could, the further into the future that it predicts the less accurate the prediction will be, but you could probably predict a few sentences of the speech correctly. The LLM can only guess from historical context and grammar though, and the speaker is unlikely to make the most likely next word choice for all words (there are some sentences where the next word is extremely likely, mostly aphorisms, but a lot of the time you don't know where it's going).

    You would want to use a raw LLM which just does next token completion rather than a model like Claude or ChatGPT, and you could probably use it to display branching chains of next most likely strings.