from Hacker News

Ask HN: Browser-Based LLM Models?

by lulzury on 7/7/24, 5:25 PM with 5 comments

Does anyone know if there are there any plans for browsers to natively integrate LLMs, LLM APIs, or LLM models like Llama for local use by web applications?

I feel there's a large opportunity here for a more privacy-friendly, on-device solution that doesn't send the user's data to OpenAI.

Is RAM the current main limitation?

by throwaway888abc on 7/7/24, 5:30 PM
https://simonwillison.net/2024/Jul/3/chrome-prompt-playgroun...
https://developer.chrome.com/docs/ai/built-in
by throwaway425933 on 7/7/24, 10:31 PM
Every big tech company is trying to do this. FB (through whatsapp), Google (through chrome/Android), Apple (through Safari/iOS/etc). As soon as they meet their internal metrics, they will release these to public
by FrenchDevRemote on 7/7/24, 7:41 PM
"Is RAM the current main limitation?"
(V)RAM+processing power+storage(I mean what kind of average user wants to clog half their hard drive for a subpar model that output 1 token a second?)
by Crier1002 on 7/8/24, 4:36 AM
check out https://github.com/mlc-ai/web-llm
IMO the main limitation is access to powerful GPUs for running models locally and the size of some models causing UX problems with cold starts