from Hacker News

Self-hosted Whisper-based voice recognition server for open Android phones

by nichohel on 2/14/23, 3:00 AM with 5 comments

• by smoldesu on 2/14/23, 3:28 AM

  I suspect something similar is possible with ChatGPT. Using the GPT-neo-125m model I've been able to get some really convincing (if lackluster) answers on 4 core ARM hardware and less than 2gb of memory. With enough sampling, you can get legible paragraph-length responses out in less than 10 seconds; that's pretty good for an offline program in my book.

  I'm using rust-bert to serve it over a Discord bot, similar to one of their examples[0]. It's running on Oracle VCPUs right now, but with dedi hardware and ML acceleration I bet it would scream!

  [0] https://github.com/guillaume-be/rust-bert/blob/master/exampl...

• by nichohel on 2/14/23, 3:00 AM

  With this little bit of code you can use excellent voice recognition (ggerganov whisper.cpp port of Whisper) hosted on your own server, for your de-Googled Android phone, for text messaging, emails, search, and so on.