from Hacker News

Show HN: Built a Real Time Visual Intelligence

by Aeroi on 2/26/25, 6:03 PM with 1 comments

I built a realtime visual intelligence that connects a users phone camera to a multimodal llm. I use the pipecat open source framework, webrtc, and a few other services to connect it all together.

It's similar to chatgpt advanced voice and grounded with google_search for asynch internet searches based on transcripts or frames from the video that run at 1fps to the LLM.

Let me know what you think and if you want to work on some fun scaling problems with me on this project.

www.withsen.com

  • by Aeroi on 2/26/25, 6:09 PM

    One interesting note with voice AI is that you can shove static datasets into the long context windows of these newer models like 2.0-flash-lite. It creates a Model Assisted Generation(MAG) and returns super low latency and 99% relevant information to the bot. Theres a good example in the foundational example of the pipecat github.