from Hacker News

Show HN: Monitoring your OpenAI usage without a third party

by zekone on 10/4/23, 12:54 PM with 4 comments

LLM observability is an absolute must-have for anyone running something in prod (or prod-like). While all the observability startups are great, you're essentially sending all your OpenAI usage history - prompts, generations, chats - to a random third party.

So this script deploys a basic proxy in your Azure account, catches all incoming OpenAI requests, stores logs in your own resource group, and comes with visualizations premade (charts, timelines, chat history, cost estimation, etc).

Thanks for any thoughts and feedback!

  • by iamjackg on 10/4/23, 2:03 PM

    This looks great, although the Azure requirement makes it a no-go for me. I just keep wishing for a simple solution that is easy to self-host locally. After a lot of work I managed to get an instance of Helicone running, but it's made of 6 or 7 different services, each with a pretty large footprint.

    I feel like there's definitely room for a single-binary tool made in Go with a small sqlite database and in-memory caching.

  • by siva7 on 10/5/23, 3:27 PM

    Thanks, that's great work. I will use it.