from Hacker News

Show HN: A lightweight AI gateway to 100+ models, in TS

by roh26it on 1/8/24, 1:35 PM with 13 comments

We are Rohit & Ayush, we created Portkey this year March to help tackle some challenges we had seen while building apps based on GPT3, 3.5, 4, and the DevOps principles we brought to the scene to help tackle them.

We believe, a solid, performant, and reliable gateway lays the foundation to help build the next level of LLM apps. It decreases excessive reliance on any one company and takes the focus back to building instead of spending time fixing the nitty gritties of different providers and making them work together.

Features:

  Blazing fast (9.9x faster) with a tiny footprint (~45kb installed)
  Load balance across multiple models, providers, and keys
  Fallbacks make sure your app stays resilient
  Automatic Retries with exponential fallbacks come by default
  Plug-in middleware as needed
  Battle tested over 100B tokens and millions of requests
For the folks serious about gateway, separation of concerns, TS developers.. I'd love to hear your thoughts and we're hungry for feedback!

Reach out to us at hello@portkey.ai or explore the project: https://github.com/portkey-ai/gateway

  • by shaial on 1/8/24, 8:43 PM

    Congrats, this looks neat, and surely great to have more TS products in the ecosystem.

    One plugin or feature that I will like to see in an AI gateway: *Cache* per unique request. So if I send the same request (system, messages, temperature, etc.), I will have the option to pull if from a cache (if it was already populated) and skip the LLM generation. This is much faster and cheaper - especially during development and testing.

  • by ayush-garg on 1/8/24, 1:58 PM

    A few months back, we embarked on what seemed like another regular project. Little did we know, it would turn into such a game changer for LLM deployments.

    Feels surreal that this gateway is already processing upwards of 3B tokens a day in such a short time.

  • by retrovrv on 1/8/24, 1:40 PM

    Pretty excited to announce this! While there are some popular and awesome AI gateways out there, like litellm, bricksai - none are written in TS, and for the TS ecosystem. Looking forward to the community's feedback

  • by CShorten on 1/8/24, 1:58 PM

    Portkey is super cool, congrats Rohit and Ayush on the HN launch!

  • by aravindputrevu on 1/8/24, 2:29 PM

    Congratulations Rohit, Ayush and team Portkey on the launch!

    Excited to take it for a spin and contribute!

  • by pranavgoyal on 1/8/24, 5:32 PM

    Congrats on the launch folks! This sounds really amazing - rooting for success!

  • by kesavan_kk on 1/9/24, 2:14 AM

    Congratulations on this launch. Looks awesome.

  • by kaushik92 on 1/8/24, 6:33 PM

    Congrats on the launch! Looks amazing