from Hacker News

Hidden Rate Limits: How Providers Throttle LLM Throughput During Peak Demand

by carlcortright on 3/21/24, 9:36 PM with 4 comments

• by carlcortright on 3/21/24, 9:36 PM

  Over the past few days we did an investigation of the main LLM providers, and have observed up to a 40% difference in average speed (tokens / second) from the leading LLM providers like GPT4.

• by canada_dry on 3/24/24, 2:37 AM

  Looking ahead, I suspect as AI becomes even more ubiquitous/mainstream, AI service providers will offer various levels of analysis at different price points. E.g. the cheapest service will provide reliably accurate answers, but only to simple queries that consume little compute power.

  Also envisioned is the all too common race-to-the-bottom scenario where services will simply tune their service to respond with the least compute power needed while harvesting and capitalizing on it users data.