from Hacker News

100M Token Context Windows

by gklitt on 8/29/24, 5:32 PM with 22 comments

  • by shazami on 8/29/24, 7:53 PM

    FYI wouldn't interview here. Got rejected after a 30 minute behavioral screen after spending 8 hours on an unpaid take-home.

  • by dinobones on 8/29/24, 7:54 PM

    Long context windows are IMO, “AGI enough.”

    100M context window means it can probably store everything you’ve ever told it for years.

    Couple this with multimodal capabilities, like a robot encoding vision and audio into tokens, you can get autonomous assistants than learn your house/habits/chores really quickly.

  • by smusamashah on 8/29/24, 6:42 PM

    It should be benchmarked against something like RULER[1]

    1: https://github.com/hsiehjackson/RULER (RULER: What’s the Real Context Size of Your Long-Context Language Models)

  • by fsndz on 8/29/24, 8:01 PM

    Context windows are becoming larger and larger, and I anticipate more research focusing on this trend. Could this signal the eventual demise of RAG? Only time will tell. I recently experimented with RAG and the limitations are often surprising (https://www.lycee.ai/blog/rag-fastapi-postgresql-pgvector). I wonder if we will see some of the same limitations for long context LLM. In context learning is probably a form of semantic / lexical cues based arithmetic.

  • by Sakos on 8/29/24, 8:00 PM

    I was wondering how they could afford 8000 H100’s, but I guess I accidentally skipped over this part:

    > We’ve raised a total of $465M, including a recent investment of $320 million from new investors Eric Schmidt, Jane Street, Sequoia, Atlassian, among others, and existing investors Nat Friedman & Daniel Gross, Elad Gil, and CapitalG.

    Yeah, I guess that'd do it. Who are these people and how'd they convince them to invest that much?

  • by anonzzzies on 8/30/24, 2:17 PM

    What is the state of art on context on open models? Magic won't be open I guess after getting 500m in VC money.

  • by samber on 8/29/24, 6:48 PM

    Based on Mamba ?

  • by htrp on 8/29/24, 9:45 PM

    does anyone have a detailed tech breakdown of these guys? not quite sure how their LTM architecture works.