from Hacker News

Ask HN: When did OpenAI first observe emergent reasoning and generalization?

by semessier on 4/17/24, 7:15 PM with 2 comments

I am wondering when the seminal moment occurred that signs of reasoning and generalization transpired when scaling the parameters and training data in transformer-based models. Presumably it happened at OpenAI, but that's just an assumption. It would be fascinating to learn more on the moment of discovery and if there are any technical write-ups or blog posts detailing the -- fair to say -- historical process and findings. Does anybody have insights or pointers to share on this topic?

  • by cmcollier on 4/18/24, 5:57 PM

    In terms of "the moment", I would imagine it happened during development inside Google (Lamda) or OpenAI (GPT2/3).

    More technically, here's one of the key papers discussing the topic (from google):

    * https://arxiv.org/abs/2206.07682

    Emergent Abilities of Large Language Models

    Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models.

    Version history (for relevant dates):

       [v1] Wed, 15 Jun 2022 17:32:01 UTC (59 KB)
   [v2] Wed, 26 Oct 2022 05:06:24 UTC (88 KB)

  • by illuminant on 4/17/24, 7:23 PM

    Emergence literally means to create more potential than the mere sum of parts.

    By this technical definition, the moment it did something useful which the parts themselves could not do was an emergent moment.

    Remember the very first release? GPT scared OpenAI so much they didn't release it to the public until after pressure from competing open source alternatives began to surface.

    Maybe you want to check the basic algorithms? When where they first demonstrating non gibberish?

    Most changes since have been refinements.