from Hacker News

Ask HN: What is Q* (Q star) at OpenAI and how does it threaten humanity

by quietthrow on 11/23/23, 5:05 PM with 13 comments

Reuters broke that the precursor to all the recent drama at open AI started with researchers at open AI writing to the board about a recent breakthrough they made and the threat it poses humanity. (It could be possible that Sam was aware of this and didn’t care but that’s a tangent)

Given where AI stands today what kind of breakthroughs are possible. What are the big gaps to AGI that exist today?

would be great to know of the gaps to follow progress and closeness to achieving AGI

  • by MAXPOOL on 11/23/23, 5:15 PM

    Speculating from the name only.

    Q* might be name derived from Q-learning and A* search algorithm.

    In that case it would be informed best best-first search using reinforcement learning.

  • by DicksonX on 11/29/23, 7:49 AM

    I think it's nothing but an obvious first step to have AGI not limited to fine tuned with static biases and human feedbacks. It's the idea I was in my mind for last 2 to 3 years. We use tree of thoughts chaain them and use a massive q learning probability array to find the best path for decision making. Seems a common sense concept and a known idea for long time. Open AI now moving from static rewards to dynamic rewards . That's AGI and agents will have the truth aligned by its own . A good step in mimicking us.

  • by wahnfrieden on 11/23/23, 5:20 PM

    Its threat to humanity is that VC-backed businesses will use it to justify regulatory capture and recommendations of total state authoritarianism under the guise of safety, leading us to autocrat rule and subsequent demise.

    It’s all out in the open, you can look at the papers coming from the EA community which as Frontier AI Regulation and the freedoms it claims are necessary to strip from society to protect ourselves.

  • by quietthrow on 11/23/23, 5:42 PM

    https://drpippa.substack.com/p/q-tigris

    Interesting but not sure who this author is.

  • by jonincanada on 11/23/23, 11:37 PM

    balderdash? "Q-star". Yes, the Q as in q-learning -- optimize a long term goal. The "star points" are the embedded algorithms discovered and joined within the transformer/NN architecture. Stars where formed after SGD discovered the best representation of said embedded alg type. I'm running a scaled down version myself -- somewhat impressive. Do it at 1k B parameters? hold my beer.

  • by andyjohnson0 on 11/24/23, 9:58 AM

    The Guardian is reporting [1] that Q* "was able to solve basic maths problems it had not seen before" and cites a paywalled article on The Information [2]. They also say "the pace of development behind the system had alarmed some safety researchers" and "The artificial intelligence model triggered such alarm with some OpenAI researchers that they wrote to the board of directors before Altman’s dismissal warning it could threaten humanity,"

    Sounds like it might be something notable, perhaps related to Q-learning and A* search as others here have speciulated. How it represents a specific or general existential threat is less clear, to me at least.

    [1] https://www.theguardian.com/business/2023/nov/23/openai-was-...

    [2] https://www.theinformation.com/articles/openai-made-an-ai-br...

  • by MrCoffee7 on 11/23/23, 5:17 PM

    Gary Marcus just put out a column about this: https://garymarcus.substack.com/p/about-that-openai-breakthr...

  • by NoZZz on 11/23/23, 7:34 PM

    I think it means that the letter Q is the answer to life the universe and everything. Notice the line entering the circle, it symbolises the initial act required to create life.