from Hacker News

Genie: Best AI Software Engineer

by 0xedb on 8/12/24, 2:39 PM with 12 comments

• by pacomerh on 8/12/24, 4:56 PM

  These tools remind me of the Dreamweaver era of solving problems no matter how the code looks

• by uzumak on 8/12/24, 10:17 PM

  looks like they trained their model on SWE-bench and tried to submit https://github.com/swe-bench/experiments/pull/45

• by mronetwo on 8/12/24, 5:03 PM

  Sorry for not discussing the product itself, but...

  I'm just not seeing a machine that is "likely correct", constantly interrupting the "operator" to be that much of a win. I have seen some software influencers reflect on how much more fun it is to code, after dropping the LLM assistant.

  All of these feel like offerings to the Productivity God. As a salary guy I'll never get excited that I can do more during my work day. It's already easy to hit my capacity.

• by henning on 8/12/24, 5:18 PM

  So most of the time it still gets it wrong. And then when it gets it right it will still be subtly wrong. What a waste of electrical power and time.

• by difosfor on 8/12/24, 10:49 PM

  God I hate pages that hijack scrolling..

• by Y_Y on 8/12/24, 4:48 PM

  Any external verification ofthe benchmark results?

• by ramon156 on 8/12/24, 4:53 PM

  I thought this was going to be a blog post and just turned out to be a "use our product!" jumpscare. I'll gladly pass

• by Bjorkbat on 8/12/24, 5:25 PM

  Something I'm kind of curious about is the degree to which eval performance might be due to parts of the SWE-bench dataset getting into the latest LLM models.

  A while back someone on Twitter seemed to confirm that Claude-3.5 was aware of the Github issues inside the dataset by mentioning them, but I couldn't find the original post.

  30% performance on the full SWE-bench benchmark is quite the leap, but just how "real" of an achievement is this? Anecdotal reports mention that GPT-4o is marginally better than GPT-4 turbo at best, and yet agents leveraging the LLM did perform better.

  What would happen if SWE-bench was updated, top to bottom, with completely new Github issues? Would all these agents just completely shit the bed?

• by log101 on 8/12/24, 5:24 PM

  “…a human reasoning lab”

  closes the tab