by qianli_cs on 5/20/25, 4:57 AM with 299 comments
by raesene9 on 5/20/25, 6:44 AM
For me the key quote in the article is
"Most scientists aren’t trying to mislead anyone, but because they face strong incentives to present favorable results, there’s still a risk that you’ll be misled."
Understanding people's incentives is often very useful when you're looking at what they're saying.
by rhubarbtree on 5/20/25, 6:24 AM
by angry_moose on 5/20/25, 5:14 PM
At best, they're sortof ok for linear, small deformation problems. The kind of models where we could get an exact solution in ~5 minutes vs a fairly sloppy solution in ~30 seconds. Start throwing anything non-linear in and they just fall apart.
Maybe enough to do some very high-level concept selection but even that isn't great. I'm reasonably convinced some of them are just "curvature detectors" - make anything straight blue, anything with high curvature red, and interpolate everything else.
by nicoco on 5/20/25, 6:09 AM
Anyway, winter is coming, innit?
by Flamentono2 on 5/20/25, 8:20 AM
I have not seen anything like it before. We literaly had not system or way of even doing things like code generation based on text input.
Just last week i asked for a script to do image segmentation with a basic UI and claude just generated that for me in under 1 Minute.
I could list tons of examples which are groundbreaking. The whole Image generation stack is completly new.
That blog article is fair enough, there is hype around this topic for sure, but alone for every researcher who needs to write code for their research, AI can make them already a lot more efficient.
But i do believe, that we have entered a new ara: An ara were we take data again very serious. A few years back, you said 'the internet doesn't forget' then we realized that yes the internet starts to forget. Google deleted pages, removed the cache feature and it felt like we stoped caring for data because we didn't knew what to do with it.
Then ai came along. And not only is now data king again but we are now in the mids of reinforcment ara: We now give feedback and the systems incorporate that feedback into their training/learning.
And the ai/ml topic is getting worked on on every single aspect of it: Hardware, Algorithm, use cases, data, tools, protocols, etc. We are in the middle of incorporating and building for and on it. This takes a little bit of time. Still the progress is crazy exhausting.
We will only see in a few years if there is a real ceiling. We do need more GPUs, bigger Datacenters to do a lot more experiments on AI architecture and algorithm. We have a clear bottleneck. Big companies train one big model for weeks and month.
by omneity on 5/20/25, 7:00 AM
by ausbah on 5/20/25, 3:48 PM
not really related to AI but this reflects a lesson I learned too late during some research in college: constant collaboration is important because it helps you avoid retreading over areas where others have already failed
by sublimefire on 5/20/25, 9:05 AM
by -__---____-ZXyw on 5/20/25, 10:36 PM
Title is:
"I got fooled by AI-for-science hype—here's what it taught me"
by shalmanese on 5/20/25, 8:02 AM
None of the claims made in the article are surprising because they’re the natural outgrowth of the hodgepodge of incentives we’ve accreted as what we call “science” over time and you just need to practice over time to be able to place the output of science in the proper context and understand that a “paper” is an artifact of a sociotechnical system with all the entailing complexity that demands.
by reify on 5/20/25, 7:30 AM
Most scientists aren’t trying to mislead anyone, but because they face strong incentives to present favorable results, there’s still a risk that you’ll be misled.
In other words scientists are trying to mislead everyone because there are a lot of incentives; money and professional status to name just two.
A common problem across all disciplines of science.
by pawanjswal on 5/20/25, 6:46 AM
by jxjnskkzxxhx on 5/21/25, 3:00 PM
This is true in so many aspects of human life - anyone trying to run an organisation should be aware of it.
by plasticeagle on 5/20/25, 7:25 AM
Where are the AI-driven breakthroughs? Or even the AI-driven incremental improvements? Do they exist anywhere? Or are we just using AI to remix existing general knowledge, while making no progress of any sort in any field using it?
by indoordin0saur on 5/20/25, 2:23 PM
by toss1 on 5/20/25, 2:56 PM
>>We also found evidence, once again, that researchers tend not to report negative results, an effect known as reporting bias.
>>But unfortunately, the scientific literature is not a reliable source for evaluating the success of AI in science.
>> One issue is survivorship bias. Because AI research, in the words of one researcher, has “nearly complete non-publication of negative results,” we usually only see the successes of AI in science and not the failures. But without negative results, our attempts to evaluate the impacts of AI in science typically get distorted.
While these biases will absolutely create overconfidence and wasted effort, the fact that there are rapid advances with some clear successes such as protein folding, drug discovery, &weather forecasting, leads me to expect there will be more very significant advances, in no small part because of the massive investment in funds and time to the problem of making AI-based advances.
For exactly the reasons this researcher spent his time and funds to research this, despite his negative results, there was learning, and the effect of millions of people effectively searching & developing will result in more great good advances being found and/or built.
Whether they are worth the total financial & human capital being spent is another question, but I'm expecting that to be also positive
by wrren on 5/20/25, 9:40 AM
by tonii141 on 5/20/25, 7:46 AM
by therebase on 5/20/25, 9:04 AM
Just about tech engineering here but I do think it transfers to science as well.
https://dev.to/sebs/the-quiet-crisis-how-is-ai-eroding-our-t...
by eviks on 5/20/25, 9:05 AM
Lesson learned: don't trust ads
> Most scientists aren’t trying to mislead anyone
More learning ahead, the exciting part of being a scientist!
by bwfan123 on 5/20/25, 3:04 PM
1) Instead of identifying a problem and then trying to find a solution, we start by assuming that AI will be the solution and then looking for problems to solve.
hammer in search of a nail
2) nearly complete non-publication of negative results
survivorship (and confirmation bias)
3) same people who evaluate AI models also benefit from those evaluations
power of incentives (and conflicts therein)
4) ai bandwagon effect, and fear of missing out
social-proof
by scuff3d on 5/20/25, 2:22 PM
This is a huge problem in software, and it's not restricted to AI. So much of what has been adopted over the years has everything to do with making the programmers life easier, but nothing to do with producing better software. AI is a continuation of that.
by shantnutiwari on 5/21/25, 8:30 AM
by abhinavsns on 5/20/25, 3:10 PM
by i_c_b on 5/20/25, 1:27 PM
This is the sort of thing I mean, I guess, by way of close parallel in a pre-AI context. For a while now, I've been doing a lot of private math research. Whether or not I've wasted my time, one thing I've found utterly invaluable has been the OEIS.org website, where you can just enter sequence of numbers and then search for it to see what contexts it shows up in. It's basically a search engine for numerical sequences. And the reason it has been invaluable is that I will often encounter some sequence of integers, I'll be exploring it, and then when I search for it on OEIS, I'll discover that that sequence shows up in much different mathematical contexts. And that will give me an opening to 1) learn some new things and recontextualize what I'm already exploring and 2) give me raw material to ask new questions. Likewise, Wolfram Mathematica has been a godsend. And it's for similar reasons - if I encounter some strange or tricky or complicated integral or infinite sum, it is frequently handy to just toss it into Mathematica, apply some combination of parameter constraints and Expands and FullSimplify's, and see if whatever it is I'm exploring connects, surprisingly, to some unexpected closed form or special function. And, once again, 1) I've learned a ton this way and gotten survey exposure to other fields of math I know much less well, and 2) it's been really helpful in iteratively helping me ask new, pointed questions. Neither OEIS nor Mathematica can just take my hard problems and solve them for me. A lot of this process has been about me identifying and evolving what sorts of problems I even find compelling in the first place. But these resources have been invaluable in helping me broaden what questions I can productively ask, and it's through something more like a high powered, extremely broad, extremely fast search. There's a way that my engagement with these tools has made me a lot smarter and a lot broader-minded, and it's changed the kinds of questions I can productively ask. To make a shaky analogy, books represent a deeply important frozen search of different fields of knowledge, and these tools represent a different style of search, reorganizing knowledge around whatever my current questions are - and acting in a very complementary fashion to books, too, as a way to direct me to books and articles once I have enough context.
Although I haven't spent nearly as much time with it, what I've just described about these other tools certainly is similar to what I've found with AI so far, only AI promises to deliver even more so. As a tool for focused search and reorganization of survey knowledge about an astonishingly broad range of knowledge, it's incredible. I guess I'm trying to name a "broad" rather than "deep" stance here, concerning the obvious benefits I'm finding with AI in the context of certain kinds of research. Or maybe I'm pushing on what I've seen called, over in the land of chess and chess AI, a centaur model - a human still driving, but deeply integrating the AI at all steps of that process.
I've spent a lot of my career as a programmer and game designer working closely with research professors in R1 university settings (in both education and computer science), and I've particularly worked in contexts that required researchers to engage in interdisciplinary work. And they're all smart people (of course), but the silofication of various academic disciplines and specialties is obviously real and pragmatically unavoidable, and it clearly casts a long shadow on what kind of research gets done. No one can know everything, and no one can really even know too much of anything out of their own specialties within their own disciplines - there's simply too much to know. There are a lot of contexts where "deep" is emphasized over "broad" for good reasons. But I think the potential for researchers to cheaply and quickly and silently ask questions outside of their own specializations, to get fast survey level understandings of domains outside of their own expertise, is potentially a huge deal for the kinds of questions they can productively ask.
But, insofar as any of this is true, it's a very different way of harnessing of AI than just taking AI and trying to see if it will produce new solutions to existing, hard, well-defined problems. But who knows, maybe I'm wrong in all of this.
by intended on 5/20/25, 7:21 AM
by yapyap on 5/20/25, 7:23 AM
by stonemetal12 on 5/20/25, 2:34 PM
by blitzar on 5/20/25, 2:44 PM
by rajnathani on 5/20/25, 9:55 AM
by thrdbndndn on 5/20/25, 7:04 AM
by BenFranklin100 on 5/20/25, 6:18 AM
A reason used to justify these massive cuts is that AI will soon replace traditional research. This post demonstrates this assumption is likely false.
by cadamsdotcom on 5/20/25, 9:06 PM
by KurSix on 5/20/25, 8:50 AM
by kumarvvr on 5/20/25, 7:02 AM
I mean, there ought to be an element of abstract thought, abstract reasoning, abstract inter-linking of concepts, etc, to enable mathematicians to solve complex math theorems and problems.
What am I missing?
by spwa4 on 5/20/25, 8:30 AM
by nathias on 5/20/25, 7:34 AM
by Workaccount2 on 5/20/25, 1:57 PM
I don't know if it's intentional, but the word "AI" means different things almost every year now. Its worse than papers getting released with "LLMs unable to do basic math" and then you see they used GPT-3 for the study.