by erwald on 3/27/23, 9:00 PM with 256 comments
by WheelsAtLarge on 3/27/23, 10:02 PM
Reasons:
1) We are currently mining just about all the internet data that's available. We are heading towards a limit and the AIs aren't getting much better.
2) There's a limit to the processing power that can be used to assemble the LLM's and the more that's used the more it will cost.
3) People will guard their data more and will be less willing to share it.
4) The basic theory that got us to the current AI crop was defined decades ago and no new workable theories have been put forth that will move us closer to an AGI.
It won't be a huge deal since we probably have decades of work to sort out what we have now. We need to figure out its impact on society. Things like how to best use it and how to limit its harm.
Like they say,"interesting times are ahead."
by nuancebydefault on 3/27/23, 9:22 PM
Is this case really worth exploring? Or was the article written by a bored AI?
I find it striking that there are still so many people downplaying the latest developments of AI. We all feel that we are at the verge of a next revolution on par or even greater than the emergence of the www, while some people just can't to seem to let it sink in.
by karmasimida on 3/27/23, 10:08 PM
ChatGPT, at least for GPT-4, can already be considered as someone coined, baby AGI. It is already practical and useful, so it HAS to be REAL.
If it is already REAL, there is no need for another winter to ever come to reap the heads of liars. Instead AI will become applied technology, like cars, like chips. It will evolve continuously, and never go away.
by RandomLensman on 3/27/23, 9:20 PM
by codelord on 3/27/23, 10:00 PM
You can absolutely build high precision ML models. Using a transformer LM to sum numbers is dumb because the model makes little assumptions about the data by design, you can modify the architecture to optimize for this type of number manipulation or you can change the problem to generating code for summing values. In fact Google is using RL to optimize matmul implementations. That’s the right way of doing it.
by numinary1 on 3/27/23, 9:51 PM
by anonzzzies on 3/27/23, 9:25 PM
If advancing means ever larger and more expensive systems and ever more data, we will enter a cold winter soon.
by pixl97 on 3/27/23, 9:24 PM
And
>Take for example the sorting of randomly generated single-digit integer lists.
These seem like very confused statements to me.
For example, lets take banking. It's actually two (well far more) different parts. You have calculating things like interest rates and issues like 'sorting integers' like above. This is very well solved in simple software at extremely low energy costs. If you're having your AI model spend $20 trying to figure out if 45827 is prime, you're doing it wrong. The other half of banking is figuring out where to invest your money for returns. If you're having your AI read all the information you can feed it for consumer sentiment and passing that to other models, you're probably much closer to doing it right.
And guess what, ask SVB about 99% correct correct solutions that do/don't capture value. Solutions that have correct answers are quickly commoditized and have little value in themselves.
Really the most important statement is the last one, mostly the article is telling is the reasons why AI could fail, not that those reasons are very likely.
>I still think an AI winter looks really unlikely. At this point I would put only 5% on an AI winter happening by 2030, where AI winter is operationalised as a drawdown in annual global AI investment of ≥50%. This is unfortunate if you think, as I do, that we as a species are completely unprepared for TAI.
by kromem on 3/27/23, 10:46 PM
Two years ago was an opinion piece from NIST on the impact optoelectronics would bring specifically to neural networks and AGI, and watching as nearly every major research institution has collectively raised probably half a billion for AI photonics plays through their VC partnerships or internal resource allocations on the promise of order of magnitude improvements much closer than something like quantum computing, I think we really haven't seen anything yet.
We're probably just at the very beginning of this curve, not approaching its diminishing returns.
And that's both very exciting and terrifying.
After decades in tech (including having published a prediction over a decade ago that mid 2020s would see roles shift away from programming towards emergence of specialized roles for knowing how to ask AI to perform work in natural language) I think this is the sort of change so large and breaking from precedent we really don't know how to forecast it.
by blintz on 3/27/23, 9:24 PM
by endisneigh on 3/27/23, 9:47 PM
I think the current crop of AI is good enough. It will happen because people will actually grow resentful of things that AI can do.
I anticipate a small, yet growing segment of populations worldwide to start minimizing internet usage. This, will result in fewer opportunities for AI to be used and thus the lack of investment and subsequent winter.
by nico on 3/27/23, 9:38 PM
When this happened in the 60s-70s, the psychedelic revolution was crushed by the government. And we entered an AI winter.
I’m not implying causation. Just pointing out a curious correlation between the two things.
I wonder what will happen now.
by macrolime on 3/28/23, 2:14 PM
The text in itself won't be that interesting, the magic happens once you essentially train three different token predictors, one that predicts image tokens (16x16 pixels) and then combine that to predict video frames, one that predicts audio tokens and one that predicts text tokens. Then you use cross-attention between these predictors. To train this model you first pre-train the text predictor, after that's done you continue training the text predictor from the transcribed videos, while combining it with the video predictor and audio predictor with cross-attention.
Such a model will understand physics and actions in the real world much better than GPT-4, combined with all the knowledge from all the text on the internet it should turn out to be something quite interesting.
I think there probably doesn't exist enough compute yet to train such a model on something like all of YouTube, but I wouldn't be surprised if GPT-5 is a first step in this direction.
by thomastjeffery on 3/28/23, 2:50 PM
We can do better. All we have to do is be constructive when we write narratives about LLMs.
Unfortunately, that's hard work: we basically have to start over. Why? Because every narrative we have today personifies LLMs.
It's always a top-down perspective about the results, and never about how the actual thing works from the ground up.
The reality is that we never left AI winter. Inference models don't make decisions or symbolically define subjects.
LLMs infer patterns of tokens, and exhibit those patterns. They don't invent new patterns or new tokens. They rely entirely on the human act of writing: that is the only behavior they exhibit, and that behavior does not belong to the LLM itself.
We should definitely stop calling them AI. That may be the category of pursuit, but it misleadingly implies itself to be a descriptive quality.
I propose that we even stop calling them LLMs: they model tokens, which are intentionally misaligned with words. After tokenization, there is no symbolic categorization: no grammar definitions: just whatever patterns happen to be present between tokens.
That means a pattern that is not language will still show up in the model. Such a pattern may be considered by humans after the fact to be exciting, like the famous Othello game board study, or limiting, like prompts that circumvent guardrails. The LLM can't draw any distinction between grammar-aligned, desirable, or undesirable patterns; yet that is exactly what most people expect to be possible after reading about a personified Artificially Intelligent Large Language Model.
I would rather call them "Text Inference Models". Those are the clearest descriptors of what the thing itself is and does.
by crop_rotation on 3/27/23, 9:18 PM
by mpsprd on 3/27/23, 9:28 PM
The entertainment industry disagrees with this.
These systems are transformative for any creative works and in first world countries, this is no small part of the economy.
by stuckinhell on 3/27/23, 10:42 PM
The avengers if they had 90's actors is going viral.
https://cosmicbook.news/avengers-90s-actors-ai-art
Also the avengers as a dark sci fi https://www.tiktok.com/@aimational/video/7186426442413215022
AI art and generative text is just astounding, and it's only getting better.
by gumby on 3/27/23, 9:55 PM
I think the rather breathless posts (which I also remember from the 80s and apparently used to be common in the 60s when computers just appeared) will die down as the limits of the LLMs become more widely understood, and they become ubiquitous where they make sense.
by _nalply on 3/28/23, 7:55 AM
Of course, there's a bubble, but after that bubble pops, people will realize that current models are useful enough, even with their quirks. People all have quirks and mostly they get along, so they will accept quirks from machines. Anthropomorphizing machines will help accepting models. I know, this is dangerous, but I have this mental image: a doll in form of a seal baby with soft white fur with a Whisper model helping lonely handicapped people (note that I myself am a person with a disability, so don't cancel me, please). Or someone who technically is not very adept phoning for support and a Whisper model helping along and having a lot of time to chit-chat.
And technically I think, something will happen in about five years. A new floating number format, the posit (https://spectrum.ieee.org/floating-point-numbers-posits-proc...), is too useful to be ignored. It will take years because we need new hardware. Why do I think that posits are very useful? Posits could encode weights using very little storage (down to 6 bit per weight). Models perhaps need to be retrained using these weights because 6 bits are not precise. After all, you have only 64 different values. And I think with the new hardware supporting posits they will also have more memory for the weights. Cell phones will be able to run large and complex models efficiently. In other words, Moore's law is not dead yet. It just shifted to a different, more efficient computation implementation.
When this happens, immediate feedback could become feasible. With immediate feedback we do another step to achieve AGI. I could imagine that people get delivered a partially trained model and then they have a personal companion helping them through life.
by greatwave1 on 3/27/23, 9:21 PM
I was under the impression that the size and quality of the training dataset had a much bigger impact on performance versus the sophistication of the model, but I could be mistaken.
by skybrian on 3/27/23, 10:09 PM
The AI generators work similarly. They're like slot machines, literally built on random number generators. If you don't like the result, try again. When you get something you like, you keep it. There are diminishing returns to re-running the same query once you got a good result, because most results are likely to be worse than what you have.
Randomness can make games more fun at first. But I wonder how much of a grind it will be once the novelty wears off?
by nojvek on 3/28/23, 5:34 AM
LLMs - OpenAI, Google Brain, Meta FAIR, HuggingFace and others are now routinely training models with the entire corpus of the internet in a few months. The models are getting larger and more efficient.
Diffusion models - MidJourney, StableDiffusion, Dall-E and it's control net cousins - Trained on terabytes of images, almost entire corpus of internet.
Same with voice and other multimodal models.
The transformer algorithm is magical but we're just getting started.
There are now multiple competing players who can drop millions of dollars on compute and have access to internet sized datasets.
The compute, the datasets, the algorithms, the human reinforcement loops, all are getting better week over week. Millions of users are interacting with these systems daily. A large subset even paying for it.
There is the gold rush.
by muyuu on 3/27/23, 9:56 PM
in the late 90s and early 2000s, neural network had a significant stigma for being dead ends and were unpromising grads were sent - people didn't want to go there because it was a self-fulfilled prophecy that if you went to research ANNs then you were a loser, and you were seen as such, and in academia that is all you need to be one
but, in real life, they worked
sure, not for everything because of hardware limitations among other things, but these things worked and they were a useful arrow in your quiver as everybody else just did whatever was fashionable at the time (simulated annealing, SVMs, conditional random fields, you name it)
hype or no hype, if you know what you are doing and the stuff you do works, you will be okay
by ChatGTP on 3/27/23, 10:18 PM
Silicon Valley Tech is already promising that AI will be the likely solution to climate change..., if there is any more disruption to the economy it's just going to yet again slow down mitigation steps for climate change, thus having negative affects on the amount of capital available for these projects.
Printing money works, until it doesn't.
by beepbooptheory on 3/27/23, 10:15 PM
If there isn't a winter, will ChatGPT et al be able solve the energy crises they might be implicated in? Is there something in its magic text completion that can stop global warming? Coming famines?
Is perhaps the fixation on these LLMs right now, however smart and full of Reason they are, not paying the fullest attention to the existential threats of our meat world, and how they might interfere with what ever speculative dystopia/utopia we can imagine at the moment?
by totoglazer on 3/27/23, 9:17 PM
I think it’s unlikely, but no less likely than the compute issues mentioned.
by mikewarot on 3/27/23, 11:00 PM
There are too many single source suppliers in the chain up to EUV lithography. We may in fact be at peak IC.
by brucethemoose2 on 3/27/23, 9:09 PM
If they effectively shut out other hardware companies, that is going to slow scaling and price/perf reduction.
by chess_buster on 3/27/23, 9:38 PM
by HarHarVeryFunny on 3/27/23, 10:06 PM
In the meantime we've got LangChain showing what's possible when you give systems like this a chance to think more than one step ahead ...
I don't see an AI winter coming anytime soon... this seems more like an industry changing iPhone or AlexNet moment, or maybe something more. ChatGPT may be the ENIAC of the AI age we are entering.
by Hizonner on 3/27/23, 9:43 PM
by collaborative on 3/27/23, 10:08 PM
The winter before the AI winter will consist in all the cheap data disappearing. What fun will it be to write a blog post so that it can be scraped by a bot and regurgitated without attribution? Dito for code
Or, how will info sites survive without ad revenue? Last I checked bots don't consume ads
When the internet winter comes, all remaining sites will be behind login screens and a strict ToS popup
by stephc_int13 on 3/27/23, 10:04 PM
That said, I don't think we're going to see a new AI winter anytime soon, what we're seeing is already useful and potentially transformative with a few iterative improvements and infrastructure.
by javaunsafe2019 on 3/27/23, 10:02 PM
For sure we will get a lot stuff automated with it in the near future but this is far away from anything real intelligent.
It just doesn’t really understand and or feel things. It’s dead cause it just outputs data based on it’s model.
Intelligence contains a will and chaos.
by superb-owl on 3/27/23, 9:41 PM
It's late spring right now, a strange time to start forecasting winter.
by happycube on 3/28/23, 4:19 AM
There are too many actually useful things coming out of this for a true winter. And for there not to be a bubble.
by karmasimida on 3/27/23, 9:56 PM
ChatGPT as is, is already transformative. It CAN do human level reasoning really well.
The only winter I can see, is the AI gets so good, there is little incentive to improve upon it.
by zitterbewegung on 3/27/23, 10:06 PM
by ahofmann on 3/27/23, 9:32 PM
by boringuser1 on 3/27/23, 10:26 PM
by Havoc on 3/27/23, 9:54 PM
Humans are unreliable AF and we employ them just fine. Better reliability would certainly be nice but I don’t think it is strictly speaking necessary
by christkv on 3/27/23, 10:13 PM
My lawyer has been doing pretty much every public filing for civil cases and licenses assisted by GPT. So much bureaucracy could probably be removed by just having GPT validated permissions and manage the correctness of the submissions leaving a human to rubber stamp the final result if at all.
by andsoitis on 3/28/23, 2:56 PM
by arisAlexis on 3/28/23, 2:56 PM
by atleastoptimal on 3/27/23, 10:55 PM
by virtual_nikola on 3/29/23, 2:27 AM
by macawfish on 3/27/23, 10:08 PM
by thelazydogsback on 3/27/23, 10:47 PM
lol. 5%? - that's really laying it on the line
by HervalFreire on 3/27/23, 9:19 PM
I don't think compute is the issue. It's an issue with LLMs. Current LLMs are just a stepping stone for true AGI. I think there's enough momentum right now that we can avoid a winter and find something better through sheer innovation.