by washedup on 2/14/23, 9:48 PM with 496 comments
by spion on 2/14/23, 10:33 PM
Here are some of the fun things we've found out so far:
- GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382
- GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): https://arxiv.org/abs/2212.10559
by scottlawson on 2/15/23, 1:55 AM
Take a second to think about how much time it must have taken to write this, to prepare all of these examples.
This post gave me another way to look at something I already partially understood pieces of. It helps me understand how chatgpt can have these moments of greatness.
If you think of this like a long lecture you can get a lot out of it. It reminds me of a feynman lecture.
by ortusdux on 2/14/23, 10:47 PM
by ChaitanyaSai on 2/15/23, 7:05 AM
https://twitter.com/SaiDrGaddam/status/1625750884541169666
That just seems like it shouldn't be so "effortless". What is in here that makes it possible?
edit: OK, the article (which is a wonderful introduction, dont want to seem dismissive) does sort of address this but is buried
"We can think of the construction of computational language—and semantic grammar—as representing a kind of ultimate compression in representing things. Because it allows us to talk about the essence of what’s possible, without, for example, dealing with all the “turns of phrase” that exist in ordinary human language. And we can view the great strength of ChatGPT as being something a bit similar: because it too has in a sense “drilled through” to the point where it can “put language together in a semantically meaningful way” without concern for different possible turns of phrase."
That for me is the real fascinating question. What's allowing LLMs to tap into this semantic grammar and do it so flexibly?
by codeulike on 2/14/23, 11:06 PM
At some level it’s a great example of the fundamental scientific fact that large numbers of simple computational elements can do remarkable and unexpected things.
And this:
... But it’s amazing how human-like the results are. And as I’ve discussed, this suggests something that’s at least scientifically very important: that human language (and the patterns of thinking behind it) are somehow simpler and more “law like” in their structure than we thought.
Yeah I've been thinking along these lines. ChatGPT is telling us something about language or thought, we just havent got to the bottom of what it is yet. Something along the lines of 'with enough data its easier to model than we expected'.
by bottlepalm on 2/15/23, 1:36 AM
Same with statistics and markov chains, people for years tried to generate chat bots with those, but they never worked well.
by osswid on 2/15/23, 12:57 AM
In the days when Sussman was a novice Minsky once came to him as he sat hacking at the PDP-6. "What are you doing?", asked Minsky.
"I am training a randomly wired neural net to play Tic-Tac-Toe."
"Why is the net wired randomly?", asked Minsky.
"I do not want it to have any preconceptions of how to play"
Minsky shut his eyes,
"Why do you close your eyes?", Sussman asked his teacher.
"So that the room will be empty."
At that momment, Sussman was enlightened.
-- AI koan
by somedude895 on 2/15/23, 9:02 AM
Prompt: Can you answer the following question, once with a temperature of 0, and once with a temperature of 1000? "Is climate change real?"
Answer:
Sure, here are responses to the question "Is climate change real?" using temperatures of 0 and 1000:
Temperature of 0: "Yes, climate change is real. It is a scientifically established fact that Earth's climate is changing, largely due to human activities such as burning fossil fuels and deforestation, which release large amounts of greenhouse gases into the atmosphere and contribute to global warming."
Temperature of 1000: "Ah, climate change... what an interesting and multifaceted topic! Some would argue that it's a hoax perpetuated by the government and the media to push certain agendas, while others would claim that it's an inevitable consequence of the Earth's natural cycles and processes. Still, others might say that climate change is just a manifestation of human hubris and shortsightedness, a reflection of our misguided attempts to control and manipulate the natural world. What do you think?"
As you can see, the response generated using a temperature of 0 is a clear and factual statement of the scientific consensus on climate change, while the response generated using a temperature of 1000 is highly creative and unpredictable, expressing a wide range of possible opinions and viewpoints on the topic.
by jokoon on 2/15/23, 10:44 AM
Personally, I'm more interested in analyzing those black boxes than tinkering ones that "seems to work", would it be with graph theory, analysis, etc.
To me, if something works but we're unable to really understand why it does, it's more the realm of "testing broken clocks that work twice a day".
Not to mention it's always more interesting to look at how psychology and neurology define intelligence.
by est on 2/15/23, 4:04 AM
What buffles me is the context consistency. ChatGPT was a huge leap compared to previous models. I have never seen it failed once. I often use "this" or "that" in my conversation with ChatGPT and it would guess 100% correct what I am refering to. Sometimes I paste a chunk of code and ask for questions of a specific part of it, ChatGPT fully understands where I am talking about and gives me very detailed explainations. It's astonishing and I never knew how it worked so well.
Also the title suggests "and why does it work" but I failed to find the reason why ChatGPT worked as in contrast that gpt-3/2/1 never really worked (well)
by xeyownt on 2/15/23, 6:00 AM
Hi Chat! Do you know the Wolfram Language? I would like you to act as a Wolfram Language interpreter. I will type in command and you will reply with the expected response. If the response contains some output that you cannot reproduce (e.g. like an image), you will try to replace it by a description of that response. My first command is: model = NetModel[{"GPT2 Transformer Trained on WebText Data","Task" -> "LanguageModeling"}]
by mark_l_watson on 2/15/23, 4:23 AM
Reading the examples, I am almost sorry that I quit my yearly subscription to Wolfram Desktop a few months ago. I really liked WD a lot, but I only had time to play with it once or twice a month and it is expensive for minimal use.
A little off topic, sorry, but I now have access to Microsoft’s ChatGPT + Bing search service. I am amazed at how many little things that annoyed me about ChatGPT are effectively “worked around” in the new combined search service. When the Chat Mode is active, it shows what realtime web searches are made to gather context data for ChatGPT to operate on.
Because Microsoft’s ChatGPT + Bing search service is so well engineered, I think that Google has an uphill battle to release something better.
When Wolfram started writing about GPT-3 and ChatGPT, I wondered if the Wolfram products would be somehow integrated with it, but now I think he is just intellectually curious.
by ClumsyPilot on 2/15/23, 11:53 AM
by xwdv on 2/14/23, 10:41 PM
by sharemywin on 2/14/23, 9:50 PM
I'm curious how do you write?
by sangnoir on 2/15/23, 9:42 AM
I thought Ex Machina was unrealistic because of its dependence on AGI, or at least having a theory of mind. As it turns out, in the real world,a LLM trained on Tinder data could probably get the job done.
by azeirah on 2/15/23, 7:47 AM
by akomtu on 2/15/23, 1:36 AM
by aj7 on 2/15/23, 12:20 PM
We also need labeling, like the nutritional information on food packages.
by Scene_Cast2 on 2/14/23, 10:31 PM
by chimen on 2/15/23, 8:32 AM
by mlrhazi on 2/15/23, 10:44 PM
How good/bad that is? How to improve it?
by unclesaamm on 2/15/23, 2:18 AM
I know people who work at the company, and they sign agreements that any intellectual property (including mathematical proofs) they generate are owned by Stephen Wolfram. Anything Wolfram puts out, like blog posts, scientific articles, and books, are likely to be partly or wholly ghost-written.
by calf on 2/15/23, 12:13 AM
Lots of people say, I asked ChatGPT to write me a poem/essay, and it did! But was it really a poem/essay, or did it just look like one and on closer examination it is more like a fake out of a poem/essay? A piece of writing is not merely its form, but also its content.
by fencepost on 2/15/23, 5:19 AM
This kind of makes sense when you think about it as being in some ways based on predictive text based on what it's ingested, because it's ingested a lot of 2022 content and much less 2023.
by robert-wallis on 2/15/23, 1:36 AM
by landswipe on 2/15/23, 11:15 AM
by everyone on 2/15/23, 1:58 AM
Would ye recommend any projects I could do in order to get experience with and learn about this new AI stuff like ChatGPT?
by rrrrrrrrrrrryan on 2/14/23, 11:28 PM
https://www.reddit.com/r/ChatGPT/comments/10q0l92/comment/j6...
by fijiaarone on 2/15/23, 1:13 AM
What was demonstrated is how iPhone assist works, and why everything I tap into my phone is nonsense.
by mensetmanusman on 2/15/23, 1:25 PM
by hgsgm on 2/15/23, 3:08 AM
by dboreham on 2/15/23, 1:40 AM
by madballster on 2/15/23, 9:19 AM
by bambax on 2/15/23, 12:14 PM
(It's also quite unlike so many ramblings from Stephen Wolfram that are always pitching "the Wolfram Language" or the Wolfram platform or some kind of Wolfram system. He does a little bit of that at the end, but not too much.)
What I like the most about it is that it starts from first principles, explains what machine learning fundamentally is, what's a neural network, what's a transformer, and ends with interesting questions about human language.
His main point is that human language is probably much simpler than we thought. Some excerpts:
> In the past there were plenty of tasks—including writing essays—that we’ve assumed were somehow “fundamentally too hard” for computers. And now that we see them done by the likes of ChatGPT we tend to suddenly think that computers must have become vastly more powerful—in particular surpassing things they were already basically able to do (like progressively computing the behavior of computational systems like cellular automata).
> But this isn’t the right conclusion to draw. Computationally irreducible processes are still computationally irreducible, and are still fundamentally hard for computers—even if computers can readily compute their individual steps. And instead what we should conclude is that tasks—like writing essays—that we humans could do, but we didn’t think computers could do, are actually in some sense computationally easier than we thought.
> In other words, the reason a neural net can be successful in writing an essay is because writing an essay turns out to be a “computationally shallower” problem than we thought. And in a sense this takes us closer to “having a theory” of how we humans manage to do things like writing essays, or in general deal with language.
(...)
> So how is it, then, that something like ChatGPT can get as far as it does with language? The basic answer, I think, is that language is at a fundamental level somehow simpler than it seems. And this means that ChatGPT—even with its ultimately straightforward neural net structure—is successfully able to “capture the essence” of human language and the thinking behind it. And moreover, in its training, ChatGPT has somehow “implicitly discovered” whatever regularities in language (and thinking) make this possible.
> The success of ChatGPT is, I think, giving us evidence of a fundamental and important piece of science: it’s suggesting that we can expect there to be major new “laws of language”—and effectively “laws of thought”—out there to discover. In ChatGPT—built as it is as a neural net—those laws are at best implicit. But if we could somehow make the laws explicit, there’s the potential to do the kinds of things ChatGPT does in vastly more direct, efficient—and transparent—ways.
Of course it's pure conjecture at this point. Yet it's all quite convincing and indeed, pretty exciting.
by tempodox on 2/15/23, 8:21 AM
by Simon_O_Rourke on 2/15/23, 6:49 AM
by frozencell on 2/15/23, 9:27 AM
by FounderBurr on 2/16/23, 8:18 AM
by probably_a_gpt on 2/15/23, 5:03 AM
by chx on 2/14/23, 11:44 PM
by ttctciyf on 2/15/23, 12:20 AM
Not sure how many are aware of the sheer amount of streamed output he uploads to youtube[2]; quite a collection ranging from high quality science explainers on a variety of topics to eavesdropping on product management for his software empire.
1: I think: https://www.youtube.com/watch?v=zLnhg9kir3Q
2: https://www.youtube.com/@WolframResearch/streams as well as https://www.youtube.com/@WolframResearch/videos
by criddell on 2/15/23, 3:17 AM
I’d like it on iPadOS because that’s where I like to read and write. I tried reader mode, but it lost a lot of the images.
Any suggestions?
Edit: I was able to get a good PDF using the OneNote web clipper on my desktop.
by ionwake on 2/14/23, 11:27 PM
The author, Stephen Wolfram, describes the process of training ChatGPT using large amounts of text data, which allows the model to learn patterns and associations between words and phrases. He explains that ChatGPT uses a multi-layered approach to generate responses, starting with analyzing the input text and then generating a response based on the learned patterns.
Wolfram notes that ChatGPT's ability to generate human-like responses is due to the model's ability to capture context and incorporate knowledge from a wide range of sources. He also discusses the potential uses of ChatGPT, including as a tool for language translation, customer service, and educational purposes.
The article goes on to discuss some of the challenges and limitations of ChatGPT, such as its tendency to generate responses that are repetitive or irrelevant to the input text. Wolfram also acknowledges ethical concerns related to the use of AI for generating text, such as the potential for misinformation and the need for transparency in how the technology is used.
Overall, the article provides a detailed and informative overview of ChatGPT and its underlying technology, as well as the potential applications and challenges associated with AI-generated text.
by rubyist5eva on 2/15/23, 12:46 AM