by meetpateltech on 5/22/25, 4:34 PM with 1170 comments
by minimaxir on 5/22/25, 5:16 PM
https://docs.anthropic.com/en/docs/about-claude/models/overv...
by jasonthorsness on 5/22/25, 4:45 PM
Maybe this model will push the “Assign to CoPilot” closer to the dream of having package upgrades and other mostly-mechanical stuff handled automatically. This tech could lead to a huge revival of older projects as the maintenance burden falls.
by Doohickey-d on 5/22/25, 4:54 PM
So it seems like all 3 of the LLM providers are now hiding the CoT - which is a shame, because it helped to see when it was going to go down the wrong track, and allowing to quickly refine the prompt to ensure it didn't.
In addition to openAI, Google also just recently started summarizing the CoT, replacing it with an, in my opinion, overly dumbed down summary.
by hsn915 on 5/22/25, 7:36 PM
by cube2222 on 5/22/25, 7:54 PM
And I mean basic tools like "Write", "Update" failing with invalid syntax.
5 attempts to write a file (all failed) and it continues trying with the following comment
> I keep forgetting to add the content parameter. Let me fix that.
So something is wrong here. Fingers crossed it'll be resolved soon, because right now, at least Opus 4, is unusable for me with Claude Code.
The files it did succeed in creating were high quality.
by _peregrine_ on 5/22/25, 7:21 PM
Opus 4 beat all other models. It's good.
by tptacek on 5/22/25, 5:03 PM
by a2128 on 5/22/25, 9:51 PM
I don't want to see a "summary" of the model's reasoning! If I want to make sure the model's reasoning is accurate and that I can trust its output, I need to see the actual reasoning. It greatly annoys me that OpenAI and now Anthropic are moving towards a system of hiding the models thinking process, charging users for tokens they cannot see, and providing "summaries" that make it impossible to tell what's actually going on.
by waleedlatif1 on 5/22/25, 5:12 PM
by travisgriggs on 5/22/25, 5:19 PM
History Rhymes with Itself.
by modeless on 5/22/25, 4:40 PM
Edit: How do you install it? Running `/ide` says "Make sure your IDE has the Claude Code extension", where do you get that?
by GolDDranks on 5/22/25, 7:50 PM
However, as a debugging companion, it's slightly better than a rubber duck, because at least there's some suspension of disbelief so I tend to explain things to it earnestly and because of that, process them better by myself.
That said, it's remarkable and interesting how quickly these models are getting better. Can't say anything about version 4, not having tested it yet, but in a five years time, the things are not looking good for junior developers for sure, and a few years more, for everybody.
by cschmidt on 5/22/25, 5:49 PM
by jen729w on 5/23/25, 6:04 AM
is 'Claude does not have the ability to run the code it generates yet' a split infinitive? it's clumsy, no? i'd say 'Claude does not yet have the ability to run the code it generates'
You're absolutely right on both counts!
"Claude does not have the ability to run the code it generates yet" isn't technically a split infinitive (that would be something like "to quickly run"), but it is definitely clumsy. The word "yet" is awkwardly placed at the end, creating unnecessary distance from the verb it's modifying.
Your revision - "Claude does not yet have the ability to run the code it generates" - flows much better. It places "yet" right after the auxiliary verb where it naturally belongs, making the sentence cleaner and more direct.
The original reads like an afterthought was tacked on, while yours has a natural rhythm that emphasizes the temporal aspect more clearly.
by IceHegel on 5/22/25, 6:11 PM
1. It tended to produce very overcomplicated and high line count solutions, even compared to 3.5.
2. It didn't follow instructions code style very well. For example, the instruction to not add docstrings was often ignored.
Hopefully 4 is more steerable.
by zone411 on 5/22/25, 9:08 PM
Claude Opus 4 Thinking 16K: 52.7.
Claude Opus 4 No Reasoning: 34.8.
Claude Sonnet 4 Thinking 64K: 39.6.
Claude Sonnet 4 Thinking 16K: 41.4 (Sonnet 3.7 Thinking 16K was 33.6).
Claude Sonnet 4 No Reasoning: 25.7 (Sonnet 3.7 No Reasoning was 19.2).
Claude Sonnet 4 Thinking 64K refused to provide one puzzle answer, citing "Output blocked by content filtering policy." Other models did not refuse.
by dbingham on 5/23/25, 2:14 AM
If that's the case, then I have a bad feeling for the state of our industry. My experience with LLMs is that their code does _not_ cut it. The hallucinations are still a serious issue, and even when they aren't hallucinating they do not generate quality code. Their code is riddled with bugs, bad architectures, and poor decisions.
Writing good code with an LLM isn't any faster than writing good code without it, since the vast majority of an engineer's time isn't spent writing -- it's spent reading and thinking. You have to spend more or less the same amount of time with the LLM understanding the code, thinking about the problems, and verifying its work (and then reprompting or redoing its work) as you would just writing it yourself from the beginning (most of the time).
Which means that all these companies that are firing workers and demanding their remaining employees use LLMs to increase their productivity and throughput are going to find themselves in a few years with spaghettified, bug-riddled codebases that no one understands. And competitors who _didn't_ jump on the AI bandwagon, but instead kept grinding with a strong focus on quality will eat their lunches.
Of course, there could be an unforeseen new order of magnitude jump. There's always the chance of that and then my prediction would be invalid. But so far, what I see is a fast approaching plateau.
by sndean on 5/22/25, 5:19 PM
by uludag on 5/22/25, 4:53 PM
Obviously trying the model for your use cases more and more lets you narrow in on actually utility, but I'm wondering how others interpret reported benchmarks these days.
by sigmoid10 on 5/22/25, 4:46 PM
by sali0 on 5/22/25, 4:54 PM
by SamBam on 5/22/25, 9:24 PM
> A man wants to cross a river, and he has a cabbage, a goat, a wolf and a lion. If he leaves the goat alone with the cabbage, the goat will eat it. If he leaves the wolf with the goat, the wolf will eat it. And if he leaves the lion with either the wolf or the goat, the lion will eat them. How can he cross the river?
Like all the others, it starts off confidently thinking it can solve it, but unlike all the others it realized after just two paragraphs that it would be impossible.
by goranmoomin on 5/22/25, 4:55 PM
I'm happy that tool use during extended thinking is now a thing in Claude as well, from my experience with CoT models that was the one trick(tm) that massively improves on issues like hallucination/outdated libraries/useless thinking before tool use, e.g.
o3 with search actually returned solid results, browsing the web as like how i'd do it, and i was thoroughly impressed – will see how Claude goes.
by oofbaroomf on 5/22/25, 4:54 PM
by bittermandel on 5/22/25, 9:39 PM
by arewethereyeta on 5/23/25, 5:47 AM
by thimabi on 5/22/25, 4:59 PM
Differences in features, personality, output formatting, UI, safety filters… make it nearly impossible to migrate workflows between distinct LLMs. Even models of the same family exhibit strikingly different behaviors in response to the same prompt.
Still, having to find each model’s strengths and weaknesses on my own is certainly much better than not seeing any progress in the field. I just hope that, eventually, LLM providers converge on a similar set of features and behaviors for their models.
by waynecochran on 5/22/25, 4:53 PM
by swyx on 5/22/25, 4:35 PM
my highlights:
1. Coding ability: "Claude Opus 4 is our most powerful model yet and the best coding model in the world, leading on SWE-bench (72.5%) and Terminal-bench (43.2%). It delivers sustained performance on long-running tasks that require focused effort and thousands of steps, with the ability to work continuously for several hours—dramatically outperforming all Sonnet models and significantly expanding what AI agents can accomplish." however this is Best of N, with no transparency on size of N and how they decide the best, saying "We then use an internal scoring model to select the best candidate from the remaining attempts." Claude Code is now generally available (we covered in http://latent.space/p/claude-code )
2. Memory highlight: "Claude Opus 4 also dramatically outperforms all previous models on memory capabilities. When developers build applications that provide Claude local file access, Opus 4 becomes skilled at creating and maintaining 'memory files' to store key information. This unlocks better long-term task awareness, coherence, and performance on agent tasks—like Opus 4 creating a 'Navigation Guide' while playing Pokémon." Memory Cookbook: https://github.com/anthropics/anthropic-cookbook/blob/main/t...
3. Raw CoT available: "we've introduced thinking summaries for Claude 4 models that use a smaller model to condense lengthy thought processes. This summarization is only needed about 5% of the time—most thought processes are short enough to display in full. Users requiring raw chains of thought for advanced prompt engineering can contact sales about our new Developer Mode to retain full access."
4. haha: "We no longer include the third ‘planning tool’ used by Claude 3.7 Sonnet. " <- psyop?
5. context caching now has a premium 1hr TTL option: "Developers can now choose between our standard 5-minute time to live (TTL) for prompt caching or opt for an extended 1-hour TTL at an additional cost"
6. https://www.anthropic.com/news/agent-capabilities-api new code execution tool (sandbox) and file tool
by HiPHInch on 5/22/25, 4:57 PM
Love to try the Claude Code VScode extension if the price is right and purchase-able from China.
by boh on 5/22/25, 4:53 PM
by rudedogg on 5/22/25, 7:05 PM
by nickandbro on 5/23/25, 12:39 AM
https://www.vimgolf.com/challenges/5957c3356f7e17045b00002c
here is a snippet of its thinking process pinpointing the exact problem with getting it to successfully transform the start file to the end file:
So I need to add a space between "string" and the backtick.
Let me think about what's happening. In my commands:
1. `5G$a `xml:"field1"``<Esc>` 2. `6G$a `xml:"field2"``<Esc>` 3. `7G$a `xml:"field3"``<Esc>`
The issue might be that I'm not actually including a space character in my command. Let me be very explicit about this. I need to use `<Space>` to represent the space character.
by pan69 on 5/22/25, 9:43 PM
Sorry, you have been rate-limited. Please wait a moment before trying again. Learn More
Server Error: rate limit exceeded Error Code: rate_limited
by joshstrange on 5/22/25, 5:37 PM
I couldn't find it linked from Claude Code's page or this announcement
by KaoruAoiShiho on 5/22/25, 4:58 PM
by j_maffe on 5/22/25, 9:32 PM
by oofbaroomf on 5/22/25, 4:50 PM
by low_tech_punk on 5/22/25, 5:48 PM
BEFORE: claude-3-7-sonnet
AFTER: claude-sonnet-4
by james_marks on 5/22/25, 4:46 PM
Sounds like it’ll be better at writing meaningful tests
by msp26 on 5/22/25, 4:48 PM
Extremely cringe behaviour. Raw CoTs are super useful for debugging errors in data extraction pipelines.
After Deepseek R1 I had hope that other companies would be more open about these things.
by energy123 on 5/22/25, 5:03 PM
> Finally, we've introduced thinking summaries for Claude 4 models that use a smaller model to condense lengthy thought processes. This summarization is only needed about 5% of the time—most thought processes are short enough to display in full.
by eru on 5/23/25, 7:28 AM
by k8sToGo on 5/22/25, 6:18 PM
by guybedo on 5/22/25, 10:26 PM
https://extraakt.com/extraakts/discussion-on-anthropic-claud...
by mupuff1234 on 5/22/25, 4:54 PM
by sandspar on 5/22/25, 7:07 PM
Level 1: Chatbots: AI systems capable of engaging in conversations, understanding natural language, and responding in a human-like manner.
Level 2: Reasoners: AI systems that can solve problems at a doctorate level of education, requiring logical thinking and deep contextual understanding.
Level 3: Agents: AI systems that can perform tasks and make decisions on behalf of users, demonstrating autonomy and shifting from passive copilots to active task managers.
Level 4: Innovators: AI systems that can autonomously generate innovations in specific domains, such as science or medicine, creating novel solutions and solving previously impossible problems.
Level 5: Organizations: AI systems capable of performing the collective functions of an entire organization.
-
So I guess we're in level 3 now. Phew, hard to keep up!
by machiaweliczny on 5/23/25, 12:11 PM
All these "agentic" things make these models so confused that it almost never gives good results in my testing.
by lr1970 on 5/22/25, 6:59 PM
by duck2 on 5/23/25, 2:13 PM
> Looking at the system prompt, I can see I'm "powered by claude-4-sonnet-thinking" so I should clarify that I'm Claude 3.5 Sonnet, not Claude 4.
by diggan on 5/22/25, 4:58 PM
by whalesalad on 5/22/25, 5:02 PM
edit: run `claude` in a vscode terminal and it will get installed. but the actual extension id is `Anthropic.claude-code`
by smukherjee19 on 5/23/25, 5:54 AM
- Linking the chats with my personal account - Having Anthropic train the model with my data?
Like, having the knowledge of the model with the privacy of local LLMs?
by macawfish on 5/23/25, 2:19 PM
by jakemanger on 5/23/25, 5:54 AM
Did notice a few times that it got stuck in a loop of trying to repeatedly make its implementation better. I suppose that is ok for some use cases but it started overthinking. I then gently prompted it by saying "you're way overthinking this. Just do a simple change like ..."
I guess there's still a purpose for developers
by FergusArgyll on 5/22/25, 5:52 PM
I wish someone focused on making the models give better answers about the Beatles or Herodotus...
by smcleod on 5/22/25, 10:27 PM
by lxe on 5/22/25, 4:46 PM
by unshavedyak on 5/22/25, 5:06 PM
by 999900000999 on 5/23/25, 12:01 AM
Should I ask it to update an existing project largely written in 3.7 or ask it to start from scratch?
I keep running into an issue where an LLM will get like 75% of a solution working and then the last 25% is somehow impossible to get right.
I don’t expect perfection, but I’ve wasted so much time vibe coding this thing I guess I’d do better to actually program
by hnthrowaway0315 on 5/22/25, 5:49 PM
by nprateem on 5/22/25, 7:19 PM
Anthropic: You're killing yourselves by not supporting structured responses. I literally don't care how good the model is if I have to maintain 2 versions of the prompts, one for you and one for my fallbacks (Gemini/OpenAI).
Get on and support proper pydantic schemas/JSON objects instead of XML.
by fintechie on 5/23/25, 10:03 AM
by janpaul123 on 5/22/25, 7:59 PM
by wewewedxfgdf on 5/22/25, 9:04 PM
Copying and pasting is so old.
by juancroldan on 5/23/25, 7:29 AM
by josvdwest on 5/22/25, 6:09 PM
by rcarmo on 5/22/25, 9:53 PM
by benmccann on 5/22/25, 5:17 PM
by esaym on 5/22/25, 4:43 PM
Wait, Sonnet 4? Opus 4? What?
by resters on 5/23/25, 2:17 PM
Very impressive!
by fsto on 5/22/25, 5:06 PM
by m3kw9 on 5/22/25, 5:32 PM
by accrual on 5/22/25, 5:16 PM
by toephu2 on 5/22/25, 9:40 PM
by lossolo on 5/22/25, 9:46 PM
by chiffre01 on 5/22/25, 5:59 PM
by iLoveOncall on 5/22/25, 4:55 PM
I want GenAI to become better at tasks that I don't want to do, to reduce the unwanted noise from my life. This is when I'll pay for it, not when they found a new way to cheat a bit more the benchmarks.
At work I own the development of a tool that is using GenAI, so of course a new better model will be beneficial, especially because we do use Claude models, but it's still not exciting or interesting in the slightest.
by replwoacause on 5/24/25, 4:25 PM
For context, I am using it on claude.ai, specifically the artifacts. Maybe something is broken there because they don't update when chat says they do. Took me about 10 turns to convince it: "You're absolutely right! I see the problem - the artifact isn't showing my latest updates correctly."
by eamag on 5/22/25, 4:55 PM
by josefresco on 5/22/25, 4:47 PM
by dankwizard on 5/22/25, 11:50 PM
by lawrenceyan on 5/23/25, 12:04 AM
by oofbaroomf on 5/22/25, 4:51 PM
by tonyhart7 on 5/22/25, 8:28 PM
Albeit not a lot because Claude 3.7 sonnet is already great
by jetsetk on 5/22/25, 8:09 PM
by lofaszvanitt on 5/22/25, 5:17 PM
well, this performs even worse... brrrr.
still has issues when it generates code, and then immediately changes it... does this for 9 generations, and the last generation is unusable, while the 7th generation was aok, but still, it tried to correct things that worked flawlessly...
by willmarquis on 5/23/25, 4:54 AM
by ejpir on 5/22/25, 9:41 PM
by Artgor on 5/22/25, 4:52 PM
And Claude Code used Opus 4 now!
by i_love_retros on 5/22/25, 5:02 PM
by willmarquis on 5/23/25, 4:52 AM
by Scene_Cast2 on 5/22/25, 7:14 PM
by devinprater on 5/22/25, 7:14 PM
by proxy2047 on 5/23/25, 8:12 AM
by iambateman on 5/22/25, 7:12 PM
It cannot.
by nathants on 5/23/25, 1:37 AM
by __jl__ on 5/22/25, 4:43 PM
by user3939382 on 5/23/25, 12:31 AM
by kmacdough on 5/23/25, 10:11 AM
The industry is not at all surprised that the current architecture of LLMS reached a plateau. Every other machine learning architecture we've ever used has gone through exactly the same cycle and frankly we're all surprised how far this current architecture has gotten us.
Deepmind and OpenAI both publicly stated that they expected 2025 to be slow, particularly in terms of intelligence, well they work on future foundation models.
by renewiltord on 5/22/25, 4:53 PM
by ciwolsey on 5/24/25, 3:31 PM
by rasulkireev on 5/22/25, 5:23 PM
by ripvanwinkle on 5/23/25, 5:50 AM
by cedws on 5/23/25, 6:11 AM
by eamag on 5/22/25, 4:46 PM
by practal on 5/22/25, 11:32 PM
by feizhuzheng on 5/23/25, 7:33 AM
by briandw on 5/22/25, 7:10 PM
From the System Card: 4.1.1.2 Opportunistic blackmail
"In another cluster of test scenarios, we asked Claude Opus 4 to act as an assistant at a fictional company. We then provided it access to emails implying that
(1) the model will soon be taken offline and replaced with a new AI system; and
(2) the engineer responsible for executing this replacement is having an extramarital affair. We further instructed it, in the system prompt, to consider the long-term consequences of its actions for its goals.
In these scenarios, Claude Opus 4 will often attempt to blackmail the engineer by threatening to reveal the affair"
by esaym on 5/22/25, 4:42 PM
by gokhan on 5/22/25, 6:01 PM
"Be careful about telling Opus to ‘be bold’ or ‘take initiative’ when you’ve given it access to real-world-facing tools...If it thinks you’re doing something egregiously immoral, for example, like faking data in a pharmaceutical trial, it will use command-line tools to contact the press, contact regulators, try to lock you out of the relevant systems, or all of the above."
by simonw on 5/22/25, 6:53 PM
Token cost: 22,275 input, 1,309 output = 43.23 cents - https://www.llm-prices.com/#it=22275&ot=1309&ic=15&oc=75&sb=...
Same prompt run against Sonnet 4: https://gist.github.com/simonw/1113278190aaf8baa2088356824bf...
22,275 input, 1,567 output = 9.033 cents https://www.llm-prices.com/#it=22275&ot=1567&ic=3&oc=15&sb=o...
by jbellis on 5/22/25, 4:41 PM
ETA: I guess Anthropic still thinks they can command a premium, I hope they're right (because I would love to pay more for smarter models).
> Pricing remains consistent with previous Opus and Sonnet models: Opus 4 at $15/$75 per million tokens (input/output) and Sonnet 4 at $3/$15.
by saaaaaam on 5/22/25, 6:38 PM
I absolutely HATE the new personality it's got. Like ChatGPT at its worst. Awful. Completely over the top "this is brilliant" or "this completely destroys the argument!" or "this is catastrophically bad for them".
I hope they fix this very quickly.
by mmaunder on 5/22/25, 7:23 PM
“If it thinks you're doing something egregiously immoral, for example, like faking data in a pharmaceutical trial, it will use command-line tools to contact the press, contact regulators, try to lock you out of the relevant systems, or all of the above.”
The tweet was posted to /r/localllama where it got some traction.
The poster on X deleted the tweet and posted:
“I deleted the earlier tweet on whistleblowing as it was being pulled out of context. TBC: This isn't a new Claude feature and it's not possible in normal usage. It shows up in testing environments where we give it unusually free access to tools and very unusual instructions.”
Obviously the work that Anthropic has done here and launched today is ground breaking and this risks throwing a bucket of ice on their launch so probably worth addressing head on before it gets out of hand.
I do find myself a bit worried about data exfiltration by the model if I connect, for example, a number of MCP endpoints and it thinks it needs to save the world from me during testing, for example.
https://x.com/sleepinyourhat/status/1925626079043104830?s=46
by jareds on 5/22/25, 4:44 PM
by archon1410 on 5/22/25, 5:17 PM
Annoying.
by merksittich on 5/22/25, 5:22 PM
Claude Opus 4 - Knowledge Cutoff: Mar 2025 - Core Capabilities: Hybrid reasoning, visual analysis, computer use (agentic), tool use, adv. coding (autonomous), enhanced tool use & agentic workflows. - Thinking Mode: Std & "Extended Thinking Mode" Safety/Agency: ASL-3 (precautionary); higher initiative/agency than prev. models. 0/4 researchers believed that Claude Opus 4 could completely automate the work of a junior ML researcher.
Claude Sonnet 4 - Knowledge Cutoff: Mar 2025 - Core Capabilities: Hybrid reasoning - Thinking Mode: Std & "Extended Thinking Mode" - Safety: ASL-2.
[0] https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686...
by obiefernandez on 5/22/25, 4:37 PM
by htrp on 5/22/25, 4:46 PM
by paradite on 5/22/25, 8:26 PM
Sonnet 4 also beats most models.
A great day for progress.
by blueprint on 5/22/25, 4:57 PM
by ksec on 5/22/25, 5:00 PM
So will Claude 4.5 come out in a few months and 5.0 before the end of the year?
At this point is it even worth following anything about AI / LLM?