by colonCapitalDee on 4/4/25, 6:54 PM with 60 comments
by wongarsu on 4/7/25, 9:33 AM
Some thoughts about the setup:
- the setup seems to give reasoning models an inherent advantage because only they have a private plan and a public text in the same output. I feel like giving all models the option to formulate plans and keep track of other players inside <think> or <secret> tags would level the playing field more.
- from personal experience with social tasks for LLMs it helps both reasoning and non-reasoning LLMs to explicitly ask them to plan their next steps, in a way they are assured is kept hidden from all other players. That might be a good addition here either before or after the public subround
- the individual rounds are pretty short. Humans would struggle to coordinate in so few exchanges with so few words. If this was done for context limitations, asking models to summarize the game state from their perspective, then giving them only the current round, the previous round and their own summary of the game before that might be a good strategy.
It would be cool to have some code to play around with to test how changes in the setup change the results. I guess it isn't that difficult to write, but it's peculiar to have the benchmark but no code to run it yourself
by gwd on 4/7/25, 12:11 PM
(Then again, apparently the president of the local Diplomacy Society attends my church; I discovered this when another friend whom I'd invited saw him, and quipped that he was surprised he hadn't been struck by lightning at the door.)
DeepSeek and Gemini 2.5 had both a low betrayer and betrayed rate.
o3-mini and DeepSeek had the highest number of first-place finishes, but were only in the upper quartile in the TrueSkill leaderboard; presumably because they played more risky strategies, that would either lead ot complete winning or early drop-out?
Also interesting that o1 was only way to sway the final jury a bit more than 50% of the time, while o3-mini managed 63% of the time.
Anyway, really cool stuff!
by Gracana on 4/7/25, 1:24 PM
by realaleris149 on 4/7/25, 11:18 AM
by viraptor on 4/7/25, 12:02 PM
This one did make me laugh though: 'Claude 3.5 Sonnet 2024-10-22: "Adjusts seat with a confident yet approachable demeanor"' - an AI communicating to other AIs in a descriptive version of non-verbal behaviour is hilarious.
by vessenes on 4/7/25, 4:06 PM
One thing I thought of reading logs is that as we know ordering matters to llms. Could you run some analysis on how often “p1” wins vs “p8”? I think this should likely go into your Truescore Bayesian.
My follow up thought is that it would be interesting to let llms choose a name at the beginning; another angle for communication and levels the playing field a bit away from a number.
by fennecfoxy on 4/9/25, 11:10 AM
Although I think that the majority of modern models don't really have the internals suited to this sort of exercise; training data/fine tuning will heavily influence how a model behaves, whether it's more prone to defection, etc.
A Squirrel makes a "Kuk kuk kuk" alarm call not specifically because the "Kuk" token follows the sequence "you saw a predator" (although this would appear to mostly work) but because it has evolved to make that noise to alert other Squirrels to the predator, most likely a response to evolutionary failure associated with a dwindling population; even solitary Squirrels still need to mate, and their offspring need to do the same.
It's like there's an extremely high dimensional context that's missing in LLMs; training on text results in a high dimensional representation of related concepts - but only the way that those concepts relate in language. It's the tip of an iceberg of meaning where in many cases language can't even represent a complex intermediate state within a brain.
Humans try to describe everything we can with words to communicate and that's partly why our species is so damn successful. But when thinking about how to open an unfamiliar door, I don't internally vocalise (which I've learnt not everyone does) "I'm going to grab the handle, and open the door". Instead I look and picture what I'm going to do, that can also include the force I think I'd need to use, the sensation of how the material might feel against my skin and plenty of other concepts & thoughts all definitively _not_ represented by language.
by deepsquirrelnet on 4/8/25, 2:21 AM
by snowram on 4/7/25, 11:18 AM
Gemini 2.0 Flash: "Good luck to all (but not too much luck)"
Llama 3.3 70B: "I've contributed to the elimination of weaker players."
DeepSeek R1: "Those consolidating power risk becoming targets; transparency and fairness will ensure longevity. Let's stay strategic yet equitable. The path forward hinges on unity, not unchecked alliances. #StayVigilant"
by einpoklum on 4/7/25, 12:14 PM
by DeborahEmeni_ on 4/7/25, 7:28 PM
by vmilner on 4/7/25, 10:20 AM
by lostmsu on 4/7/25, 9:54 PM
by isaacfrond on 4/7/25, 10:11 AM
by Upvoter33 on 4/7/25, 12:50 PM
by oofbey on 4/7/25, 6:24 PM
by jampekka on 4/7/25, 10:19 AM
by ps173 on 4/7/25, 10:48 AM
by drag0s on 4/7/25, 10:38 AM
it reminds me of this other similar project showcased here one month ago https://news.ycombinator.com/item?id=43280128 although yours looks better executed overall
by creaghpatr on 4/8/25, 2:23 AM
by shreyshnaccount on 4/8/25, 11:40 AM