by mychaelangelo on 2/1/25, 3:12 PM with 84 comments
by yapyap on 2/1/25, 4:44 PM
(small spoiler)
In Money Monster it turns out the hedge fund manager who is blaming their state of the art AI bot for malfunctioning and rapidly selling off certain stock, tanking it because of that, did so out of a machine code error. He can’t explain what the error was or how or why it happened cause ‘he didn’t program their trading bot, some guy in Asia did.’ But as it turns out he did do it in some way.
I feel like using AI as a way to abstract blame even more when something goes wrong will be a big thing, even when secretly it was not the AI (ML) or who trained the thing’s fault.
by alephnerd on 2/1/25, 4:50 PM
It feels like a lot of "Existential AI Risk" types are divorced from the physical aspects of maintaining software - eg. your model needs hardware to compute, you need cell towers and fiber optic cables to transmit.
It feels like they always anthropomorphize AI as some sort of "God".
The "AI Powered States" aspect is definetly pure sci-fi. Technocratic states have been attempted, and econometrics literally the exact same mathematical models used in AI/ML (Shapely values are an Econometrics tool, Optimization Theory itself got it's start thanks to GosPlan and other attempts and modeling and forecasting economic activity, etc).
As we've seen with the Zizian cult, very smart people can fall into a fallacy trap of treating AI as some omnipotent being that needs to either be destroyed or catered to.
by randomcatuser on 2/1/25, 6:18 PM
For example:
> "Instead of merely(!) aligning a single, powerful AI system, we need to align one or several complex systems that are at risk of collectively drifting away from human interests. This drift can occur even while each individual AI system successfully follows the local specification of its goals"
Well yes, making systems and incentives is a hard problem. But maybe we can specify a specific instance of this, instead of "what if one day it goes rogue!"
In our society, there are already many superhuman AI systems (in the form of companies) - and somehow, they successfully contribute to our wellbeing! In fact life is amazing (even for dumb people in society, who have equal rights). And the reason is, we have categorized the ways it goes rogue (monopoly, extortion, etc) and responded adequately.
So the "extinction by industrial dehumanization" reads a lot like "extinction by cotton mills" - i mean, look on the bright side!
by tiborsaas on 2/1/25, 5:40 PM
It's quite creepy that in his view all tools, features, access and capabilities will be accessible to an AI agent which can just do the task. This sounds fine in a narrow scope, but if it's deployed at the scale of Windows, then it suddenly becomes a lot more scary. Don't just think of the home users, but businesses and institutions will be running these systems.
The core problem is that we can't be sure what a new generation of AI models will be capable of after a few years of iteration. They might find it trivial to control our machines which can provide them unprecedented access to follow an agenda. Malware exists today to do this, but they can be spotted, isolated and analyzed. When the OS by design is welcoming these attacks there's nothing we can do probably.
But please tell me I've consumed too many sci-fi.
by etaioinshrdlu on 2/1/25, 5:29 PM
Now we have the tools to make it happen more often.
by whodidntante on 2/1/25, 5:33 PM
Each time we choose to allow an AI to "improve" what we write/create, each time we choose to allow AI to "summarize" what we read/consume, we choose to take another step along this road. Eventually, it will be a simple "optimization" to allow AI to do this on a protocol level, making all of our lives "easier" and more "efficient"
Of course, I am not sure if anyone will actually see this comment, or if this entire thread is an AI hallucination, keeping me managed and docile.
by Jordan-117 on 2/1/25, 6:13 PM
I've heard it said that corporations are in many ways the forerunners of the singularity, able to act with superhuman effectiveness and marshall resources on a world-altering scale in pursuit of goals that don't necessarily align with societal welfare. This paper paints a disturbing picture of what it might look like if those paperclip (profit) maximizing systems become fully untethered from human concerns.
I was always a little skeptical of the SkyNet model of AI risk, thinking the danger was more in giving an AI owner class unchecked power and no need to care about the wants or needs of a disempowered labor class (see Swanwick's "Radiant Doors" for an unsettling portrayal of this). But this scenario, where capitalism takes on a mind of its own and becomes autonomous and even predatory, feels even bleaker. It reminds me of Nick Land's dark visions of the posthuman technological future (which he's weirdly eager to see, for some reason).
by upghost on 2/1/25, 5:09 PM
by tehjoker on 2/1/25, 7:42 PM
by randomcatuser on 2/1/25, 6:00 PM
for example, regarding human marginalization in states, it's just rehashing basic tropes about government (tldr, technology exacerbates the creation of surveillance states)
- "If the creation and interpretation of laws becomes far more complex, it may become much harder for humans to even interact with legislation and the legal system directly"
Well duh. That's why as soon as we notice these things, we pass laws against it. AI isn't posing the "existential risk", the way we set up our systems are. There are lots of dictators, coups, surveillance states today. And yet, there are more places in which society functions decently well.
So overall, I'm more of the opinion that people will adapt and make things better for themselves. All this anthropomorphization of "the state" and "AI" obscures the basic premise, which is we created all this stuff, and we can (and have) modified the system to suit human flourishing
by marstall on 2/1/25, 4:23 PM
by brookst on 2/1/25, 4:49 PM
It seems wild that someone could unironically talk about tools “disempowering” their users. Like, I get it, C disempowers programmers by shielding them from assembly language, and Cuisinarts disempower chefs, and airplanes disempower travelers by not making them walk through each territory.
But… isn’t that a pretty tortured interpretation of tool use? Doesn’t it lead to “the Stone Age was a mistake”, and right back to Douglas Adams’ “Many were increasingly of the opinion that they'd all made a big mistake in coming down from the trees in the first place”
I get that AI can be scary, and hey, it might kill us all and that would be bad, but this particular framing is just embarrassing.