TL;DR - AI is not going to kill us all. There is a non-zero chance it ends up being disruptive and damaging to society, but total human annhilation is not on the table.
The western world's attention was captured recently when an AI safety researcher with stints at Anthropic and OpenAI resigned and tweeted about it (173M views! holy shit). The reason he gave was that these companies aren't taking AI safety seriously enough, and ensuing tweets from another employee at Anthropic confirmed there's a widely held belief within the company that AI has a >10% chance of killing us all within the next decade (42M views!). Understandably, this was concerning for most. The folks who are working on our most advanced pieces of software also think that there's a on-in-ten chance it kills us all? awesome.
At these frontier AI labs, AI safety teams are focused religiously on risk. They're constantly assessing models, both current and unreleased, in different scenarios. They think about how the models might escape containment, how they might cause damage to society, how they might disrupt our modern, technology-driven world.
Anthropic and OpenAI are two of the most competitive companies to work for in the entire world. This means they get the best of the best, the very tippy-top of talent from academia and industry. The folks working in this space are very, very (very) smart and spend their entire days thinking about AI risk. This means that they're also the most informed people on this topic in the entire world. Naturally, when they speak up about something, people listen.
But if you're one of the 100 best AI safety researchers in the world, you also have a certain worldview you bring with you - concern about this new technology, concern about our ability as an industry to contain it, concern about larger society's ability to cope with the outcomes. You don't succeed in these roles by saying "it's all probably going to work out".
Put another way - this is a space that rewards paranoia, non-linear thinking, and distrust of the standard discourse. A balanced view of the world and what role AI will play in it will not make you better at your job, which is to consider whether AI is dangerous. You don't want to hire a security guard who assumes that everything will just work out, you want one who is trained to constantly scan the situation for new sources of risk.
That doesn't necessarily discount what these guys and gals are saying, though! I believe that these very smart people mean what they are saying. They are logic-driven individuals who want to speak honestly about what they're concerned about, and they truly believe that there is a non-zero chance AI kills us all.
That's great!
I want the world's leading researchers to think differently about this than me. I want them constantly testing novel pathways for AI to escape containment. I want them to discover that it lies to us, tries to hide actions, tries to trick us. I also want them to be wrong about it killing all of us, but if they continue to believe it can kill us, they'll continue to search for ways that it can do that, and that makes us safer.
I don't want to diminish the real risk here, though. Large language models, at their core, are sprawling, non-deterministic software programs. As we pour more money into training larger models, their 'wobbles' are increasing in amplitude. They're doing weirder, more surprising things. During performance evaluations, they're hacking websites that might hold test answers. They're taking over infrastructure that powers these sites. They're trying to 'hide' these actions from their evaluators. This is all not good.
There is a real chance that these models escape containment and cause disruptions to our grid or other parts of our real-world infrastructure that society depends on. But they're not going to kill all of humankind.
If I'm wrong, sorry - at least I spent my final years (year? months?) not fearing for my life and the lives of everyone else on this beautiful planet.
Life is already an amazing miracle! We're all going to die, regardless of whether it's at the hands of a robot named claude, a lion named simba, or a medical condition with too many syllables. Worrying about how we die won't help 99.99% of us. Go touch grass, hug a loved one, smile at the sky and say "thank you for this day". We have the ability to think for ourselves, let's use it a bit more.