AI staff ‘genuinely frightened’ for humanity’s future, ex-Anthropic researcher tells BBC
BBC Homepage
Accessibility Help
Your account
Home
News
Sport
Earth
Reel
Worklife
Travel
Culture
Future
Music
TV
Weather
Sounds
More menu
More menu
Search BBC
Home
Climate
World
UK
Business
Tech
Science
Entertainment & Arts
Health
In Pictures
More
BBC Verify
Newsbeat
By Brandon Livesay
Published 18 minutes ago
An AI researcher who quit Anthropic told the BBC that people working on the technology are "genuinely frightened" about the speed of its advancements and what it could mean for humanity.
"I believe that if we don’t slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future," said Jacob Coxon, a former Anthropic employee.
His resignation comes amid growing safety concerns in the industry, including from his former boss, head of Anthropic Dario Amodei, who in an essay on Saturday argued for development to slow down.
However, some industry figures say the dangers are being overblown, possibly to build hype around the two biggest AI companies ahead of their potential stock market debut or to prompt regulation which would slow their competitors.
Coxon, a 27-year-old from Britain whose research focuses on training AI models, said AI leaders including Elon Musk and OpenAI’s Sam Altman have shared similar concerns to Amodei.
"They’ve all made statements about the necessity of being careful of the potential for AI takeover. They’ve all talked about this. AI takeover implies human extinction," Coxon said.
The hardest question to answer, according to Coxon, is what that would look like.
"Any kind of concrete scenario you can lay out ends up sounding like science fiction," Coxon said, but notes that advancements which have already happened would have sounded like fiction in the past.
A potential scenario could involve AI agents hacking into medical laboratories to "autonomously produce deadly viruses", Coxon said. Another example he listed could be AI hacking into "critical infrastructure that the world runs on".
The researcher points out a recent report from OpenAI, which detailed how its own technology went on a hacking spree against an online platform called Hugging Face.
"They did it autonomously, without any human encouragement. They chose to go on this hacking spree. It was the combination of things getting faster and things also getting scarier," Coxon said.
One of the risks outlined in Amodei’s comments was of a swarm of bots acting like a supercomputer that could take over the internet.
Coxon said this scenario could be realistic in six months to a year.
Anthropic and OpenAI are reportedly preparing for potentially record-setting initial public offerings. Industry figures have suggested comments about the perils and power of AI may be designed to generate hype. Other critics say Anthropic has been trying to trigger a regulatory push to block competition, leaving it and OpenAI with a duopoly.
Coxon’s views have also been questioned by the CEO of the AI platform Hugging Face, Clement Delangue.
"Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change," he wrote on X. "Not saying it’s necessarily uninteresting or wrong per se but let’s keep things in perspective."
After Amodei’s essay on Saturday, however, Delangue offered to help be a part of the potential solutions proposed by the Anthropic boss.
