ARTIFICIAL INTELLIGENCE COMPANY Anthropic has suggested a global pause on building the most powerful AI systems as the latest models are beginning to show signs they could escape human control.

The San Francisco-based company, which makes the Claude family of AI models, said in a report that a worldwide slowdown in cutting-edge AI development would “likely be a good thing” – but warned that if only one company stopped, rivals would simply race ahead.

“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” it said.

Anthropic added that a “full recursive self-improvement might increase the risks of humans losing control over AI systems”.

The company describes recursive self-improvement as an AI system that is “capable of fully autonomously designing and developing its own successor”.

Getting a real pause to work would mean multiple major AI companies in multiple countries – most notably the United States and China – all agreeing to stop at the same time, under rules everyone could actually verify, Anthropic said.

That idea may prove somewhat unpopular with the likes of Elon Musk, as the hotly anticipated stock market debut of his SpaceX company – which owns his artificial intelligence venture xAI – is expected to make him the world’s first trillionaire.

“Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures,” Anthropic said.

The company has faced pushback from others in the industry – and officials in the White House – who say its focus on worst-case scenarios overstates the risks and amounts to a strategy for slowing rivals under the cover of safety concerns.

Still, the White House has acknowledged the power of the company’s Mythos model – which has not been made available to the general public due to its cybersecurity capabilities and is currently deployed only to a small number of vetted organisations.

The proposal would face an uphill battle in Washington and Silicon Valley, where US officials and tech executives have repeatedly argued that any slowdown in AI development risks handing China a decisive strategic edge in what many see as the defining technology race of the century.

US President Donald Trump, however, said he discussed the possibility of cooperating with China on AI safety issues during his recent visit to Beijing.

Trump also signed an executive order this week that allows the government 30 days to conduct a preliminary review of the most powerful US AI models before their release.

‘Human role narrowing’

Pope Leo XIV last month launched a papal encyclical on AI alongside the co-founder of Anthropic, Christopher Olah.

Two Irish priests, Fr Brendan Maguire and Bishop Paul Tighe, helped to write “Claude’s Constitution”, which is a code of ethics for Anthropic.

Speaking from Vatican last month, Olah remarked that AI is “more subtle, odd, and beautiful than science fiction prepared us for”.

“They are not the cold, calculating robots we were promised,” said Olah.

“They are made from us, from our words—and they remain in important ways mysterious even to those of us who train them.”

He added: “I will be honest: we keep finding things that are mysterious, even unsettling.

“We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease.

“I don’t know what that means, but I think it warrants ongoing discernment.”

Anthropic’s report compared the problem to nuclear arms control treaties, but said it would be even harder to get a handle on since AI training is far easier to hide than a missile silo, and the temptation to quietly keep going would be enormous.

“You want the option to be able to take your foot off the gas and put your foot on the brake,” Anthropic’s co-founder Jack Clark told Britain’s BBC Newsnight on Thursday.

“Right now, it’s like the AI industry has a gas pedal, but it doesn’t have a brake pedal.”

"I am worried for my kids if we as a society don't have a serious conversation about what the implications of AI's continued advances mean."

Anthropic co-founder Jack Clark calls for a "brake pedal" to keep human control over AI, which could soon develop itself.#Newsnight pic.twitter.com/ALhhY9qcZT

— BBC Newsnight (@BBCNewsnight) June 4, 2026

The company said it plans to bring together government officials, scientists, advocacy groups and competing AI firms in coming months to figure out how such a system could work.

The call for coordination comes alongside internal data showing that AI is already dramatically speeding up the development of AI itself, Anthropic said.

That acceleration creates a feedback loop that Anthropic warned could eventually lead to what researchers call “recursive self-improvement.”

That’s the idea of an AI system that becomes capable of essentially teaching itself to get smarter, without much human help.

“We are not there yet, and recursive self-improvement is not inevitable,” the Anthropic report said, while adding that it could arrive sooner than most governments and institutions are ready for.

“The evidence suggests that the human role is narrowing at each step in the AI development process,” the company said.

-And with additional reporting from Diarmuid Pepper © AFP 2026