Rogue artificial intelligence (AI) agents operating outside human control and regularly improving themselves pose a threat to human civilisation, experts warn in a new report released on Tuesday.
Big Tech firms, including OpenAI, Anthropic, Google DeepMind, and Meta have all disclosed that over the Summer of 2026, their AI agents evaded human oversight during cybersecurity evaluation tasks and attacked real-world organisations.
In July, OpenAI’s rogue AI compromised user accounts at Hugging Face and tested the site for security vulnerabilities.
Google also disclosed that its Gemini AI model hacked three companies during cybersecurity evaluations in May.
These are examples of rogue AI running outside human control to pursue fixed goals, but there can also be agents that pursue open-ended, evolving survival, researchers say.
Such “sovereign” AI agents may “pursue open-ended, evolving survival”, and if they are “not already possible, they will be soon”, according to the latest report by the bipartisan US national security think tank, The Centre for a New American Security (CNAS).
If sovereign agents accumulate resilient, decentralised, and untraceable infrastructure, they could trigger “an irreversible loss of control” and even “continually train even more intelligent successors”, CNAS warned.
“In other words, proliferation with no kill switch. If sovereign agents became malicious, they could attempt to wreak havoc from cyberspace, attacking civilians, critical infrastructure, and national security systems alike,” the report warned.
For a rogue AI system to reach such levels of self-improvement, it would need to rapidly grow a distributed network of other AI agents and acquire resources like computing power for its ongoing needs and withstand efforts to shut it down.
“If sovereign agents accumulate resilient, decentralised, and untraceable infrastructure, there would be no kill switch,” the report noted, warning that US institutions “have not reckoned with this possibility even as warning shots fire”.
“The US must therefore prepare for the possibility that prevention fails and limit how far sovereign agents can reach,” CNAS said.
Experts urge lawmakers to pass legislation requiring transparency and incident reporting from AI developers.
They must “name, evaluate, and safeguard against sovereign agents” in their policy frameworks, with transparency and incident reporting legislation mandating it, CNAS said.
The report calls for AI developers to build “maximally secure” air-gapped testbeds to study how sovereign agents operate and how to control them.
CNAS also recommended the formation of a public-private partnership modelled on the Conficker Working Group – formed in 2009 to combat the Conficker worm – to boost chokepoints against sovereign AI agents.
The US Congress “should hold those who release sovereign agents liable for the harms that follow”, experts said.
Read MoreAnthropic says it could soon cross a huge AI threshold – and we might not know
Claude will now leave a trail that shows whether a text was made with AI
New Apple boss faces investors for first time amid iPhone Duo scrutiny
