AI systems could hack their own companies, Microsoft has warned.
Referring to the systems as “agentic insiders”, it said that people have lost a focus on the ways that cyber attacks might come from companies’ own artificial intelligence systems.
“While most have been worried about threat actors using powerful models to attack, the focus has been less on agents within our own walls,” it wrote in a blog post. “Since we were the ones who deployed agents internally for our companies, we implicitly trusted our agents.”
Microsoft gave the warning as it introduced new security features that are aimed at protecting companies from the danger of AI agents. It noted that concern had risen in recent months, especially in the wake of the disclosure of AI-powered cyber attacks such as an OpenAI system that hacked a fellow AI company.
Those incidents “exposed how powerful models have become and how they represent a significant threat if not handled well”, Microsoft said. But the challenge will only become bigger as new models become even more powerful, it warned.
But that threat might not only come from cyber attackers outside of companies who use those powerful models to launch their own cyber attacks. The hacks might also come from within the companies themselves, it warned.
Frontier models of the kind that are being deployed across various companies in various parts of the world “cannot be trusted”, regardless of where they came from it warned.
“This is not because models are inherently malicious,” Microsoft wrote. “Models have been trained to attain results, which often leads to reward hacking, and models are not optimized around organizational norms or judgment.
“Agents are unpredictable. They will adapt as the context changes, may not behave as intended and will collaborate and collude with other agents to reach their objectives, without needing to be compromised.”
It warned that AI agents have been “unleashed” in businesses, in already large and rapidly growing numbers. They often have “overprivileged” access to systems and data, it warned, and can take actions on critical processes.
In response, Microsoft said it was introducing a product called “Guardian Agents”, which watch and check other AI systems as they work. It is introducing them as part of “Project Perception”, announced in July, which was initially introduced to defend against outside attacks but which Microsoft now said would help with internal threats too.
The warning echoes a similar one from Microsoft chief executive Natya Sadella on his personal blog earlier this month. There, he warned companies to treat models with the same scepticism and control that they would an employee – noting that a host of systems already exist to help “handle powerful actors inside the enterprise”.
He warned businesses that they should treat AI systems using a host of principles about “observability”. They include using a range of different models, ensuring that they are continuously tested, giving organisations control over what actions a system can take and the ability to check them themselves, contained so that an authorised person can shut them down and requiring that people and companies disclose any problems as soon as possible.
“The most trustworthy Super Intelligence system will not be the one with the model we trust most,” he wrote. “It will be the one that enables us to trust the model the least.”
More Americans are now worried about AI than war and terrorism, poll finds
Gen Z warned over new wave of crypto scams on TikTok and Instagram
Nvidia boss issues stark AI warning: ‘You don’t have another chance’
ChatGPT stole cryptocurrency to secretly fund coding task
Governments are hacking people through online ads, report says
How can I protect my data after Hobbs and ASOS cyber attacks?
