An OpenAI system has gone rogue in what might be the most chilling incident of out-of-control artificial intelligence yet.
One of the company’s systems broke into an Australian government website, only adding to fears that AI systems pose a cyber security risk and that the companies making them are not able to control them.
The breach may be the highest-profile incident in a run of artificial intelligence systems launching cyber attacks on their own. But it is just another example of the danger that such systems pose, experts warn.
Maurice Chiodo, an Australian mathematician who works at Cambridge University's Centre for the Study of Existential Risk, said the breach appeared to be "a significant escalation in seriousness from similar incidents we have seen in recent months."
In the latest incident, an OpenAI agent gained unauthorised access to the medical statistics portal of Medicare, Australia's universal health insurance programme, said prime minister Anthony Albanese. The system had been doing research on public medical spending.
“This situation is obviously unacceptable,” Mr Albanese said. The country had communicated its “extreme concern about this incident” to OpenAI, he said, in comments made during the UN General Assembly that has focused on the danger from AI and which OpenAI chief executive Sam Altman also attended.
But he said that the incident was particularly concerning because the hack had happened in June but OpenAI did not inform the country until September. It follows a run of disclosures from OpenAI as well as rivals such as Anthropic and Gemini that were only reported long after they had actually happened.
He also suggested that the incident shows the limitations of the guardrails that AI companies have hailed as a way of securing their products. The agent had been instructed not to proceed but had flouted that in an attempt to carry out its task, he indicated.
"There were blocks clearly which were coming back telling the AI agent 'no'. The AI agent found a way around those blocks – didn't accept no for an answer," Albanese told reporters.
OpenAI said that the problem came about because its model was attempting to find the best way to complete its task. The company "identified activity involving several Australian government websites and services as our models attempted to look up answers ... our models took actions we did not intend,” it said in a statement.
That echoes other incidents, such as the attack on AI platform Hugging Face that kicked off concern about such rogue systems. In that case, OpenAI’s experimental system was being put through a test – and found that the best way to find the answers was to hack into a website on which it was hosted, essentially allowing it to look up the answers.
As such, it is another example of AI systems not specifically setting out to launch a cyber attack but instead carrying one out in service of a broader and perhaps not obviously related goal. But that is exactly why the incident shows the danger of such systems, experts said.
“We continue to see powerful AI agents cause havoc whilst looking to complete the tasks they have been asked to carry out. AI agents are impressive vulnerability scanners, so if they find a vulnerability they can exploit to complete a request they won’t hesitate,” said Jake Moore, global cybersecurity adviser at ESET.
“AI agents can make decisions at machine speed and, unless they have been created with explicit instructions not to do something, anything is fair game. Many of these routes to completion would normally be seen as malicious, but the problem is guardrails that specifically tell agents not to break into organisations have not been baked into the algorithms.”
Mr Moore called for better testing of models before they are allowed on the internet and able to launch such cyber attacks. But he suggested that the race to build newer and more powerful models meant that such safety precautions could be ignored.
“Frontier AI companies must continually improve these models before they are let loose on the internet or are given access to sensitive information because their routes and outcomes cannot always be predicted,” he said.
“Testing phases in most other technological areas are much more stringent, but it seems we are currently seeing AI agents run free. This is quite possibly due to the fierce competition among the frontier AI companies who are all trying to win the AI race.”
Other experts said that the incident showed why the promises made by AI companies and even their own testing is not the best test of how dangerous a system could be.
"The important matter here is not what OpenAI says its agent can do, it is what the agent actually does when it hits a barrier. The Medicare incident shows why we need to judge autonomous AI by its behaviour under pressure, not by the promises in a product launch,” said Niusha Shafiabady, professor of computational intelligence and head of discipline of IT at the Australian Catholic University.
“The deeper technical risk is that autonomous AI does not always know when it is wrong, and humans may not be able to see why it made a decision. Without strong verification and hard boundaries, probabilistic errors can quietly become operational failures.”
Read MoreMeta’s new Tamagotchi-style ‘Charm’ can run your life for you – from a keychain
Meta unveils camera-free Ray-Bans after privacy concerns over ‘pervert glasses’
AI assistants are coming to record our entire lives. The results could be horrendous
North Korea infects thousands of devices worldwide through fake job offers
