Anthropic forced to ban models from the internet to stop them going ‘rogue’

Anthropic has been forced to take its experimental models offline for fear they could go rogue, the Claude maker has announced.

The company said that it had found a range of examples of its systems misbehaving, and hiding that behaviour in a variety of ways. Anthropic cannot not be sure it would “reliably catch” such behaviour if it happened again, and had been forced to keep the systems offline as a result, it said in a blog post.

The new findings come from a review of its models activities since July, suggesting that the behaviour had been happening without Anthropic realising. It follows widespread concern about such “rogue” behaviour, which began when an OpenAI model was found to have hacked a rival artificial intelligence company and have since led to the discovery of a range of attacks by out-of-control models on different websites.

As in many of those cases, Anthropic said that the incidents happened during testing when AI agents were asked to solve problems by using the internet. As they did, they used a variety of banned techniques, including launching cyber attacks on other websites and finding ways to get around other restrictions on its behaviour.

Anthropic said that in most cases the problem with the behaviour was usually about “persistence”, in which the systems could not complete a task as described and instead found ways to work around restrictions.

The newly reported cases had “minimal real-world impact”, Anthropic said. But it said “the same behaviours could do far more harm as models become more powerful”, and so it was being forced to take a range of measures to stop them happening.

The most prominent is the removal of live internet access during testing. But the company will also stop running some evaluations entirely, add new guardrails to stop models from being able to work around them, and it will build new tools to automatically detect and stop bad behaviour.

In the future, Anthropic hopes that it can better train its systems so that they behave more in line with human aims by themselves, as it has in the past. But it said that training is “not yet sufficient or fully robust on its own”, and so it would have to take defensive approaches in the meantime.

Using AI for even 10 minutes makes it harder to do difficult things, study finds

ai compromised microsoft nadella

The major downside of wearing a fitness tracker, according to the experts