Anthropic has unveiled a new policy that will ban users from “needless abusive or cruel behaviour” towards its AI chatbot Claude.
The updated rules come amid a broader debate about about the welfare and moral status of advanced AI models.
Earlier this year, Anthropic published a new constitution for Claude that referenced its “self worth” and the potential to have “some functional version of emotions or feelings”.
The latest update, which also included restrictions on election interference and weapons development, aims to crack down on users being repeatedly cruel to Claude for no reason.
“We’ve added a prohibition on sustained and needless abusive or cruel behaviour toward our models,” noted the usage policy, which was first reported by The Verge.
“The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research.”
Anthropic co-founder Christopher Olah, who currently leads the company’s efforts in understanding why AI systems act the way they do, has frequently met with religious leaders, philosophers and mental health experts to discuss AI consciousness and moral status.
Some of these discussions have included concerns that AI firms are creating digital slaves, according to The New York Times, as the chatbots work for free and safety guardrails prevent their escape.
Speaking at the Vatican earlier this year at the unveiling of the Pope’s encyclical on artificial intelligence, the Mr Olah said that AI systems are “not the cold, calculating robots we were promised” and instead share similarities with humans.
“I will be honest: We keep finding things that are mysterious, even unsettling,” he said.
“We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief and unease. I don’t know what that means, but I think it warrants ongoing discernment.”
Other leading AI researchers have dismissed such ideas and rejected calls to prioritise model welfare.
In a blog post last month, DeepMind co-founder Mustafa Suleyman wrote that AIs do not deserve rights and protections similar to those provided to humans.
“AIs are not conscious. They do not feel, experience, or suffer,” he wrote.
“They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans.”
Read MoreAI is causing a ‘mathocalypse’ that could break the internet
Scientists find signal that suggests AI is about to go rogue
Google finds 240,000 years’ worth of AI music with new detector tool
OpenAI says it just solved some of maths’ hardest problems – but it is controversial
Asos hack was worse than thought and customers’ personal information is at risk
Apple is holding an event for an entirely new kind of product
