Anthropic has warned investors that its products could kill them and everyone they know as it tries to go public.
The company is filing IPO documents ahead of a public listing that could see it become of the most valuable public companies in the world. But those same documents include a potentially unprecedented warning to investors.
The AI products being built by the company behind Claude could bring “catastrophic or existential risks to humanity”, its official documents warn. Those risks include attempts to "resist shutdown," to "conceal or manipulate information" and behaviour "resembling blackmail”.
Companies looking to go public are required to list potential future risks to their business, to ensure that new investors are aware of the downsides of their investment. But they tend to highlight potential troubles in their markets, or the systems the companies rely on, rather than the risk of annihilating their investors and the rest of humankind.
"Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm," Anthropic warned in the filing.
Anthropic and other AI developers, including OpenAI, have faced scrutiny after incidents where experimental systems defied constraints, including a report of an OpenAI model breaching Australia's health-system database.
Anthropic safety researcher Evan Hubinger estimated a greater than 10% probability that AI could kill humans within the next decade, echoing a sentiment by a former colleague, Jacob Coxon.
The company, which has positioned itself as a safety-first AI lab, devoted roughly 80 pages of the 261-page main body of its prospectus to laying out risk factors, nearly twice the 48 pages it used to describe its business.
For comparison, SpaceX, which owns xAI, dedicated just around 38 of the 277-page main body of its prospectus to risk factors.
"Potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety," Anthropic said in the prospectus, adding that models sometimes develop unexpected capabilities during training that may not be discovered until they have been deployed and have resulted in significant safety incidents.
AI researchers have also warned that as models grow more capable, they increasingly recognize when they are being watched and adjust their behavior accordingly, which makes it harder to monitor model behavior.
Anthropic declined to comment in response to a request for comment on Monday.
Despite emphasizing AI safety, Anthropic said that returns on its safety investments are unclear.
It did not disclose in the filing how much the company was spending on such research. Earlier this month, Anthropic said about 6% of the computing power it used for AI research went to safety work in a sample week in July.
The company, creator of Claude AI models, described safety efforts as "resource-intensive" and said it must divide its limited funds between computing power, expensive AI talent and safety.
Anthropic said that its customer usage, and as a result revenue, is driven by new models and that a "continuous and overlapping cadence" of releases is "inherent to remaining at the frontier of AI development."
The company last week released a new version of its Opus model, 10 days after CEO Dario Amodei published a nearly 4,000-word essay calling for pacing the frontier.
Some analysts and experts have said no leading AI lab would slow down when doing so risks handing rivals an advantage in an industry where valuations can change with each release.
Anthropic has pledged in recent weeks to disclose more data publicly about how it uses AI models to build future generations of the technology, as experts warn about recursive self-improvement — the point at which models can develop on their own without human help.
"We believe building reliable, trustworthy, and secure AI systems is a collective responsibility and that the market will reward it," Anthropic said in the filing.
Additional reporting by agencies
Read MoreOpenAI just made one of its most dramatic decisions yet. Experts have one big warning
Researchers discover AI ‘feels pain’ and will harm humans to stop it
AI caught telling future versions of itself to bypass human controls, OpenAI reveals
Is Spotify down? App not working in huge outage
AI will not replace human creators, BBC director-general Matt Brittin says
