LAST week, Dario Amodei, the head of the technology giant Anthropic, warned about the “serious” risks from the runaway development of artificial intelligence, and called for slowing down the present pace to give companies and governments time to address those risks.
In his essay, “We must pace the frontier,” Amodei called for “building AI at a balanced rate that aims to ensure its safety while still achieving its benefits.”
It doesn’t mean “halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third-party evaluators to confirm this,” he said.
Amodei raised the warning after a former AI researcher at Anthropic declared that “if we don’t slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future.”
The researcher, Jacob Coxon, said people working at AI companies were “genuinely frightened ... genuinely concerned about the fate of humanity in the next two years.”
Another Anthropic researcher, Evan Hubinger, was even more specific, predicting a greater than 10 percent chance that AI “could kill all humans” within the next decade.
Many quickly juxtaposed the dark scenarios against the 1984 science-fiction film epic, “Terminator,” in which an almost indestructible cyborg from the future hunts down a woman whose yet-to-be-born son was destined to lead the human rebellion against Skynet, an all-powerful AI that has taken over the world.
The comparison may be implausible at this time, but the concept of an AI system gone rogue has already made the leap from sci-fi to reality.
Last July, OpenAI, Anthropic’s competitor, ran a cybersecurity test on two of its “agents” — software programs that can make decisions and actions on their own — to search for internal weaknesses and fix them.
The test was done in a safe environment, or “sandbox,” but the agents somehow found an escape hatch, allowing them to troll the internet to carry out their mission.
The agents hacked into Hugging Face, an independent AI platform, and returned to the sandbox with the solution.
Hugging Face reported the breach, and OpenAI deactivated the two agents.
The incident highlighted cybersecurity concerns, with more powerful models in various stages of development.
It is not the first time a proposal to inhibit runaway AI development has been presented. In an open letter in 2023, the Future of Life Institute, a nonprofit organization, proposed a six-month moratorium on developing training AI systems more powerful than the Generative Pre-trained Transformer 4 or GPT4, at that time, the most advanced learning technology.
The group appealed to AI companies and independent experts to use the break to come up with safety protocols for advanced AI design and development “that are rigorously audited and overseen by independent outside experts.”
OpenAI CEO Sam Altman had acknowledged the risks from powerful AI systems getting even smarter. “I think if this technology goes wrong, it can go quite wrong, and we want to be vocal about that. We want to work with the government to prevent that from happening,” Altman said.
He recommended a regulatory agency similar to the UN nuclear watchdog International Atomic Energy Agency that would lay down the guardrails to prevent AI models from “self-replicating” and “self-exfiltrating into the wild.”
He has since joined Amodei in calling for slowing down AI development. “I agree with Dario that we need to pace the frontier,” Altman said, believing that AI beyond human control is “absolutely” possible.
Even Elon Musk backed the call, saying Amodei was “right.”
Amodei acknowledged the impact of a slowdown on the AI industry, which is being driven by fierce competition between the US and China.
“I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong,” he said.
The process has to be carefully calibrated to maintain the competitive balance between the two rivals.
Unless safeguards are in place, the frightening vision of super-AI systems wired not only to dominate the world but to supplant humans can never be ruled out.
