Tech industry whistleblowers cautioned New York City officials on Monday over the risks of emerging technology, calling for stronger oversight as local governments respond to growing concerns that automated systems could endanger humanity.
At a hearing of the New York City Council, recently departed employees of Anthropic, OpenAI and Google DeepMind described a reckless corporate culture, saying tech giants are racing to build technology they cannot control.
"The companies run on a startup mindset: Move fast, break things, fix them later. That works for a photo sharing app. It does not work for building the most powerful technology ever built," said Jacob Coxon, a former engineer at Anthropic who has become a leading voice for regulation.
State and city leaders are seeking ways to regulate AI as the Trump administration maintains a passive approach and Congress remains inactive.
President Donald Trump has dismissed AI fears as a "hoax" and signed an agreement late last month allowing firms to self-police.
Coxon was a little-known engineer at Anthropic until he announced his resignation in social media posts that drew global attention. He told the Council that AI could bring tremendous benefits, but without transparency and safeguards, things are likely to spiral out of control.
"On the current path, I think it is more likely than not that humanity loses control to these AIs and it could end in human extinction," Coxon said.
A panel of corporate staffers representing OpenAI, Anthropic, Google and Meta also testified in Manhattan on Monday, telling officials their systems are already enhancing daily life and stressing they remain laser-focused on safety.
"I share your commitment to ensuring that technology is developed safely and that its benefits are widely shared," said Morgan Dwyer of OpenAI.
At one point during the hearing, Julie Menin, the speaker of the City Council, asked each company representative to "quantify the risk posed by AI in the worst case catastrophic scenario."
Dwyer answered: "I don’t know — I also don’t think it matters whether it’s 1% or 10% or 20% chance that something catastrophic will go wrong. None of these levels is remotely acceptable. We should not train models that we cannot make an extremely strong case that we can keep under human control."
Menin slammed the response as "flippant at best."
"I would say that the percentage chance doesn’t matter. What matters is the commitment to safety, and that is what OpenAI is committed to doing," Dwyer said.
The City Council is considering several AI regulations, including proposals to force companies to disclose details about AI tools, require third-party approval for local AI systems, and allow whistleblowers to collect portions of fines paid by violating firms.
Separately, New York Gov. Kathy Hochul has moved to impose reporting and transparency standards around AI, joining governors elsewhere, though their reach remains limited given that the technology easily crosses state and national borders.
Read MoreOpenAI CEO sends chilling warning: ‘World should accept some bad things happening’
ChatGPT is going to show you more ads
Meta’s Muse personal AI creates a detailed file on everyone in your life
OpenAI’s Sam Altman warns against turning AI into a god
