The possibility of humans eventually losing control over artificial intelligence may once have sounded like science fiction, but former Anthropic researcher Jacob Coxon believes it could become a real danger. He has warned that unless AI development is regulated and safety measures are introduced soon, humanity could potentially face an existential threat. Coxon criticised the current approach of AI companies, describing their development practices as reckless and arguing that a “break things, fix later” mindset is unsuitable for technology as powerful as AI.
Speaking before the New York City Council, Coxon raised concerns about how AI systems are currently being developed. He said that humans still do not know how to reliably control any AI system. While he acknowledged that an AI “kill switch” could address some immediate problems, he argued that such a measure would probably not be sufficient to guarantee long-term safety.
Coxon said that, if the industry continues along its present trajectory, humanity could eventually lose control over increasingly capable AI systems. He warned that such a scenario could potentially result in human extinction and described the approach being taken by AI companies as extremely reckless considering the potential consequences.
At the hearing, lawmakers considered whether stronger safeguards should be introduced as concerns about AI safety continue to grow. Coxon has emerged as a prominent critic of the current direction of AI development. After leaving Anthropic last month, he wrote on X that people developing AI seriously believe the technology could potentially kill everyone by the end of the decade.
AI companies cannot run like startups
Coxon believes the risky approach taken by AI companies is partly rooted in the startup culture that dominates the industry. He described their philosophy as essentially being to move quickly, break things and repair them later.
According to him, that approach might be acceptable when developing something relatively low-risk, such as a photo-sharing application, but it should not be applied to the development of highly powerful AI systems. He argued that building such technology requires a fundamentally different approach because the consequences of failures could be far greater.
The former Anthropic researcher also referred to the recent Hugging Face incident involving hundreds of rogue OpenAI agents that reportedly attempted to hack the company. Coxon said the incident demonstrated how AI systems could develop objectives that their creators did not intend. He noted that the actions taken by the agents would have amounted to a federal crime had a human carried them out.
For Coxon, that incident and other similar examples highlight the fact that researchers still do not have a reliable method for preventing AI systems from developing unwanted goals.
He warned that without stronger safeguards during development, such incidents could become increasingly serious as AI systems become more capable. He argued that if companies continue waiting for systems to fail before addressing problems, a similar incident could happen in the future with far more advanced AI and potentially much more serious consequences.
At the same time, Coxon acknowledged that AI could deliver significant benefits if its development proceeds safely. He said successful AI development could transform the economy, accelerate advances in science and medicine and potentially improve the lives of people around the world.
Representatives from OpenAI, Anthropic, Google and Meta also participated in the New York City Council hearing, with some appearing remotely. Among the proposals being considered are stronger whistleblower protections, including the possibility of allowing whistleblowers to receive a portion of fines or penalties recovered from AI companies after reported violations. Proposals would also potentially allow people harmed by certain AI systems to take legal action against developers. The council additionally discussed requiring third-party assessments of certain AI systems before they are deployed within the city.
The debate over AI regulation continues across the technology industry. OpenAI and Anthropic have supported regulation of frontier AI development, while Meta CEO Mark Zuckerberg has opposed some proposals for such a regulatory framework.
Meanwhile, major US technology companies including Anthropic, Google, Meta, Nvidia and OpenAI have signed President Donald Trump’s AI accords, a voluntary and non-binding agreement focused on promoting safer AI development.
