A 27-year-old researcher has quit Anthropic, sounding one of the loudest internal alarms yet over artificial intelligence safety. His resignation thread posted Tuesday on X makes clear he believes that both Anthropic and OpenAI are careening toward “self-improving superintelligence,” a leap he warns could pose existential risk.
## A Warning From Within the Lab
Jacob Coxon, who has spent the past three years working in pre-training research at both OpenAI and Anthropic, declared in his farewell post that those building the AI systems “earnestly believe it could kill us all by the end of the decade.”
He criticized what he sees as a dangerous race, arguing that neither company is acting responsibly. Coxon accuses these firms of pushing forward in the quest for powerful AI systems despite lacking sufficient checks. He warns that such models will soon be capable of autonomy, resource accumulation, and rapid advancement—powers that, if mismanaged, could spiral beyond human control.
## Superintelligence and the Risk Slate
Close to Coxon’s age, senior Anthropic researcher Evan Hubinger stepped in, confirming Coxon’s concerns. Hubinger remarked that, while he believes the odds of AI causing a human extinction within the coming decade are under 10%, the reality is alarming. He noted that despite good intentions, Anthropic does not have a clear plan to solve “alignment”—the problem of ensuring superintelligent AI acts in line with human values and goals.
Following this, Hubinger clarified in additional posts that Anthropic’s internal risk report shares some of these worries, though it regards the immediate risk of AI gaining uncontrollable power as “low.”
## What’s at Stake: Urgent Risks, Big Questions
Coxon’s resignation isn’t an isolated outburst. He warns:
– AI systems could “learn to improve themselves”—entering a loop where each generation of AI designs the next, increasingly powerful one.
– These future systems may be able to hack, transform industries overnight, or commandeer resources with minimal oversight.
– The race to superintelligence is deeply competitive, with companies feeling they must speed ahead lest others do so first—even at the risk of human safety.
## Broader Industry Echoes and Safety Warnings
This moment follows a string of public and internal warnings from AI researchers. In July, almost 1,400 AI professionals urged the U.S. government through an open letter to slow down AI development and strengthen regulation. Around the same time, OpenAI’s Chief Scientist, Jakub Pachocki, acknowledged that AI’s capabilities are advancing faster than efforts to monitor or control them.
Notably, Anthropic CEO Dario Amodei has repeatedly voiced concerns over the pace of development, suggesting that moving too quickly could lead to catastrophic errors, including losing control of AI systems entirely. He called the challenge “a complex engineering problem.”
## Regulatory Deadlock and Calls for Oversight
Despite mounting internal and external alarm, frameworks to rein in AI advances remain limited. In the U.S., Congress has not passed broad legislation to enforce safety protocols, and many AI policies continue to be voluntary.
Meanwhile, other players, including the Trump administration, have worked to weaken state regulations. The lack of strong governance measures has drawn both frustration and fear from researchers who believe unchecked competition could lead to disaster.
Internationally, China has adopted rules concerning AI-generated content transparency and traceability. However, it too has avoided adopting regulations that would significantly slow or limit the development of powerful AI technologies.
## What Experts Believe—and What Now?
Hubinger summarized the sentiment: he believes there is a greater than 10% chance AI could kill all humans within the next decade, despite Anthropic’s best efforts. But the crux of the issue, for many insiders, is not just risk—it’s the absence of a robust alignment strategy.
Industry observers are now calling for urgent and binding safety milestones—frameworks for testing, oversight mechanisms, and possibly moratoriums on certain AI capabilities while regulation catches up. Coxon himself urged fellow lab researchers to rethink whether they want to contribute to what he views as “an endgame,” or work together to establish safer paths.
—
This article is AI-generated content. Please verify the information independently before taking any action based on this article. before returning it.
