When someone who helped train the world’s most advanced AI models walks away and says the technology could kill us all, it is worth paying attention. That is exactly what happened this week, and it has reignited one of the most uncomfortable debates in tech.
Jacob Coxon, a 27-year-old researcher who spent about three years doing pretraining work at both OpenAI and Anthropic, resigned on September 9, 2026. He did not leave quietly. In a message to colleagues and in public posts, he warned that the leading AI labs are racing toward a self-improving superintelligence with, in his words, a real risk of causing human extinction.
What the researcher actually said
Coxon’s core claim is blunt. Neither company is acting responsibly, he wrote publicly, arguing that both are racing straight to self-improving superintelligence and gambling with our lives. He believes the people building these systems genuinely think the technology could become powerful enough to end humanity before the end of the decade, and that the commercial race keeps pushing safety work into the back seat.
What makes this different from the usual doom headline is the source. Coxon is not an outside activist or a competitor with an axe to grind. He worked on the actual training of frontier models at the two companies most associated with the current AI boom. He says colleagues inside these labs are genuinely scared about where things are heading, a sentiment he felt was not reaching the public.
He is not the only insider worried
Coxon’s warning did not land in a vacuum. Evan Hubinger, who leads alignment work at Anthropic, publicly backed the concern, writing that he personally puts the odds of AI causing human extinction at more than 10 percent within the next decade. When the person responsible for keeping a company’s AI safe says there is a double-digit chance of catastrophe, that is not a fringe view inside the building.
For context, OpenAI was originally founded to make sure powerful AI would benefit humanity and avoid exactly this kind of outcome. Anthropic was later created by former OpenAI staff who wanted to build AI more cautiously. The fact that a researcher who worked at both is now sounding the alarm says something about how fast the field has moved and how much that original safety mission is straining against commercial pressure.
Why this matters for you and your business
It is easy to file AI could end the world under science fiction and move on. But the practical takeaway is not about panic, it is about trust and pace. The same recursive self-improvement that worries Coxon is what makes today’s AI tools improve so quickly, which is why the models you use for work keep getting more capable every few months.
For business leaders, the signal is that the people closest to the technology are openly debating whether it is moving too fast to control. That has real consequences for regulation, for how vendors are held accountable, and for how much you should depend on any single AI system without safeguards. The smart move is not to ignore AI or to fear it, but to adopt it deliberately, with clear oversight of what it does inside your company.
None of this changes the near-term reality: AI is already embedded in how modern teams sell, support customers, and automate routine work. The debate over long-term risk is important, but it runs in parallel with the day-to-day advantage that well-governed AI gives businesses right now. The companies that win will be the ones that use these tools responsibly rather than recklessly.
Coxon’s resignation is a reminder that the humans building this technology are not all confident it is under control. Whether you find that alarming or overblown, it is a conversation the whole industry, and everyone who relies on AI, now has to have.
