The profitable panic behind AI doomsday warnings

The arrival of the year 2000 was accompanied by predictions that a computer bug could bring modern life to a standstill. Aircraft might fall from the sky, banks could lose their records and essential infrastructure could fail as computer clocks rolled over from 1999 to 2000.

The Y2K problem was real, and preparation helped prevent serious disruption. It was also a commercial opportunity. Companies and governments spent heavily replacing computers, updating software and hiring consultants against a threat whose full consequences nobody could reliably calculate.

Microsoft was among the main beneficiaries. In the quarter ending 31 December 1998, the company reported net income of $1.98 billion – 75% more than a year earlier – while revenue rose 38% to $4.94 billion. Microsoft’s then chief financial officer, Greg Maffei, said that sales had “spiked due in part to demand caused by year 2000 concerns” as customers replaced older computers.

Microsoft did not manufacture Y2K. It was a genuine date-processing risk that required extensive correction. The figures demonstrate a narrower point: preparing for an uncertain technological threat can be highly profitable for the companies supplying the remedy.

A quarter of a century later, artificial intelligence may be following the same pattern on a far larger scale.

Executives and researchers at the largest AI companies increasingly describe their own products in apocalyptic language. OpenAI Chief Executive Sam Altman has warned of severe risks and called for the most powerful AI systems to be licensed. Anthropic, another leading AI developer, has made protection against catastrophic AI risks central to its public identity.

Former OpenAI and Anthropic researcher Jacob Coxon recently pushed the argument further. Announcing his resignation from Anthropic, he claimed that people building advanced AI believe it “could kill us all by the end of the decade.” He accused OpenAI and Anthropic of “gambling with our lives.” His warning spread across social media and was followed by a series of interviews.

It is a compelling story: an insider leaves a valuable industry to warn humanity. There is only one major problem. Neither Coxon nor the companies promoting similar warnings have produced verifiable evidence that present AI systems are on a path to human extinction.

AI is not harmless. Existing systems can assist fraud, cyberattacks, surveillance and propaganda. Those observable risks justify safeguards.

But there is a vast difference between saying AI can cause damage and claiming it could eliminate humanity. That requires evidence that AI will surpass humans across almost every field, improve itself without effective control, gain access to critical resources, and remain impossible to stop.

No experiment has observed an AI system transform itself into an uncontrollable superintelligence. No current dataset allows researchers to calculate a reliable probability of extinction by 2030, 2035, or any other date. Widely circulated claims of a 10% risk are subjective estimates at best – not measured probabilities. Attaching a number to speculation does not turn it into science.

The International AI Safety Report, prepared with contributions from more than 100 experts, does not present extinction as an established forecast. Instead, it highlights deep disagreement among researchers and substantial gaps in the evidence surrounding a possible loss of control. Far from reflecting a scientific consensus, the doomsday scenario remains a contested hypothesis that cannot presently be verified or assigned a reliable probability.

Why, then, do the largest AI companies portray their technology as potentially uncontrollable while continuing to sell it and reportedly preparing for eventual stock-market listings? One possible answer is that doomsday warnings may serve their commercial interests.

The language of existential danger elevates AI companies above ordinary software businesses. Their executives gain political access, their research becomes matters of national security and investors are encouraged to believe that they control the most important technology ever created.

Fear also provides a powerful case for regulation that favors incumbents and prevents new players from entering the field.

In 2023, Altman told the US Senate that governments should consider licensing the most powerful AI systems and establishing an agency to oversee them. Such a regime might improve safety. It would also impose costs that only the largest companies could easily absorb.

Anthropic endorsed California’s SB 53, the bill that governs powerful AI systems, which requires large frontier developers to publish safety frameworks and report serious incidents. These reasonable provisions also help divide the market between officially recognised “frontier” companies and everybody else.

Licensing, testing and reporting require lawyers, security specialists and compliance departments. OpenAI, Anthropic, Google and Microsoft can afford them. On the other hand, smaller companies and open-source developers may not. Regulation designed around today’s leading companies could therefore strengthen them.

The message is remarkably convenient: AI is too dangerous to be left to smaller competitors or the public, but apparently not dangerous enough for its largest developers to stop releasing new models, raising capital and racing one another to build more powerful systems.

Researchers may sincerely believe catastrophe is possible. But sincerity and self-interest can coexist. Like companies that benefited from the Y2K upgrade cycle, today’s AI leaders can profit from fears they genuinely hold.

AI companies are also asking governments to accept their definition of the threat and build rules around it. The companies warning about uncontrollable technology could gain greater control over who is permitted to develop it.

In Coxon’s case, although his concerns appear sincere, his warning reinforces the same narrative promoted by the industry’s leading companies – that advanced AI poses an extraordinary danger understood principally by the people building it. Whether intended or not, that narrative strengthens the case for treating those companies as indispensable guardians of the technology, giving them competitive advantage.

Furthermore, Coxon’s concern about existential risk appears to predate his employment at the leading AI companies. He was involved in London’s rationalist community and was a fellow at Newspeak House, which describes itself as a college of political technology. Reporting has also linked him to a “long-term future” scholarship funded by Good Ventures, the foundation created by Facebook co-founder Dustin Moskovitz and Cari Tuna. Moskovitz is a prominent supporter of Democratic candidates and progressive political causes that encourage government intervention.

This does not prove bad faith. It shows that the image of a neutral engineer suddenly converted by secret evidence inside Anthropic is incomplete. Coxon already had an interest in existential risk.

Calling him a whistleblower also stretches the term. A whistleblower normally reveals concealed documents, wrongdoing or evidence. Coxon has principally revealed what he and some colleagues believe may happen. The fact that the belief is shared inside parts of the industry does not verify it.

None of this means AI should be left unregulated. Rules should address demonstrated capabilities and clearly defined risks, particularly in critical infrastructure, military systems and biological research.

What governments should not do is allow the largest AI players to define themselves as the only institutions capable of controlling a danger they have not proved exists. 

The lesson of Y2K is not that technological threats are imaginary. It is that genuine uncertainty can also create commercial opportunity. Microsoft did not create the underlying problem, but it benefited from the spending required to address it. Today’s leading AI companies could similarly gain from the fear surrounding their technology – through greater investment, political influence and regulations that protect their market position.

The question therefore remains open: are these companies protecting humanity – or building their dominance around a scenario that nobody can verify? The sky may one day fall. For now, however, the people selling the umbrellas are also writing the weather forecast.