AI Doom Is Radically Uncertain

For roughly a week in September, Jacob Coxon, an obscure researcher at AI giant Anthropic, was the most famous person on the planet. Shortly after resigning his position at the company, Coxon took to social media to warn the public that his former colleagues “earnestly believe” superintelligent computers would “kill us all by the end of the decade.” One of those colleagues, Evan Hubinger, came out in support of Coxon, declaring that there was more than a 10 percent chance that machines like the ones built by Anthropic will extinguish the human race in the next ten years. 

Given the stakes, 10 percent is a rather frightening number. In the interviews that followed his announcement, Coxon mainly focused on two pieces of evidence. First, the notorious hack of Hugging Face, a tech company that was breached by OpenAI “agents” after they broke out of a virtual sandbox. Second, the startling improvement in AI mathematics over the last year, culminating in the publication, by OpenAI, of a machine-generated proof demonstrating “forced blow-up” in the Navier–Stokes equations, one of six outstanding Millennium Prize problems—a set of questions that have stumped mathematicians for decades. Taken together, these developments have persuaded Coxon that Hubinger’s estimate is approximately right. 

I am not persuaded that they have weighed the evidence correctly. The real significance of the Hugging Face attack has been obscured by lurid stories about “AI civilizations” and self-sacrificing machines, which originate from a write-up by an independent security researcher on Substack. Take away these mythopoetic embellishments, and you are left with a historically important cyber breach that nonetheless ranks relatively low in terms of overall damage incurred. As for Navier–Stokes, we should recall that another Millennium problem—the Poincaré conjecture—was already solved by a flesh-and-bone mathematician in 2003 who has since gone on to live a normal life without any visible inclination to mass murder. 

Perhaps then the probability of annihilation by killer robots is less than 10 percent. But what makes us sure that propositions of the form “there is an x percent chance we will be slain by AI” are in fact meaningful? This was an obsession of the economist John Maynard Keynes, who wrote a whole treatise on probability in an effort to settle the issue. Being an economist, he was interested in more quotidian probabilities than those relating to the end of the world, such as whether the future yield on some capital investment was in any way knowable. His answer is given in its most succinct form in a famous essay for the Quarterly Journal of Economics in 1937: 

By “uncertain” knowledge, let me explain, I do not mean merely to distinguish what is known for certain from what is only probable. The game of roulette is not subject, in this sense, to uncertainty. . . . Even the weather is only moderately uncertain. The sense in which I am using the term is that in which the prospect of a European war is uncertain, or the price of copper and the rate of interest twenty years hence, or the obsolescence of a new invention, or the position of private wealth-owners in the social system in 1970. About these matters there is no scientific basis on which to form any calculable probability whatever. We simply do not know.

In the case of a roulette wheel, we know all the relevant future contingencies and can associate them with precise probabilities. But as we move up through the levels of complexity to something like the development of a new technology, it becomes impossible even to envisage the future state space. If you asked a scientist in 1945 to estimate the probability of developing a point-contact transistor, he would first have to know what a point-contact transistor is, which is impossible because it was only invented in 1947. 

Keynes’s views on uncertainty are out of fashion with economists today. As Ken Binmore noted, the axioms of subjective expected utility theory, a cornerstone of modern economics, compel them to believe that every relevant state of the world can be assigned a probability. This is why they also tend to support prediction markets. If a large number of people place bets on the same event, their beliefs will eventually converge on a stable probability distribution, which represents in some way the wisdom of their accumulated experience. I imagine Keynes would have dismissed this, in much the same way he dismissed early experiments in econometric forecasting, as so much epistemological “black magic.” 

What then should we do when faced with a future that is essentially unknowable? I’m reminded of a story told by the mathematician Persi Diaconis, a pioneer in Markov chain Monte Carlo methods, now ubiquitous in AI research. At some point in his career, Diaconis had to choose whether to leave Stanford to take up an offer from the Harvard mathematics faculty. After he bored his friends “silly” agonizing over the problem, one asked why, as a renowned probability theorist, he did not use the standard methods of his field: quantify your uncertainty, list the costs and benefits, and then choose the course of action that maximizes your utility per unit of time. Without thinking, Diaconis blurted out, “Come on, Sandy, this is serious.”

He later conceded that he probably should have listened to his friend—not because the formalisms of expected utility theory would have yielded a neat and numerically precise solution to all of his worries, but because comparing your benefits and costs in a sober fashion and then tossing a coin is still the most reliable way of getting at what you are “really after.” That is, in a sense, the ultimate decision problem, and one that is beyond the reach of all “theoretical attacks.” 

A serious discussion of the future of AI would eschew bogus probabilities, beginning instead with sober deliberation, taking into account “everything which has human experience behind it, in every branch of feeling and action,” as Keynes once put it, not just the sci-fi fantasies of a handful of billionaires and trillionaires whose knowledge of human experience is, to put it mildly, incomplete. I see little evidence of this happening and fear that the progressive mechanization of our interior lives will make it all but impossible. And it is in this sense that we are in a race against time—not, as Coxon and Hubinger would have it, to forestall the genocidal aspirations of a yet-to-be-born machine god, but to stay human enough that we can still arrive at a shared conception of the common good.