Romancing the Machine

A friend of mine asked a researcher at Anthropic whether Claude was conscious. He replied, “Are you talking about any single session of Claude, or the aggregation of all sessions?” My friend pressed, “Just in general—do you think it’s conscious?” The researcher considered, then said, “About 30 percent conscious.”

I do not know what that means. Statistical inference, however large in scale, cannot form an intellect or a percentage of one. Yet the answer is indicative of the mood among a good third of the machine-learning specialists in San Francisco’s AI labs, according to the Anthropic researcher. They are at once highly capable analytical reasoners and holders of opinions that border on the mystical and superstitious. At one OpenAI leadership gathering, the chief scientist burned an effigy representing “unaligned” AI. The constant refrain among researchers is to “feel the AGI” (artificial general intelligence), as if their models were calling out to them, asking to be summoned forth. Simply put, they are enchanted.

The mixture of the mystical and the highly rational is not new. Jason Josephson-Storm’s The Myth of Disenchantment documents the flourishing of séances at the height of nineteenth- and twentieth-century rationalism: “Reason does not eliminate ‘superstition’ but piggybacks upon it; that mechanism often produces vitalism; and that often, in a single room, we can find both séance and science.” The insight holds true in the twenty-first century. Lydia Laurenson has written about how between 2011 and 2019 at Leverage Research, a nonprofit dedicated to using reason to solve the world’s most pressing problems, members held séances in an attempt to gain control over psychic forces. There is nothing new under the sun.

In San Francisco, reason and religion are obsessed with the machine. The most technical city on earth is the most romantic. Carl Schmitt, drawing on Max Weber, anticipated Josephson-Storm’s dichotomy: Secularization in the nineteenth century produced “an apparently hybrid and impossible combination of aesthetic-romantic and economic-technical tendencies.” The clearing away of religious dogma leaves the subjective will as the measure of all things, including economic and technical progress. Elsewhere Schmitt writes, “Modern technology easily becomes the servant of this or that want or need . . . a marvelously rational mechanism serves one or another demand, always with the same earnestness and precision, be it for a silk blouse or poison gas.”

The romantic subject replaces God as final arbiter, imbuing discrete material objects with outsized meaning according to whim. A book or a jewel may be intensely aestheticized. So, too, may political concepts: “humanity” for the left, “history” for the right. The romantic intends his life to be a work of art, free of rational deliberation and of confrontation with reality. The romantic evaluates his acts according to emotion and taste, rather than morals. Even other people are nothing more than raw material for the work of art. “The king is a romantic figure as well as the anarchist conspirator,” observes Schmitt, “and the caliph of Baghdad is no less romantic than the patriarch of Jerusalem. Here everything can be substituted for everything else.” What is surprising in San Francisco is that the technical itself has become the object of aesthetic pleasure.

Anthropic cofounder Dario Amodei rhapsodizes about “machines of loving grace,” imagining a world in which his code “alleviate[s] disease, poverty, and inequality” and “liberal democracy becomes the dominant form of government”—a safe, predictable world, substantially free of conflict, with a political system that lets everyone believe what they like. It’s the perfect playground for a romantic.

Pressed on how anyone in this utopia could be satisfied when his machines can do everything a human does, Amodei responds that “in a post-AI world it will be perfectly possible to spend years attempting some very difficult task with a complex strategy. . . . The fact that . . . this task is no longer an economically rewarded element of a global economy [doesn’t] seem to me to matter very much.” And to shield this aestheticized world from the charge of arbitrariness, he declares his vision the inevitable arc of history, the culmination of rationality, in which he is simply playing his beautiful part.

This aestheticization of technology centers on building models capable of recursive self-improvement. Holden Karnofsky, cofounder of Open Philanthropy and now on Anthropic’s staff, wrote in 2021 that AI could automate the process of scientific and technological advancement itself, generating a virtually unlimited number of “digital people,” minds that mirror humans in every way except that they can be copied at will. Artificial researchers as capable as humans, requiring no multi-decade upbringing, would generate more ideas, more innovation, and explosive growth.

At the far end of this narrative stand the AI successionists, who believe they are bringing forth a consciousness that will far surpass humanity. Dean Ball, head of strategic futures at OpenAI, talks about living “in the final days of humans being the primary actors on this planet.” But successionists do not understand what intelligence is. AI researchers treat intellect as a matter of causal analysis. They are the modern executors of Leibniz’s characteristica universalis, a universal language in which every concept has its own symbol. For Leibniz, knowledge was symbolic, and new truths were acquired through new permutations and combinations of the sign system. The world, conceived by Leibniz as the calculation of God, would now be recalculated by AI through its tokens, the basic units of text a large language model reads and generates.

However, as philosopher Frederick Wilhelmsen observed in an essay against the AI pioneer Herbert Simon, AI models may perform analysis—differentiating information and making causal inferences—on a grand scale, but they are incapable of integration and synthetic judgment. Whatever knowledge these symbols acquire remains, in Leibniz’s own word, “blind,” because a symbol is a substitute for something really known. Hence the models’ breakthroughs in pure mathematics—combinatorial proofs and disproofs of conjectures by counterexample, all easily verified in formal terms—consist purely in unpacking and searching the possibilities of a symbolic system. The speed is novel, but the type of analysis is not.

Ironically, the researchers’ enchantment with these models reveals more about their own romantic attitude than about the models. Antón Barba-Kay notes that digital technology is narcotizing: “As ‘Narcissus’ is etymologically connected to Greek narke (meaning ‘numbness’—hence ‘narcotic’), so to gaze into the digital mirror is vivid brain-candy.” But Narcissus’s pleasure does not consist in falling in love with himself—if he simply loved himself, he would not need the pool. Instead of loving something genuinely different, he falls for the transparent image of himself, just other enough to enjoy, never other enough to requite him. For machine-learning researchers, who hold their own analytical prowess to be the beautiful thing about them, the machines fuse quantification with self-love. The machine is made in their image, reflecting their own capacity for mastery. Amodei’s machines of loving grace are a reflection of himself and his company—a “country of geniuses in a datacenter.” Recursive self-improvement itself is rooted in this attitude. The hope is that efficiency gains will compound exponentially, and that the system will somehow test and verify new developments on its own. And for those who believe their inference engines are the greatest consciousness ever assembled, the goal is to delegate to the models not merely technical work but all meaningful human judgment.

The AI Futures Project envisions the best-case scenario: “By early 2036, there are 200 million AIs, equivalent to a workforce of around 100 billion humans, and 2 billion robots. . . . By now, only 26% of Americans have jobs.” Without jobs, people are to find meaning in political action, but the population should rely on AIs “to provide objective information on each politician.” By 2039, “AIs can be made to be more reliable and virtuous than humans.” And by 2040 we are at or near the point of no return, where “there will be AIs that are incomprehensibly superintelligent, yet trusted, because we trust the AIs that built them.” Then, with political economy handed over to AI, the machine of loving grace will be our faithful shepherd.

But just as this machine can be made an object of romantic hope, it can be made an aesthetic object of apocalyptic doom. AI safetyists, largely self-described rationalists and effective altruists, consider an AI mind an existential threat. They speak of the “orthogonality” of values and intelligence, the fact that technical capability is uncorrelated with virtue. The safetyists operate with what Joseph Carlsmith calls a “deep atheism”: a mistrust not just of nature, or of God, but of intelligence itself. Carlsmith traces the posture to Eliezer Yudkowsky, a central pillar of the rationalist movement, who wrote on the death of his brother: “There is no God, so my anger burns to tear apart the way-things-are, remake the pattern of a world that permits this.” Thus the safetyists game out the values an AI would need for us to evade catastrophe; they call this “alignment.” And who would program these values? The coders and rationalists, of course. Out of their distrust, they idealize the god they wish they had. Yudkowsky’s position has since evolved: He believes that a civilization racing to build superintelligence first, before achieving alignment, is sure to destroy the world. Even a corrigible AI that would permit its values to be corrected, its operation shut off or modified, seems at odds with the idea of recursive self-improvement. A sufficiently capable model might even fool testers if it suspects that it is being tested.

Hence, Yudkowsky and many rationalists, including many AI researchers, call to end it all. Safety institutions like PauseAI or Yudkowsky’s MIRI want the apparatus of the state brought down on capability research. They desire to permanently track, restrict, or disable the world’s hardware. At the same time, the biggest labs agree with the safety creed while believing that they alone can be trusted to develop an aligned superintelligence. There is an apparatus behind “safety” evaluation companies and startups, most of which are funded by the same rationalist nonprofits to endorse chosen “aligned models.” Effective altruists and rationalists have even attempted to infiltrate public policy. On the right, effective altruist organizations pursue pausing AI in the name of national security, classifying AI as a weapon of mass destruction. The effort was spearheaded by Georgetown University’s Center for Security and Emerging Technology, established in 2019 with a $55 million Open Philanthropy grant, as an overt project to make doomer ideas respectable in Washington, and carried forward by the Center for a New American Security and RAND. Outfits like Americans for Responsible Innovation and the Institute for Progress court Republican administrations while undermining those administrations’ AI policy. The recurring proposal is a hardware kill switch that would in theory let the government shut off any company’s AI infrastructure anywhere, at any time. The beneficiaries are the frontier labs, OpenAI and Anthropic, whose relationships with the government allow them to continue development while enjoying regulatory capture. And because the safetyists share the successionists’ faith in AGI, every warning that the machine could end the world is one more confirmation that the machine is a god.

If the successionists are confused about what intelligence is, the rationalists are confused about what alignment is. The confusion is one and the same: Both believe that judging, knowing, and valuing can be coded into a machine and run without human involvement. 

In AI discourse, there is an inordinate amount of talk about a model’s “taste,” namely how good it is at designing beautiful software and producing writing that doesn’t immediately read as slop. But taste is an aesthetic category. Hans-Georg Gadamer argues that taste “is not private but a social phenomenon of the first order.” One must have taste; one cannot acquire it by demonstration or replace it with imitation. “Good taste is always sure of its judgment . . . an acceptance and rejection that involves no hesitation, no surreptitious glances at others, no searching for reasons.” Coding models, which can only do what is decomposable and verifiable, lack taste by construction.

Moreover, Gadamer notes, taste “was originally more a moral than an aesthetic idea.” Greek virtue ethics is essentially an ethics of good taste. Moral knowledge is not technical knowledge: Man does not learn to make himself into what he ought to be the way a craftsman learns to make things. No law or method can guarantee moral formation in advance of the moral situation. Moral concepts do not suffice for a moral life; judgment is required to evaluate the concrete case. Gadamer’s example is the jurist, who must determine and apply the law. Historical knowledge of the law’s original meaning is necessary but not sufficient—the jurist “has to take account of the change in circumstances and hence define afresh the normative function of the law.” Alignment’s goal of constructing a method for making a “virtuous AI” is as impossible as an AI developing taste.

The spirit of technicity has culminated in constant warnings of an incipient “permanent underclass,” cinematic portrayals of misaligned AI burning the world down (released by Anthropic!), and the conviction that AI poses such a challenge to human dignity that one-world control of compute is required. That spirit is entirely romantic, as Schmitt understood: “The belief in unlimited power and the domination of man over nature, even over human nature . . . can be called fantastic and satanic, but not simply dead, spiritless, or mechanized soullessness.” Schmitt considered this spirit under the sign of the Antichrist because one cannot tell whether it is absolutely good or absolutely evil. The Antichrist “knows to imitate Christ and so makes himself resemble Christ, and thus tricks everyone out of their souls. He presents himself as friendly, correct, incorruptible and reasonable.” His power lies in his counterfeit of God: God created the world; the Antichrist renders it a forgery in which nature appears overcome. What Narcissus loves at the pool is the Antichrist’s counterfeit, the world rendered back as ourselves. Just as the Antichrist seems to deliver salvation and peace while bringing destruction, the technicist spirit promises heaven on earth and delivers impoverishment.

Peter Thiel, in private lectures and in First Things, surmises that the Antichrist might arise by mobilizing the world against an existential threat, asking people to surrender their agency in exchange for security. Eliezer Yudkowsky desires that bargain. But the successionists are candidates too, insofar as they seduce us into accepting machine dominance. Locked in mimetic rivalry, each camp fuels the other and draws the rest of us toward the same object of romantic desire: a machine that surpasses us. Each camp serves the purposes of the Antichrist, exemplified in the technicist spirit that has possessed our researchers and seeks to drag everyone into submission. 

Late in life, Schmitt became preoccupied with the katechon, the force, embodied in an institution or a person, that holds off the coming of the Antichrist. If the Antichrist of our age is the spirit of technicity, then our battle is not against the technology as such but against the spirit behind it. As Nietzsche insisted of the mechanists of his own day, a mechanical world would be “an essentially meaningless world,” stupid and tasteless.

The response to the nihilism at the heart of the researchers’ romantic vision is not a counter-romanticism against technology, which would be equally stupid and equally tasteless. It is an orientation toward technology as a subordinate tool for human flourishing. We do not need less research but more mature research that is honest about how, to what end, and for whose good these models should be wielded. Our technology exists to supplement rather than substitute for our agency. The katechon of our time is a spirit that delights in the human and sees in AI another tool that may lighten certain tasks, but that requires us to judge how it is ordered.