Our Panopticon Future

My 23-year-old daughter recently passed her driving test (first time, too!). She had the usual difficulties young people in Ireland have with getting insured. After being quoted some astronomical premiums, she chose a company that put an electronic spy in her car and now gives her credits for safe driving, which can be used to reduce her premium next year. This means the insurance company has access to a constant thread of data about her driving—obedience to speed limits, safe stops, and so forth. She gets regular text updates on how her driving is shaping up.

My daughter is happy to get relatively inexpensive insurance. Her father, who has had to sign off on riding shotgun, is moderately happy that she is being monitored and corrected while he sleeps. But I can’t help thinking that this marks another stage in our journey to a new kind of unfreedom. We have been on this road for some time, but now the landscape around us is changing, becoming more austere and forbidding.

If you had predicted two decades ago that people would allow devices into their homes that could monitor their most intimate domestic moments, you would have laughed long and loudly. If you had suggested that people might be prepared to exchange for access to chit-chat the details of their daily habits, movements, and spending and consumption patterns, you would have declared it impossible. Yet all of this territory has been surrendered with barely a shrug.

We are moving toward a new form of governance operating on the basis of a panopticon, governance that uses data and algorithms, rewards and punishments, to impose control. This new model is already being tested in a Chinese experiment dubbed “social credit,” which shifts the weight of cost and responsibility for policing onto citizens, who become in effect their own probation officers.

China’s social credit system was expected to be fully operational next year but is now unlikely to meet that deadline. The pilot experiment is focused on economic, commercial, and financial regulation, and is a nationwide scheme for tracking the trustworthiness of citizens, corporations, and government officials. It is run by city councils and tech companies, which use facial-recognition and other technologies to harvest data. When fully operational, this system will be government-controlled, centralized, and mandatory.

In the world run by social credit, an individual’s score becomes the ultimate truth of his existence, determining whether he can borrow money, travel abroad, or get his children into a good school. Citizens are constantly monitored. They lose points for infractions such as “spreading fake news,” late payment of bills, tax evasion, refusal of military service, criticizing the government, defaulting on a loan, running a red light, playing too many video games, walking a dog without a leash, wasting money on frivolous pursuits or purchases, loitering in public places, or smoking in designated no-smoking zones. Punishments include traveling and holidaying restrictions, denial of Internet services, refusal of higher education to offenders or offenders’ children, confiscation of pets or vehicles, and so forth. A bad citizen drags down the scores of family members and colleagues. Good scores are rewarded with bonuses like hotel and travel upgrades, expedited travel permissions, etc.

According to the Chinese government, the purpose of social credit is to “commend sincerity and punish insincerity.” The government also ominously pledges to “Realistically implement rewards for reporting individuals, and protect the lawful rights and interests of reporting individuals.” “Reporting individuals” should be read as “snitches.”

China already has nationwide blacklisting programs—“black lists” (the offenders) and “red lists” (the compliant) used to identify and punish those breaking commercial and industrial regulations. Some local authorities have solicited help from social media platforms to orchestrate public shaming of people on such black lists, and some social media companies are co-operating with the authorities by publishing mugshots of defaulters.

Last year, Larry Catá Backer of Penn State University published a paper, “Next Generation Law: Data Driven Governance and Accountability Based Regulatory Systems in the West, and Social Credit Regimes in China,” in which he explores the possibility of social credit systems being introduced in Western societies. The idea that government overreach is reserved to China, he writes, is “incorrect thinking. The rest of the world is steps away from trailing the Chinese into a surveillance state. . . . With incredible data collection, the plumbing is already in place for such a system to take hold.”

Whereas China tends toward a centralized social credit system, Backer believes that the West can expect its version of social credit systems to involve collaborations between private enterprises and state bodies. We already have such a transnational public/private governmental collaboration in Ireland, and so can anticipate where this might be heading. Our mediocre politicians, hand-in-glove with Google and Facebook, censor criticism of their plans to transform Irish society and punish those who dissent.

Backer foresees in social credit the possible “end of law.” Lawyers will become merely “technicians of a new system, based on algorithms and surveillance, that the lawyer no longer controls.” Alternatively, we may end up with a new form of law, focused on data.

Insurance rebate schemes, loyalty cards, and cookies have broken the ice of our potential resistance. We already have digital profiling courtesy of Facebook and Google. Social media, loyalty programs, and even the logic of some video games have already trained an entire generation “to see in such systems nothing either extraordinary or threatening.” In Sweden, Backer points out, microchipping of humans—deploying technology similar to that implanted in pets to ensure animals can be found when they wander—has been used for several years. The transformation will not be experienced as tyrannical, but as individuals contributing to the formulation of collective values.  

It is only a matter of time, Backer insists,

Before the state—together with the non-state sectors through which state power will be privatized—will begin to move aggressively not merely to ‘see’ individuals as collections of data, but to use that data to make judgements about those individuals and choices, and to seek to both discipline and control. To that end, the algorithm will become the new statute and the variable in econometrics the new basis of public opinion. We appear to be passing from the age of rights to the age of information-management, and from the age of collective responsibility and constraints to the age of collective management.

Slyly, unobtrusively, social credit will reverse the presumption of innocence, shifting the burden of proving compliance onto the citizen, who will be at constant risk of becoming “the accused.”

“Legal subjects must be made to obey,” Backer says.

. . . And that compulsion no longer comes at the point of a gun or in the uttering of individual representations of the legitimate authority of the state. Instead, it comes through the gaze; systems of constant observation combined with a self-awareness of being constantly observed that together coerces a particular set of behaviors tied to the character of the observation.

Already we think of technological eavesdropping as one of the collateral tariffs on the information society, and not a particularly bothersome one. Our liberal democracies, Backer suggests, can expect populations to be prepared for these developments by “the great culture management machinery of Western society—its television, movies and other related media—to develop a narrative in which such activity is naturalized within Western culture.” We shall become convinced that, in surrendering our privacy, we are engaging in a virtuous endeavor to catch lawbreakers and reduce wrongdoing. 

In the new, camouflaged tyrannies, we shall become as children whose whims are indulged for as long as we acquiesce in the governing ideology and the rule of its custodians. Rights and freedoms will implicitly be understood to derive from the munificence of the state rather than from any pre-existing source, or from within the human person. The Faustian pact thus signed between citizen and state, the unequal relationship between the two—and the limits this lays down—will remain obscured behind freedom rhetoric that will appear irrefutable but be profoundly conditional.

Even to call this tyranny risks ridicule—part of the genius of its conception. Unlike the classical tyrannies, its use of force will be covert and contingent, protected from detection by the distraction of its subjects.

The prophecy of this totalitarianism is to be found not, as is often suggested, in George Orwell’s 1984, but in Aldous Huxley’s 1931 novel Brave New World. Whereas Orwell anticipated a world dominated by torture and terror, Huxley foresaw humanity imprisoned by seduction, sedation, and diversion. Set in London in A.D. 2540, Brave New World anticipated subsequent developments in sleep-learning and psychological manipulation being used to impose the will of the few upon the many. Huxley's society is run by a benevolent dictatorship, its subjects kept in a state of pseudo-contentment by conditioning and a drug called Soma.

To cop the benign tyranny of our brave new panopticon world, we need but reflect on things that start off being one thing and quickly become another. Remember Google’s original motto “don’t be evil,” which has now acquired exponential layers of unintended irony? One of the symptoms of our emerging condition is that, whereas many of our freedoms are increasingly circumscribed, these constrictions are quickly redefined and understood as newer and better freedoms. The Internet began as a parallel world promising near total liberty; now it is the watchtower of an undeclared regime increasingly intoxicated by its power.

We may have already traded the belief that our rights are defined absolutely for the idea that rights are simply a reflection of the aggregate collective expression of desire. It is certainly already much, much later than we think.

 John Waters is an Irish writer and commentator, the author of ten books, and a playwright.

Like First Things on Facebook, subscribe to First Things via RSS, and follow First Things on Twitter.