As a Catholic statistician, I tend to read any story headlined with “Surveys say Catholics . . .” ready to flinch. Robert Wuthnow and Emma Green have both raised serious questions about how much religion polls can tell us, and how easy they are to misinterpret.
Since I work as a data journalist for FiveThirtyEight, these polling problems make me cautious about how I cover religion. Religion surveys are facing the same problem that all public opinion polls are having: response rates are lower than ever, making it possible that the final numbers are actively misleading. But then there are further problems particular to religion polling.
Almost every discussion of “Catholics” as a single, undifferentiated group, is about as useful as articles on “Millennials.” Catholics who attend Mass regularly tend to give very different answers than those who were baptized but don’t particularly practice, but they are often all rolled up into one category.
In fairness to the pollsters, it’s expensive to poll enough people to be able to split Catholics into subcategories, so people usually don’t do it unless it’s an in-depth religion survey, but using simple “Catholic,” “Evangelical Protestant,” “Non-Evangelical Protestant” buckets in analysis gives just enough information to mislead.
Even when a pollster does a deep dive into a particular faith—I’ve found some pretty interesting statistics to dig into—there’s almost always something I’m curious about that I have to leave out, because the question is too complicated to handle in a straightforward survey.
When I covered Pew’s surveys on division within the Catholic Church, I wound up focusing on whether dissenters believed that the church would change its stance. I had wanted to make use of some of Pew’s numbers on whether American Catholics thought contraception, cohabitation, remarriage without annulments, and other hot button issues were sins, but, ultimately, I was worried that the question was too hard to address in a multiple choice survey.
On nearly every issue, a majority of Catholics doubted that these forbidden actions were really sins; 66 percent thought that contraception wasn’t a sin, 54 percent dissented on cohabitation, and 49 percent on remarriage without annulments. The trouble was, I wasn’t quite sure if that was because the Catholics being surveyed thought that these particular issues weren’t sinful, or whether they were reluctant to label anything as a sin.
I agreed with Francis Spufford, the author of Unapologetic when he wrote:
Everybody knows, then, that “sin” basically means “indulgence” or “enjoyable naughtiness.” […] The result is that when you come across someone trying to use “sin” in its old sense, you may know perfectly well in theory that they must mean something which isn't principally chocolatey, and yet the modern mood music of the word is so insistent that it's hard to hear anything except an invocation of a trivially naughty pleasure.
Pew, which deserves its reputation for solid statistics, had actually anticipated my concern. In addition to asking about specific sins, they also double-checked whether respondents were willing to agree that “sin” was a meaningful category. Nine in ten Catholics offered a yes in response to “Do you believe in the concept of sin, that some actions are offensive to God?”
But I still wasn’t sure enough that they’d wound up measuring what they meant to measure. What I really wanted was for Pew to throw some gimmes into their lineup of possible sins: murder, theft, lying. Were the survey-takers only willing to endorse sin in the abstract, or would a decent share of that 90 percent who believed in sin say that, yes, lying really was a sin?
Without questions like that to use as calibration, I just didn’t feel confident enough in the results to weave them into my reporting. Safer to stick with the empirical questions—what do you think the church will do?—than to try to guess what, exactly, the respondents meant by sin, and whether their answers would have been different if they’d had CCC 1849 in front of them, with all its vividness, instead of simply having sin defined as “offensive to God.”
In a way, these religion surveys remind me of the perennial studies of how few Americans believe that the earth goes round the sun, that evolution shaped human development, that vaccines don’t cause autism, etc. Each of those questions touches on something objective, just like questions about sin do—they’re a measure of how widely a fact is known, not a referendum to decide what we’re going to believe going forward.
And just as those studies make nerds like me all the more eager to increase scientific literacy, polls of Catholics or Christians at large are a guide for future catechesis. It’s harder for the scientists than it is for religious leaders; once people graduate from school, they only have the opportunities to learn that they create for themselves. But, when it comes to religious literacy, school is open every Sunday.
It’s for preachers, both ordained ministers and parents, friends, and anyone else engaged in the faith to try to cover the gaps. Emma Green is correct that religion polls are mostly a problem if they’re the primary way that the faith is understood and discussed:
Polling has become the only polite language for talking about religious experience in public life. Facts like church attendance are much easier to trade than messy views about what happens to babies when they die, or the nature of sin, or whether people have literal soul mates. There’s an implicit gap between people’s private self-understanding of their own moral nature and the way those complex identities are reduced in the media and public-research reports.
Data journalists like me can be careful about not asking polls to tell us more than they’re built for, media watchdogs like GetReligion can annotate the stories that fall short, but it’s up to everyone in the church to give us more to talk about, to make sure that “sin” isn’t something you hear defined only by a pollster over the phone, and to take an interest in the personal opinions of the people around us, not just as one data point in public opinion.
Leah Libresco is a blogger for Patheos and works as a statistician in Washington, DC. Her first, recently published book is called Arriving at Amen: Seven Catholic Prayers That Even I Can Offer.
Become a fan of First Things on Facebook, subscribe to First Things via RSS, and follow First Things on Twitter.
