The following is a reply by Robert Wuthnow to the Pew Research Center's official response to his article, “In Polls We Trust,” from the August/September issue of First Things.
—Ed.
I appreciate Alan Cooperman and Greg Smith taking the time to respond to “In Polls We Trust.” The Pew Research Center has certainly been a leader over the past few decades in producing polls about religion. To its credit, it does a better job than many other polling firms in making the details of its methods available.
In Inventing American Religion: Polls, Surveys, and the Tenuous Quest for a Nation’s Faith, I deal extensively with the history of polling and the current difficulties polling firms face, including not only plummeting response rates but also the public’s declining confidence in polling.
The arguments Cooperman and Smith give about why polling about religion isn’t all that bad are quite familiar to those of us who follow polling. The issue about attendance at religious services is not that respondents over-report. The issue is that polling firms with the wherewithal to generate better estimates have made few efforts to do so—let alone to explain why the various estimates differ so much.
The issue about non-response rates is not that they always produce bad results. The point is that polling firms that claim to provide definitive information about religion should provide specific information about how trends and estimates of religious attitudes and activities in particular parts of the population are affected by varying response rates.
One of the reasons polls frequently err in estimating African American religious involvement is that “easy to reach” and “hard to reach” black respondents differ on religion questions from patterns among white respondents. The studies pollsters cite to vindicate low response rates have not to my knowledge examined such variation.
My hope is that the polling industry will take up the serious challenges it currently faces by doing all it can to produce better information before polling reaches the point where its results generate no confidence at all.
Funding from the Pew Charitable Trusts for religion polling at the Pew Research Center is among the most generous from any source. Pew could be of enormous service by doing more to address the difficult questions facing religion polling.
Here are some suggestions:
Analyze the responses to religion questions among “hard to reach” respondents compared with “easy to reach” respondents not only for the overall sample but also for categories of respondents, such as African Americans, young people, and persons claiming no religious affiliation.
Report response rates, margins of error, and indications of other possible biases for any of the important categories for which data are reported; for example, in providing comparisons among states or denominations.
Report trends adjusted for differences in response rates in different years; for example, if recent data have a significantly lower response rate, then compare the results with those that would have been obtained in an earlier poll if only that lower response rate had been achieved.
Do panel studies in which interviews are conducted with the same respondents at intervals of twelve to twenty-four months to determine whether responses to questions about religion are relatively stable or unstable.
Analyze the similarities and discrepancies between RDD telephone polls and publicly available online polls that purport to norm responses to religion questions on the basis of Pew wording and responses.
Make additional use of “big data” sources for verifying and weighting the responses of particularly hard to reach respondents and to assist in interpreting those responses.
Conduct qualitative interviews to provide clearer discussions of how respondents may have interpreted questions in polls.
Self-impose a moratorium on polls about religion for a sufficient period that the savings could be used to conduct a high quality poll with a high response rate in order to have a benchmark against which to compare other results.
Conduct wiki surveys to empirically identify topics that potentially interested audiences would consider important to study.
Release embargoed datasets sooner to social scientists interested in conducting independent analysis at the time when results are of interest to news media.
Encourage journalists to be more transparent about the possible biases in polls about religion by including information about response rates in press releases.
Conduct an audit of news coverage of its polls to determine the extent to which journalists are reporting the results accurately and with sufficient transparency.
Poll clergy, seminary faculty, and other religious leaders to determine whether or not they consider poll results useful and, if so, how.
Survey social scientists with expertise in surveys and the study of religion to learn if they do or do not have confidence in polls about religion.
Call for an independent panel to evaluate the strengths, weaknesses, and future of polling about religion similar to the work the National Research Council did for government-funded surveys.
Robert Wuthnow is the Gerhard R. Andlinger Professor of Sociology at Princeton University and the author of the forthcoming Inventing American Religion: Polls, Surveys, and the Tenuous Quest for a Nation’s Faith.
Become a fan of First Things on Facebook, subscribe to First Things via RSS, and follow First Things on Twitter.
