Lying About Our Religion, And Other Problems With Polling

Robert Wuthnow isn’t going to say it, so I will: a lot of religion polling is bullshit.

This isn’t just an academic squabble. Polling firms like Pew and Gallup offer the principal lens through which journalists, pundits, religious leaders, and politicians observe the spiritual state of the union. Polls and surveys shape debates about policy and public morality. They help define key groups—evangelicals, for example, or the nones—that crop up again and again in the media.

They are also misleading us, Wuthnow argues in his timely, obscenity-free new book, Inventing American Religion. Wuthnow, a sociologist of religion whose work has informed a generation of scholarship, examines how polling has shaped perceptions of religion in the United States over the past century. And he argues, convincingly, that pollsters and journalists need to be more honest about what these studies can, and cannot, tell us.

It’s a shame, honestly, that Wuthnow has limited himself to religion polling, and to a fairly narrow critique of these polls’ limitations. The questions at play, here, are much larger. What kinds of conclusions can we reasonably draw about that shifty behemoth, the American Public? And what responsibility does the media have when making generalizations about identity and belief?

Polls address a problem that would have been unfamiliar to our democratic forebears. In a democracy with hundreds of millions of people, how do you know what the public thinks and wants? How do you figure out what binds them together, besides an annual obligation to the IRS and a love of fireworks? In short: how do you know what the public is? Like many hard questions, these problems have been rendered largely invisible, in no small part because “The Public” and “The American People” are favorite fictional characters for politicians and journalists, who speak of them without a trace of precision.

So let’s indulge in a quick reality check. The Super Bowl—that national spectacle that unites us around the flickering LCD hearth—had 115 million viewers in the United States last February; in other words, nearly two-thirds of us weren’t watching it. The most-viewed political spectacle of the year, the State of the Union address, draws around 10% of the population. Barack Obama won the 2012 presidential election with 62 million votes, meaning that fewer than 20% of us voted for him.

The people have spoken…kind of.

A viral post or article—the kind that fills your Facebook newsfeed and comes up in conversation at low-key parties—will be lucky to attract more than a couple million readers, out of the country’s more than 200 million literate adults.

In other words, the “public” is more scattered, niche-y, and diverse than you might think, given the way the term is used in highbrow media, or in political appeals like the State of the Union (not that you watched it). Polling is one way to get a handle on the madness. The basic principle is simple: a randomly selected sample of the population, when surveyed, provides a representative microcosm of the elusive whole. In more technical terms, pollsters direct a shrink-ray at the public, and then examine it up close.

When it comes to predicting elections, this method works extraordinarily well; elections are the raison d’être, the breadwinner, and the testing ground for pollsters. Their projections are often spot-on.

George Gallup, founder of the eponymous firm, was the Nate Silver of his day. His correct prediction of the 1936 presidential election helped give him the credibility to branch out into other areas. In 1944, Gallup polls first asked Americans whether they believed in God. Since then, major polling firms have been making regular sweeps of the nation’s spiritual psyche. Census data and other federal surveys ask few, if any, questions about religious practice, faith, or moral beliefs. That leaves a big niche to fill.

But there are some key assumptions at play here, most of which break down a bit under closer analysis. What you tell a pollster about an election (“I’m voting for Marco Rubio”) and what you tell a pollster about faith (“I believe in God”) are not interchangeable statements. Elections offer discrete, limited, and registerable choices; they are giant polls, which pollsters replicate at a smaller scale, using the magic of statistics.

When two people say they’re voting for Marco Rubio, they’re voting for the same Marco Rubio. When two people say they believe in God, there’s a lot of gray space left to fill, presumably with the kind of conversation and investigation that does not translate easily into percentages and pie charts.

A whole lot of anthropological work has mapped the ways that individuals adjust their beliefs across contexts, and through time. And some sharp sociological research has mapped the extraordinary degree to which individuals lie to pollsters about their religious practices. None of this necessarily gets picked up over the phone, by a surveyor working through a scripted questionnaire. Gauging something as amorphous and context-rich as religiosity, within the framework of something as amorphous and vast as The Public, is difficult work.

It’s even harder because The Public doesn’t like to answer its phones. Some people are more difficult to reach than others, such that these representative samples may not be so representative. Wuthnow points out that response rates for surveys have been declining steadily over the decades. Today, even a high-quality Pew study is unlikely to get even 25% of its randomly selected households to respond. Some recent Pew reports have reached just 9% of their intended sample. That 9% may still express beliefs that are representative of the public at large.

But other times, the evidence indicates, it does not.

As a result, there’s a profound disjunct between what studies actually say, and how they’re reported as facts. Just to give an example: earlier this year, citing Pew data, CNN reported that “More than one-third of millennials now say they are unaffiliated with any faith, up 10 percentage points since 2007.”

But what CNN really meant was “When asked by a complete stranger, over the phone, to define their spiritual lives in a single phrase, one-third of those adults who responded to this poll (response rate: one in ten), and who fall within an arbitrarily-defined generational frame, declined to tell the aforementioned complete stranger that they were affiliated with any specific religious group.”

These are not equivalent statements. For some reason, though, journalistic standards don’t require writers to acknowledge that distinction.

Manufacturing publics

So what do these polls and surveys do, exactly? We imagine that they measure public opinion. But we could just as easily say that polls use measures of opinion in order to manufacture publics.

Consider the rise of the category “evangelical.” In one of the book’s most interesting sections, Wuthnow talks about the emergence of evangelicals as a political body in the wake of Jimmy Carter’s nomination to the Democratic presidential ticket in 1976. After Carter described himself as an evangelical, polling firms rushed to map the extent and influence of this group. In doing so, they actually helped define it (and possibly overinflate it). Evangelical leaders trying to help build political muscle for their flocks—especially Jerry Falwell and Pat Robertson—encouraged these efforts, cited the polls, and at times even commissioned them.

Today, when media outlets talk about “evangelicals,” they almost always mean white, politically conservative, born-again Protestants, unless they specify otherwise.

The problem is that theological definitions of what makes someone an evangelical vary widely—and tweaking them, Wuthnow shows us, can generate substantial changes in poll results. Nobody actually agrees on the borders of this group. Many of those definitions also map onto a much more politically and ethnically diverse set of people than white heartland social conservatives. Black Protestants and Latino Pentecostals, in particular, tend to get ignored in this definition of evangelical, even though many of them fit the bill quite nicely in strictly theological terms.

Additionally, there are other divisions—particularly, between charismatic and non-charismatic churches—that often get ignored in the polling and reporting, for no clear reason.

Nevertheless, a particular vision of an evangelical bloc emerges in national discourse, in no small part thanks to the generalizing power of polls. Much the same seems to be happening with the nones today. At a more granular level, the nones—people who say that they don’t affiliate with any religious group—are a diverse lot, including atheists, agnostics, and people who say that they believe in God, but don’t feel comfortable in a specific institution.

The more closely we view them, the more fluid and diverse the category seems to be. Follow up with self-reported nones in a year, and many of them (as many as 30%) will say they’re now religiously affiliated. But when defined as a single entity, a new, largely fictional public emerges, ready to be taken as the harbinger of some secular age.

This process of lumping-and-fretting applies to all sorts of categories, not just religious ones. Generations offer a particularly egregious example. Who decided that all people born between 1982 and 2000 would be one single unit, the millennials? Why not 1992-2010, or 1985-2005, or some other range? The term gets thrown around so much, and by such reputable institutions, that you can forget that it was invented in 1991 by a pair of marketing consultants, whose methods were, at best, sketchy.

The persistence of these categories shouldn’t surprise anyone. Generalizations are appealing. They make good copy for journalists. “Polls that produced generalizations about the national population,” Wuthnow writes, “spoke to the nation’s historical awareness of itself as a distinct people.” In the case of religion, they made it possible to imagine some kind of national spiritual psyche—to invent American religion.

Are these generalizations harmful? It’s striking that one of the first great statistical generalizers of American religion, Will Herberg, was from a minority religion. In the 1950s, Herberg used Gallup data to paint a picture of a more unified American belief—one in which Catholics, Protestants, and Jews shared a common faith in God.

That seems like a lovely interfaith impulse. But there’s much in this process of generalization to make the contemporary observer queasy. In particular, polls and surveys tend to obscure minority opinions, through a process called norming. This happens often with race; white norming, Wuthnow writes, is “the phenomenon of drawing generalizations about the American public, meaning all of the American population, based on evidence that mostly reflects the sampled responses of the white majority.”

Christian norming happens, too: a poll that supposedly reflects the nation’s views on God is, often, mostly talking about Christian views. In doing so, it’s essentially conflating “Christian” with “American,” a strategy typically associated more with Ben Carson than with Gallup.

More generally, polls obscure a certain messiness in the American polity. Syncretism, idiosyncrasy, and inconsistency are regular features of individuals’ spiritual lives. Little of that gets captured when you scale people up into categories.

The complexities and weirdnesses of life

Should the polling firms themselves be blamed for these fictions? They’re just asking the questions, after all. Pew, in particular, is admirably open and precise about its methods and terms. And much of the problem comes when information that’s useful in a limited, qualified way gets spun into generalization and speculation in the media.

At the same time, polling firms do a lot to manufacture the appearance of certainty. In particular, they choose to report statistics as single numbers (for example, Pew recently found that 24% of Jehovah’s Witnesses describe themselves as born-again) rather than as margin-of-error ranges, normalized around a mean (had it taken that margin into account up-front, the Pew study would have reported that 16.8%-31.2% of Jehovah’s Witnesses believe they’re born again; somehow, that sounds less authoritative).

It’s also important to recognize that these surveys exist in a careful reciprocity with media organizations. Unlike academic work, polls and surveys are often commissioned by the very people who then report on them. Even if Pew or Gallup come to careful, qualified conclusions in their own reports, they have to be sensitive to the way that these results will be picked up by the very media outlets they’re intended to serve.

At times, the collusion between reporters and pollsters can verge on the sleazy. In 2013, for example, Pew published a widely reported study of American Jews. The study reported declining affiliation and rising intermarriage in the Jewish community. In the blitz of hand-wringing and fear-mongering that followed the report’s publication, the Jewish Daily Forward led the way with its extensive, frenzied coverage of the findings. (In all fairness, the Forward also published some more restrained, optimistic analysis). The Forward even used the survey’s more worrying results in a fundraising pitch. What few people mentioned was that the Forward had asked Pew to conduct the study in the first place.

In many ways, Robert Wuthnow, who brings a powerful academic reputation to this study, is the perfect person to critique this culture: he can afford to offend whomever he wants. It’s disappointing, then, that Wuthnow does not venture a more activist stance. After laying out an extended case that pollsters are marketing misleading results—a process called, in less polite company, “deception”—Wuthnow ends with a mild call for “closer and more critical scrutiny” of polls. Hasn’t he just written an entire book doing exactly that?

The final line of Inventing American Religion—“much more about religion remains to be understood”—was, to this reader, disappointingly vague.

That’s not to say that polls are useless, or that Wuthnow does not offer constructive suggestions. Polls and surveys are probably best at tracking very specific, narrow trends over time. They’re also stronger when paired with in-depth qualitative research. These kinds of studies are more expensive, and more limited in their aims. Wuthnow is right when he suggests, following a report from the American Academy of Political and Social Science, that “money for polls about religion would be better spent on a fewer high-quality polls than on more frequent low-quality polls.”

The results won’t be flashier. But the first goal of research, and the goal of journalism, should not be to produce generalizations. It should be, foremost, to chronicle the complexities and weirdnesses of life. And, only then, to start looking for patterns.

Also on The Cubit: A Conversation with Robert Wuthnow

 

  • Jim Reed

    Another point that could be considered along with some of these poll results is stability. In a politically fractured society, some initial conditions could be considered stable and some not. A prime example in America is the percent of atheists. When this is low, like one percent, we have a stable society. Nobody wants to be outed in this group because of the social penalties, so they downplay it, and the number easily continues at about one percent. When the number of atheists reaches 20%, we have an unstable situation, and things can’t remain in that state for long. Now the atheist group has safety in numbers, and is willing to speak out about that. They have lots to speak about, and the opposition might be kind of at a loss for words. People who were previously not exposed to this line of questioning are now hearing about it, and the more you try to shut questioning down, the more people will listen.

    Lots of current conservative approaches are unstable. Health care is a political issue, blue states accept central control of health care, and red states don’t. Different forms of socialized health care are working in every other advanced country in the world. The conservative approach is to believe America is better than everyone else, but once people look into it that might change, and this could be another unstable condition.

    Global warning is another conservative area of instability. They feel there is no evidence that we are damaging the climate, but as the years go by and the evidence keeps mounting, something will have to change, especially since we are now at a state where every other right wing conservative party around the world accepts the science behind global warning, and the Republican party now stands alone.

    When you look at different isses, there are many things like this. You can measure the polling numbers, but you have to also see the instability of the situation means these numbers are now temporary and will have to change. Unfortunately or fortunately, every issue now looks bad for Republicans on a wider time scale. They might be down to their last hope, end times destruction. What are the odds?

  • Walt Longmire

    Religious polls are, indeed, worthless. And it may be the for the same reason that political polls are also miserable. Usually, the “random sample” is a sample taken from no further west than New Jersey! What they reflect is a culture of liberality that is so far from normal American culture as to be a different universe.

  • Kevin Snyder

    What does that have to do with the above article?

  • Kevin Snyder

    “Additionally, there are other divisions—particularly, between charismatic and non-charismatic churches—that often get ignored in the polling and reporting, for no clear reason”.

    There is a very good reason. Pollsters don’t understand religion. No knock on pollsters; most people don’t. I dare say I know more about Hinduism than most pollsters/reporters know about Christianity, and I don’t know much about Hinduism at all. Many people who call themselves Christian have no concept of doctrine. They claim (as does the President) that when Jesus said “No man comes to the Father except through Me” He didn’t really mean it. Jehovah’s Witness doctrine claims that only 144,000 will be born again – yet 24% of the approximately 8 million JWs think they are. More than 90% of those, therefore, are unfamiliar with their own doctrine.
    As long as un-Churched people are allowed by the pollsters to define what it means to be churched, the polls will be wildly inaccurate.

  • Jim Reed

    I was just making the point that you can use a survey to measure the percentages, but it might be good to keep in mind some states are unstable. If the nation is deceiving itself, and half the people wake up and see it, things can’t stay that way because that half will tell the others who are still deceived about it, and since all of the arguments and evidence are on the side of those who are now awake, the number who remain deceived will rapidly drop. To me this is a very critical component of measuring current opinion. When everyone is deceived, they can stay that way. When enough change that they have safety in numbers and can no longer be silenced, the entire population might be in a state of rapid change.

  • Kevin Snyder

    The hundredth monkey?

  • Jim Jones

    Polls on religious questions result far too often in unbelievable results. The only reliable numbers come from seeing what people do, not what they say.

  • weylguy

    “But what CNN really meant was …”

    No matter how a question is phrased, there are those who will refute both the answer and the question as ambiguous and therefore meaningless. This is the result of religious and political ideology run amok in this country today. I really don’t know what the value of polls is anymore, since there will always be significant numbers of people who will refuse to acknowledge or even understand the results, especially when their faith is involved.

  • weylguy

    And just how exactly do you define a “culture of liberality” and “normal American culture”? It’s whatever you choose them to be. That’s ideology writ large, and it automatically discounts you as a rational person.

  • Murmur 1

    I always find discussions such as this so interesting! I take poll results seriously, but with salt, because I almost always take the time to answer survey questions. I’m a boomer and live in a western state. I am a born again Christian, a member of the Church in a broad sense. I have at times supported missions. I rarely attend services, but I often pray. I strive every day to follow Christ’s principles. Politically I am liberal. I vote Democratic most of the time. I believe in freedom of conscience and the separation of church and state. It’s interesting to think how many people like me there may be and how we show up in the results of polls.

  • Well_Read

    christians will always claim to be more religious than they are, yes I go to church, yes I believe jesus is god, yes I believe the bible is true. In private it’s the reverse. That’s why these polls are unreliable, people lie about religion. Atheists have to lie to their families about religion just to fit in, go along to get along.

    people who believe we are a christian nation won’t believe any poll that disputes that. when we get down to 50% they can’t believe that, they will say its a lie. we may be there now. atheists always claim there are more of us than get counted, it’s a hard thing to admit to yourself and to your family.

    chopra wrote a book, how to know god, that outlines the various ways ppl mold god into whatever they want him to be.

  • Walt Longmire

    The culture of liberality is a modern consequence of hyper-liberal thinking that is a reflection of East Coast mentality. It is, for lack of a better term, a “post-moral” society, such that any sense of morality has long since disappeared. That is what I mean by culture of liberality. Normal American culture retains a solid sense of morality, but anything or person on the East Coast would likely not be aware that such a culture remains.

    If a poll sample is taken, we all know that definition of terms will determine the outcome. The culture of liberality does not even know what religion is, so any sample taken from that culture will by definition be worthless, reflecting only the warped mindset of the culture of liberality.

  • nightgaunt

    A topic in need of being addressed. Glad to see it.

  • Another problem with the “nones” category is that people responding to a poll are often put in a forced choice situation, so “none” can mean “none of the above.” For example, if you are a Westar type progressive Christian, and your alternatives are “Catholic, Evangelical, Mormon, None,” what do you select? Probably “none.” This does NOT mean “no belief in God” or “no religion” or even “no affiliation.” It just means none of the other alternatives were satisfactory.

    I pointed this out in a comment section where an atheist was insisting that nones were non-religious. He objected to my point, insisting he and everybody else knew that “none” meant an unbeliever. But that is not realistic. One must put oneself in the place of a person responding to the poll. If none of the available categories captures your religious identification, but “none” is available, how will you respond? You are going to be attracted to that “none” alternative…what else can you do? Fail to respond, basically.

    Interpretations of “none” must take this into account. The only time one can assume it does NOT mean “none of the above” is if a poll-taker gives separate choices for “none of the above” and “atheist or agnostic.” I have seen very few polls that do that. Therefore the “none” category is almost impossible to interpret.

  • mikehorn

    I’m not sure who you were speaking to, but the Pew poll definitely had atheist as a subset of the Nones. While the whole None category was around 20%, the atheist portion was in single digits. Most Nones, according to the clearly presented data of Pew, are believers who chose not to claim any of the listed religions. That’s a pretty wide category, which includes atheists but also includes believers.

    FTR I’m an ex Catholic, current atheist. I pretty much agree with your comment. I get annoyed when fellow atheists get things so basically wrong…