Which is Better: Conducting the Survey Online or Over the Phone?

A recent article at Religion News Service describes a new study of survey responses for the General Social Survey (GSS).  This study finds that the move from conducting the GSS via phone calls to conducting surveys online has led to a change in the set of people to agree to participate in the survey, and that this can lead to mistakes in how we interpret trends in the data.  In particular, the study finds that highly religious individuals are less likely to participate in the online surveys, and that this skews the survey results so that the surveys overstate the decline in religiosity.

There is a counterargument, however, that is also mentioned in the RNS article, namely, that the phone interviews were already overstating American religiosity so the move to online survey is producing more representative data.

This debate illustrates how, even when two people agree on what the data literally report, they may still disagree on the best way to interpret what the data actually mean.