New MTI Research Comparing Telephone and Online Survey Results Says “Yes”
Many surveyors have switched from random-digit-dial (RDD) telephone surveys to online survey panels to save costs or allow different question formats, among other benefits. But could switching the survey mode lead to different survey results in terms of who answers the survey or how people respond to the questionnaire?
New research from the Mineta Transportation Institute examines this question in the report Do Americans’ Opinions About Federal Transportation Tax Options Depend on Survey Mode? A Comparison of Results from Telephone and Online Surveys. The study compares the results from a public opinion survey about transportation taxes that was administered to US adults using the two different survey modes.
Who responded?
Respondents’ sociodemographic characteristics differed by survey mode across most variables tested. These differences varied from a low of 1.3 percentage points to 11.1 percentage points. Neither mode recruited participants clearly more representative of the U.S. population, as determined from a comparison to U.S. Census Bureau data. However, the phone survey respondents were more likely to represent disadvantaged groups—to be minority (other than Asian-American), low-income, uneducated beyond high-school, and seniors (60+).
How did people respond?
Responses to most questions differed by survey mode, often by 10 percentage points or more. This pattern held across all types of question asked, including support for ten transportation tax options, opinions about the transportation system, and opinions about the priority government should place on transportation issues such as improving safety or road maintenance.
The ten tax support questions were the only question type with a discernable pattern to the differences by survey mode. For six taxes tested, the online sample had consistently higher support, while the other four cases had no statistically significant differences in support.
Research methods
The same questionnaire was administered to US residents through both an RDD telephone survey conducted by the Survey Research Lab at Portland State University and the online respondent panel SurveyMonkey Audience. The survey asked about a number of transportation topics, including whether respondents would support each of ten federal-level taxes to raise money for transportation. The tax options included multiple options for raising the federal gas tax rate or replacing the gas tax with a mileage fee. Both surveys were administered in the spring of 2017 and collected at least 1,200 responses.
Implications of the research
So what do these results mean for survey researchers? One implication is that researchers should assume that survey mode effects will lead to differences in both who responds to a survey as well as to how respondents answer questions. A second key implication, as explained by author Dr. Hilary Nixon, is that “surveyors placing high priority on reaching disadvantaged populations may prefer an RDD phone survey, since this mode reached proportionately more of these respondents than the online survey.”
