How Accurate Are Survey Responses on Social Media and Politics?

Background

So far, we know very little about how accurate respondents are at self-reported questions about their behavior on social media. In 2016, the Pew Research Center conducted a survey to better understand how people interact with and consume the news following the 2016 U.S. presidential election. They found that 9 in 10 U.S. adults learned about the election from a digital news source or social media. Despite questions of this nature being used by major research centers, we still know very little about how accurate people’s responses are to such questions of self-reflection, based on their observed behavior on social media. 

Study

In our own research, we set out to answer this question by creating a benchmark for our survey measures. Taking data from the U.S. 2016 election campaign, we created a machine learning classifier that connected respondents’ observed frequency of posting political content on social media, with their account data on Facebook and Twitter. 

Results

We find that, on average, our self-reported survey measures tend to correlate with observed social media activity. At the same time, we also find a worrying amount of individual-level discrepancy and problems related to extreme outliers. Our recommendations are twofold. The first is for survey questions about social media use to provide respondents with options covering a wider range of activity, especially in the long tail. The second is for survey questions to include specific content and anchors defining what it means for a post to be “about politics.”