- Home /
- Research /
- Academic Research /
- Accessibility and Generalizability: Are Social Media Effects Moderated by Age or Digital Literacy?
Accessibility and Generalizability: Are Social Media Effects Moderated by Age or Digital Literacy?
Often, experiments on social media and internet usage leave out the older population, making the generalizability of a study inaccurate. To address this concern, we replicate two digital literacy experiments and include a subject pool with age variability.
Citation
Munger, Kevin, Ishita Gopal, Jonathan Nagler, and Joshua A. Tucker. “Accessibility and Generalizability: Are Social Media Effects Moderated by Age or Digital Literacy?” Research & Politics 8, no. 2 (2021). https://doi.org/10.1177/20531680211016968
Date Posted
Jun 09, 2021
Authors
- Kevin Munger,
- Ishita Gopal,
- Jonathan Nagler,
- Joshua A. Tucker
Area of Study
Tags
Abstract
An emerging empirical regularity suggests that older people use and respond to social media very differently than younger people. Older people are the fastest-growing population of Internet and social media users in the U.S., and this heterogeneity will soon become central to online politics. However, many important experiments in this field have been conducted on online samples that do not contain enough older people to be useful to generalize to the current population of Internet users; this issue is more pronounced for studies that are even a few years old. In this paper, we report the results of replicating two experiments involving social media (specifically, Facebook) conducted on one such sample lacking older users (Amazon’s Mechanical Turk) using a source of online subjects which does contain sufficient variation in subject age. We add a standard battery of questions designed to explicitly measure digital literacy. We find evidence of significant treatment effect heterogeneity in subject age and digital literacy in the replication of one of the two experiments. This result is an example of limitations to generalizability of research conducted on samples where selection is related to treatment effect heterogeneity; specifically, this result indicates that Mechanical Turk should not be used to recruit subjects when researchers suspect treatment effect heterogeneity in age or digital literacy, as we argue should be the case for research on digital media effects.
Background
An emerging empirical regularity suggests that older people use and respond to social media very differently than younger people. Older people are the fastest-growing population of Internet and social media users in the US, and this heterogeneity will soon become central to online politics. However, many important experiments in this field have been conducted on online samples that do not contain enough older people to be useful to generalize to the current population of Internet users; this issue is more pronounced for studies that are even a few years old.
Study
In this paper, we report the results of replicating two experiments involving social media (specifically, Facebook) conducted on one such sample lacking older users (Amazon’s Mechanical Turk) using a source of online subjects which does contain sufficient variation in subject age. We add a standard battery of questions designed to explicitly measure digital literacy.
Results
We find evidence of significant treatment effect heterogeneity in subject age and digital literacy in the replication of one of the two experiments. This result is an example of limitations to generalizability of research conducted on samples where selection is related to treatment effect heterogeneity; specifically, this result indicates that Mechanical Turk should not be used to recruit subjects when researchers suspect treatment effect heterogeneity in age or digital literacy, as we argue should be the case for research on digital media effects.