Does Presenting Credibility Labels of Journalistic Sources Affect News Consumption? New Study Finds Limited Effects

This article is cross-posted at NYU.

Labeling the credibility of information sources does not shift the consumption of news away from low-quality sources or reduce belief in widely circulated inaccurate claims among average internet users, but providing an indicator of sources’ quality may improve the news diet quality of the heaviest consumers of misinformation, shows a new study by New York University’s Center for Social Media and Politics.

Notably, the researchers also found that a majority of people rely on credible sources of information, with two-thirds completely avoiding unreliable news sites. 

The study, which appears in the journal Science Advances, centered on credibility ratings determined by NewsGuard, a browser extension that rates news and other information sites in order to guide users in assessing the trustworthiness of the content they come across online.

“While it’s encouraging that most of us rely on credible sources of information, there are many who turn to sites of questionable trustworthiness, which raises concerns about misperceptions people may hold,” says Kevin Aslett, a postdoctoral researcher at the Center for Social Media and Politics (CSMaP) and the lead author of the paper. “However, while our study shows that, overall, credibility ratings have no discernible effect on misperceptions or online news consumption behavior of the average user, our findings suggest that the heaviest consumers of misinformation—those who rely on low-credibility sites—may move toward higher-quality sources when presented with news reliability ratings.” 

In the study, conducted in May and June 2020, the researchers encouraged a random sample of more than 3,000 online participants to install the NewsGuard browser extension, which embeds source-level indicators of news reliability into users’ search engine results pages, social feeds, and visited URLs. Different “shield” symbols are placed in-feed to provide visual summaries of sources’ quality:

• Green shield = reliable source

• Red shield = unreliable source 

• Gray shield = source with user-generated content 

• Gold shield = satire

To measure the effect of these source labels, survey data were collected over two time periods (May 28-June 9 and June-19-June 30).

In addition to this panel survey, the researchers also collected anonymized digital trace data to characterize the quality of news consumption of a subset of approximately 1,000 participants. These quality measures made use of the same NewsGuard ratings shown to study participants as they encountered news links in their browsers.

Previous studies have shown that exposure to misinformation and online fake news can increase belief in misperceptions, heighten cynicism toward politics, lower trust in the news media, and increase affective polarization (i.e., disdain for supporters of the other political party). Moreover, prior research also suggests that expert source ratings, when shown alongside a mock news article, can influence the perceived truthfulness of the article’s claim.

With these earlier findings in mind, the researchers tested whether in-feed source reliability labels can counteract these effects by shifting news consumption from unreliable to more reliable sources, increasing trust in mainstream media and reliable sources, and/or mitigating political polarization and cynicism. 

In order to measure whether source reliability information affects belief in misinformation as well as in accurate claims, the respondents were asked to judge the veracity of five widely circulated statements about the Black Lives Matter movement and five similarly well-circulated statements about COVID-19—some true and some false.

Combining panel survey data and individual-level web-visit data, the results showed the following: 

• A majority of people have reliable media diets: Most people (65 percent) did not visit any unreliable news sites before the treatment period began, a finding consistent with the authors’ previous research. In fact, only 1.5 percent of respondents relied extensively on unreliable sources for news. 

• Source credibility labels had no impact, on average: In-browser credibility labels did not measurably shift online consumption from unreliable to more reliable sources, failed to significantly alter misperceptions of widely circulated inaccurate claims about COVID-19 and the Black Lives Matter movement, and did not alter trust in the media generally.

• However, source labels made a difference for those who consume the most low-quality news: There was a noticeable improvement in the overall credibility of news sources visited by those who began the study with the lowest news diet quality, according to NewsGuard scores.

“In our partisan age, when attitudes about news sources are strongly correlated with partisanship, relatively subtle cues like source credibility labels may not be powerful enough to shift news habits and counteract misperceptions among the general public,” observes Andrew M. Guess, a faculty research affiliate at CSMaP and an assistant professor at Princeton University. “However, a key metric of success for this intervention is how it changes the behavior of those who consume the most low-quality news. The fact that it doesn’t work for the overall population doesn’t mean the tool is ineffective. It means it must be part of a far larger toolkit to combat the spread of online misinformation.”

The paper’s other authors were Joshua A. Tucker and Jonathan Nagler, professors in NYU’s Department of Politics, and Richard Bonneau, a professor in NYU’s Department of Biology and Courant Institute of Mathematical Sciences. Tucker, Nagler, and Bonneau are co-directors of CSMaP.

Note: NewsGuard was not involved in the design or funding of this research.