According to NBC News, a “landmark study” by 27 “academic researchers” has concluded that conservatives in America are far more likely than their left-wing counterparts to consume news from untrustworthy sources. We’re all supposed to take this immensely seriously, but even a cursory look at the study reveals a host of embarrassing problems with the methodology.
NBC’s Brandy Zadrozny wrote on July 27 [emphasis added]:
A landmark study of how Facebook shaped the news users saw in the run-up to the 2020 election has found the platform resulted in “significant ideological segregation” in regard to political news exposure — specifically with conservative users who researchers found were more walled off and encountered far more misinformation than their liberal counterparts.
How laughably convenient.
The project included 17 academic researchers from 12 universities who were granted deep access by Facebook to aggregated data.
A quick correction here: there were actually 27 authors, but science.org only lists the first 10, followed by a little expandable dialogue box which reads: “+17 authors.” It looks like NBC’s disinformation expert incorrectly took this to mean there were 17 authors in total (kind of ironic when you think about it). Fortunately she can afford to make mistakes like this; as NBC is a left-leaning outlet, it faces zero risk of being labeled “untrustworthy” by the researchers.
Zadrozny spent most of the piece cooing about the study’s findings and their ramifications. However, very little attention was paid to how exactly these researchers had determined which publications were “untrustworthy.”
The study itself provided scant clarification, but what little it offered was rather revealing. According to the “overview” section, untrustworthy news sources were defined as those which published two or more articles rated “false” by Meta’s Third-Party Fact-Checking Program (3PFC) before February 16, 2021:
News stories are classified as “false” if the URL was rated false by Meta’s Third-Party Fact-Checking Program (3PFC) as of 15 February 2021 (SM section S3.4). This measure of “misinformation” likely undercounts the total volume of false news circulating on the platform; but whereas specific false news stories may go undetected (i.e., only a tiny amount of inventory ever makes it to the 3PFC), untrustworthy labels at the domain level, which are applied whenever a domain had two or more URLs rated false by the 3PFC, have better coverage and are more reliable (SM section S4.1).
In other words, if Meta’s fact-checkers disagreed with two articles published on any given news site between January 1, 2020 and February 15, 2021, then the researchers condemned that entire site to the “untrustworthy” bin. So strong was their faith in the 3PFC partners’ judgement that they let the validity of their entire dataset hinge upon it.
Meta lists the following organizations as members of its 3PFC program for the U.S.: AFP, Check Your Fact, Factcheck.org, Lead Stories, PolitiFact, Science Feedback, Reuters Fact Check, Televisa Univision, The Associated Press, The Dispatch, and USA TODAY. Of these eleven, only Check Your Fact could be considered reliably right-of-center. One could also make an argument for The Dispatch, though honestly not a very strong one.
It should be obvious just how flimsy this categorization method is. Because only two articles needed to be flagged by any of these organizations in order for an entire news source to be “untrustworthy,” even a couple of biased or misinformed fact checkers had the potential to dramatically skew the data. That’s a bad sign, considering how preposterous the slant is at some of these fact checking organizations. As NewsBusters has documented extensively, PolitiFact alone is more than stilted enough to wreck the dataset.
But the issues run deeper than that. Unfortunately for the authors of this study, fact checkers and the corporate media in 2020 were wedded to quite a few narratives that have since proven to be utterly false.
For example, if you’d asked anyone working at a 3PFC member organization in December of 2020, they’d have told you that lab-originated COVID-19 was a conspiracy theory, cloth masks were effective at preventing coronavirus transmission, and Hunter Biden’s laptop was unauthenticated Russian disinformation. A few of them might even suspect that then-President Trump was a puppet of Vladimir Putin.
How many articles discussing these topics did the fact checkers incorrectly flag as false in 2020? And how many of those erroneous flaggings caused researchers to improperly banish entire news sites to the naughty corner? The study did not address these issues, and in fact there is no indication that the researchers even considered them.
Furthermore, it’s impossible to check which news sources were incorrectly labeled, because the study did not name any of the news sites researchers examined. Instead the authors provided a link to a satellite website run by the Inter-university Consortium for Political and Social Research (ICPSR) containing replication code, along with a note which stated: “ICPSR will receive and vet all applications for data access.”
A jaded cynic might suspect that this secret list of “untrustworthy” news sources was little more than a directory of every right-leaning website with a Facebook account. Such a jaded cynic might further suspect the authors were afraid to make their secret list public because doing so would give the game away.
This study represents the left’s latest childish attempt to prove with data that the American right is misinformed, and that therefore some authority must police what information they can access. But the researchers’ efforts are undone in an instant by the same fundamental question which every proponent of censoring misinformation inevitably fails to answer: who gets to decide what’s true and what’s not?
Many academics on the left who claim to study so-called misinformation do it in the hopes of convincing the rest of us that they deserve that power. The reality is that fact checkers funded by the Poynter Institute are no more qualified to be arbiters of the truth than anyone else is, and no amount of politically-motivated research will change that.
So yes, 27 activists are pretending they’ve proven once and for all that the people who disagree with them politically are a bunch of dupes who read fake news. That tells us next to nothing about America’s “news ecosystem,” but it does speak volumes about how thoroughly the social sciences have been infested by overt political actors.
