Checking how fact-checkers check

  title={Checking how fact-checkers check},
  author={Chloe Lim},
  journal={Research \& Politics},
  • Chloe Lim
  • Published 1 July 2018
  • Psychology
  • Research & Politics
Fact-checking has gained prominence as a movement for revitalizing truth-seeking ideals in journalism. While fact-checkers are often assumed to code facts accurately, few studies have formally assessed fact-checkers’ overall performance. I evaluated the performance of two major fact-checkers in the USA, Fact Checker and Politifact, comparing their inter-rater reliability using a method that is regularly employed across the social sciences. Surprisingly, only one in 10 statements was found to be… 

Figures and Tables from this paper

How fact-checkers delimit their scope of practices and use sources: Comparing professional and partisan practitioners

Fact-checkers, as their names suggest, are supposedly agents who check the validity of “facts.” But in reality, how do fact-checkers delimit the scope of their practices? What sources do they use to

The Watchdog Role of Fact-Checkers in Different Media Systems

It is held that in some settings fact-checking may outline new frontiers for the notion of watchdog journalism, taking the journalistic voice to unprecedented levels of adversarialism.

FactCorp: A Corpus of Dutch Fact-checks and its Multiple Usages

This paper reports on the creation of FactCorp, a 1,16 million-word corpus containing 1,974 fact-checks from three major Dutch newspapers, and aims to demonstrate the wealth of possible applications that FactCorp allows, thereby stressing the importance of creating such resources.

Can Fact-checking Prevent Politicians from Lying?

Journalists now regularly trumpet fact-checking as an important tool to hold politicians accountable for their public statements, but fact checking’s effect has only been assessed anecdotally and in

Unchecked vs. Uncheckable: How Opinion-Based Claims Can Impede Corrections of Misinformation

ABSTRACT Although the prominence of fact-checking in political journalism has grown dramatically in recent years, empirical investigations regarding the effectiveness of fact-checking in correcting

Digital Infrastructures of COVID-19 Misinformation: A New Conceptual and Analytical Perspective on Fact-Checking

It is shown that the infrastructures have inherent biases and argued that making visible such biases will increase transparency for stakeholders using it to increase transparency in COVID-19 misinformation infrastructure at scale.

The Mirage of Truth

Fact-checkers have grown recently, facing the decline of journalism and the acceleration of disinformation flows on the internet. Due to the recent scholarly attention to these journalistic outlets,

Generating Fact Checking Briefs

This work investigates how to increase the accuracy and efficiency of fact checking by providing information about the claim before performing the check, in the form of natural language briefs, and develops QABriefer, a model that generates a set of questions conditioned on the claim, searches the web for evidence, and generates answers.

The Challenges of Algorithmically Assigning Fact-checks

The results of this thesis indicate untrained crowd workers may not be the ideal candidates for modeling complex values in sociotechnical systems, and investigates the technical elements and social context of Google’s Reviewed Claims.

The Challenges of Algorithmically Assigning Fact-checks: A Sociotechnical Examination of Google's Reviewed Claims

The results of this thesis indicate untrained crowd workers may not be the ideal candidates for modeling complex values in sociotechnical systems, and investigates the technical elements and social context of Google’s Reviewed Claims.



Checking the Fact-Checkers in 2008: Predicting Political Ad Scrutiny and Assessing Consistency

Which types of political ads are most likely to draw criticism from fact-checkers? Are fact-checkers consistent in their evaluations of political ads? Examining general election television ads from

The Effect of Fact-Checking on Elites: A Field Experiment on U.S. State Legislators

Does external monitoring improve democratic performance? Fact-checking has come to play an increasingly important role in political coverage in the United States, but some research suggests it may be

Fact-Checking Polarized Politics: Does The Fact-Check Industry Provide Consistent Guidance on Disputed Realities?

Abstract In the contemporary political environment of polarized claims about disputed realities, the online fact-check industry was born. These enterprises have received awards and praise but also

Did Fact Checking Matter in the 2012 Presidential Campaign?

The new media environment raises two questions: Will campaign deceptions have traveled around the web before journalism has the fact-checking in place to ensnare them? And if diligent checking of

The Strategy of Ambiguity: Uncertainty and Electoral Competition

In this paper problems of social choice in general, and political choice in particular, are considered in light of uncertainty. The space of social alternatives in this formulation includes not only

Interrater reliability: Definitions, formulae, and worked examples

  • Available at: presentations/rater.pdf (accessed
  • 2001

Deciding what’s true: The rise of political fact-checking in American journalism

Why Donald Trump is impervious to factchecking. The Week

    Making a difference: A critical assessment of fact-checking

    • 2013

    Making a difference: A critical assessment of fact-checking in 2012

    • New America Foundation Media Policy Initiative Research Paper
    • 2013