Checking how fact-checkers check

  title={Checking how fact-checkers check},
  author={Chloe Lim},
  journal={Research \& Politics},
  • Chloe Lim
  • Published 2018
  • Computer Science
  • Research & Politics
Fact-checking has gained prominence as a movement for revitalizing truth-seeking ideals in journalism. While fact-checkers are often assumed to code facts accurately, few studies have formally assessed fact-checkers’ overall performance. I evaluated the performance of two major fact-checkers in the USA, Fact Checker and Politifact, comparing their inter-rater reliability using a method that is regularly employed across the social sciences. Surprisingly, only one in 10 statements was found to be… Expand

Figures and Tables from this paper

FactCorp: A Corpus of Dutch Fact-checks and its Multiple Usages
This paper reports on the creation of FactCorp, a 1,16 million-word corpus containing 1,974 fact-checks from three major Dutch newspapers, and aims to demonstrate the wealth of possible applications that FactCorp allows, thereby stressing the importance of creating such resources. Expand
Fact-Checking: A Meta-Analysis of What Works and for Whom
Despite its growing prominence in news coverage and public discourse, there is still considerable ambiguity regarding when and how fact-checking affects beliefs. Informed by theories of motivatedExpand
Can Fact-checking Prevent Politicians from Lying ?
Fact-checking has become popular in recent elections as a means to ensure that politicians make truthful statements. Due to a lack of empirical analyses and threats to inference, however, theExpand
Unchecked vs. Uncheckable: How Opinion-Based Claims Can Impede Corrections of Misinformation
ABSTRACT Although the prominence of fact-checking in political journalism has grown dramatically in recent years, empirical investigations regarding the effectiveness of fact-checking in correctingExpand
Generating Fact Checking Briefs
This work investigates how to increase the accuracy and efficiency of fact checking by providing information about the claim before performing the check, in the form of natural language briefs, and develops QABriefer, a model that generates a set of questions conditioned on the claim, searches the web for evidence, and generates answers. Expand
The Challenges of Algorithmically Assigning Fact-checks
In the era of misinformation and machine learning, the fact-checking community is eager to develop automated fact-checking techniques that can detect misinformation and present fact-checks alongsideExpand
Taking Fact-checks Literally But Not Seriously? The Effects of Journalistic Fact-checking on Factual Beliefs and Candidate Favorability
Are citizens willing to accept journalistic fact-checks of misleading claims from candidates they support and to update their attitudes about those candidates? Previous studies have reachedExpand
Taking Fact-Checks Literally But Not Seriously? The Effects of Journalistic Fact-Checking on Factual Beliefs and Candidate Favorability
Are citizens willing to accept journalistic fact-checks of misleading claims from candidates they support and to update their attitudes about those candidates? Previous studies have reachedExpand
An Analysis of the Fake News Assessment Criteria on Fact-check Coverage
It was found that fact-checking coverage was generally focused on presidential or general election candidates or politicians, as well as political topics, and primarily assigned the judgment ‘mostly false’. Expand
The Challenges of Algorithmically Assigning Fact-checks: A Sociotechnical Examination of Google's Reviewed Claims
The results of this thesis indicate untrained crowd workers may not be the ideal candidates for modeling complex values in sociotechnical systems, and investigates the technical elements and social context of Google’s Reviewed Claims. Expand


Checking the Fact-Checkers in 2008: Predicting Political Ad Scrutiny and Assessing Consistency
Which types of political ads are most likely to draw criticism from fact-checkers? Are fact-checkers consistent in their evaluations of political ads? Examining general election television ads fromExpand
The Effect of Fact-Checking on Elites: A Field Experiment on U.S. State Legislators
Does external monitoring improve democratic performance? Fact-checking has come to play an increasingly important role in political coverage in the United States, but some research suggests it may beExpand
Fact-Checking Polarized Politics: Does The Fact-Check Industry Provide Consistent Guidance on Disputed Realities?
Abstract In the contemporary political environment of polarized claims about disputed realities, the online fact-check industry was born. These enterprises have received awards and praise but alsoExpand
Did Fact Checking Matter in the 2012 Presidential Campaign?
The new media environment raises two questions: Will campaign deceptions have traveled around the web before journalism has the fact-checking in place to ensnare them? And if diligent checking ofExpand
The Strategy of Ambiguity: Uncertainty and Electoral Competition
In this paper problems of social choice in general, and political choice in particular, are considered in light of uncertainty. The space of social alternatives in this formulation includes not onlyExpand
Deciding what’s true: The rise of political fact-checking in American journalism
Interrater reliability: Definitions, formulae, and worked examples
  • Available at: presentations/rater.pdf (accessed
  • 2001
Trump repeats the same lines, even when they’re proven false. The Charlotte Observer
  • Available at: html (accessed
  • 2016
The effect of factchecking on elites: A field experiment on US state legislators
  • American Journal of Political Science
  • 2015
Why Donald Trump is impervious to factchecking
  • The Week (01 December). Available at: http:// (accessed
  • 2015