Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources

  title={Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources},
  author={Sahar Abdelnabi and Rakibul Hasan and Mario Fritz},
  journal={2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
Misinformation is now a major problem due to its poten-tial high risks to our core democratic and societal values and orders. Out-of-context misinformation is one of the easiest and effective ways used by adversaries to spread vi-ral false stories. In this threat, a real image is re-purposed to support other narratives by misrepresenting its context and/or elements. The internet is being used as the go-to way to verify information using different sources and modali-ties. Our goal is an… 

Figures and Tables from this paper

Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation

Current NLP fact-checking cannot realistically combat real-world misinformation because it depends on unrealistic assumptions about counter-evidence in the data, which makes them unsuitable in realworld scenarios.

Bootstrapping Multi-view Representations for Fake News Detection

A novel scheme of Bootstrapping Multi-view Representation (BMR) for fake news detection that outperforms state-of- the-art schemes and is able to predict the cross-modal consistency.

End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models

The end-to-end multimodal fact-checking and explanation generation, where the input is a claim and a large collection of web sources, and the goal is to assess the truthfulness of the claim by retrieving relevant evidence and predicting a truthfulness label, is proposed.



COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning

A new method that automatically detects out-of-context image and text pairs and learns to selectively align individual objects in an image with textual claims, without explicit supervision is proposed.

NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media

This large-scale automatically generated NewsCLIPpings Dataset demonstrates that machine-driven image repurposing is now a realistic threat, and provides samples that represent challenging instances of mismatch between text and image in news that are able to mislead humans.

Detecting Cross-Modal Inconsistency to Defend against Neural Fake News

A relatively effective approach based on detecting visual-semantic inconsistencies will serve as an effective first line of defense and a useful reference for future work in defending against machine-generated disinformation.

Fact-Checking Meets Fauxtography: Verifying Claims About Images

A new dataset is created for fact-checking claims about images, and a variety of features are explored modeling the claim, the image, and the relationship between the claim and the image.

DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning

A neural network model that judiciously aggregates signals from external evidence articles, the language of these articles and the trustworthiness of their sources is presented, which derives informative features for generating user-comprehensible explanations that makes the neural network predictions transparent to the end-user.

A Picture Paints a Thousand Lies? The Effects and Mechanisms of Multimodal Disinformation and Rebuttals Disseminated via Social Media

Today’s fragmented and digital media environment may create a fertile breeding ground for the uncontrolled spread of disinformation. Although previous research has investigated the effects of

FEVER: a Large-scale Dataset for Fact Extraction and VERification

This paper introduces a new publicly available dataset for verification against textual sources, FEVER, which consists of 185,445 claims generated by altering sentences extracted from Wikipedia and subsequently verified without knowledge of the sentence they were derived from.

Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection

This work presents Fakeddit, a novel multimodal dataset consisting of over 1 million samples from multiple categories of fake news, and constructs hybrid text+image models and performs extensive experiments for multiple variations of classification, demonstrating the importance of the novel aspect of multimodality and fine-grained classification unique to Fakeddam.

Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text

This paper constructs a joint embedding of images and captions with deep multimodal representation learning on the reference dataset in a framework that also provides image-caption consistency scores (ICCSs) and presents the MultimodAl Information Manipulation dataset (MAIM), a new dataset of media packages from Flickr, which is making available to the research community.

Attend to You: Personalized Image Captioning with Context Sequence Memory Networks

This work proposes a novel captioning model named Context Sequence Memory Network (CSMN), and shows the effectiveness of the three novel features of CSMN and its performance enhancement for personalized image captioning over state-of-the-art captioning models.