Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond

@article{Feder2021CausalII,
  title={Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond},
  author={Amir Feder and Katherine A. Keith and Emaad A. Manzoor and Reid Pryzant and Dhanya Sridhar and Zach Wood-Doughty and Jacob Eisenstein and Justin Grimmer and Roi Reichart and Margaret E. Roberts and Brandon M Stewart and Victor Veitch and Diyi Yang},
  journal={Transactions of the Association for Computational Linguistics},
  year={2021},
  volume={10},
  pages={1138-1158}
}
Abstract A fundamental goal of scientific research is to learn about causal relationships. However, despite its critical role in the life and social sciences, causality has not had the same importance in Natural Language Processing (NLP), which has traditionally placed more emphasis on predictive tasks. This distinction is beginning to fade, with an emerging area of interdisciplinary research at the convergence of causal inference and language processing. Still, research on causality in NLP… 

Debiasing NLU Models via Causal Intervention and Counterfactual Reasoning

This paper shows that there is an unobserved confounder for the natural language utterances and their respective classes, leading to spurious correlations from training data, and provides a new perspective with causal inference to find out the bias.

Causal Inference Principles for Reasoning about Commonsense Causality

A novel framework, ROCK, is proposed, which uti-lizes temporal signals as incidental supervision, and balances confounding effects using temporal propensities that are analogous to propensity scores, and demonstrates good CCR capabilities.

CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior

This paper introduces CEBaB, a new benchmark dataset for assessing concept-based explanation methods in Natural Language Processing (NLP), which consists of short restaurant reviews with human-generated counterfactual reviews in which an aspect of the dining experience was modified.

Measuring Causal Effects of Data Statistics on Language Model's 'Factual' Predictions

The causal framework provides a language for describing how training data causes a model to make a certain prediction, through a causal framework, and demonstrates the importance of studying datasets and the benefits of causality for understanding NLP models.

Causally Estimating the Sensitivity of Neural NLP Models to Spurious Features

This work quantifies model sensitivity to spurious features with a causal estimand, dubbed CENT, which draws on the concept of average treatment effect from the causality literature, to hypothesize and validate that models that are more sensitive to a spurious feature will be less robust against perturbations with this feature during inference.

The Causal News Corpus: Annotating Causal Relations in Event Sentences from News

This work annotated 3,559 event sentences from protest event news with labels on whether it contains causal relations or not, providing a valuable resource for causal text mining researchers.

Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal View

This paper investigates the prompt-based probing from a causal view, highlights three critical biases which could induce biased results and conclusions, and proposes to conduct debiasing via causal intervention.

How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis

A causal-inspired analysis quantitatively measures and evaluates the word-level patterns that PLMs depend on to generate the missing words and concludes that the PLMs capture the factual knowledge ineffectively because of depending on the inadequate associations.

Conceptualizing Treatment Leakage in Text-based Causal Inference

The treatment-leakage problem is defined, the identification as well as the estimation challenges it raises are discussed, and the conditions under which leakage can be addressed by removing the treatment-related signal from the text in a pre-processing step the authors define as text distillation.

An Empirical Study on Explanations in Out-of-Domain Settings

Results show that in many cases out-of-domain post-hoc explanation faithfulness measured by sufficiency and comprehensiveness is higher compared to in-domain, and select-then predict models demonstrate comparable predictive performance in out- of-domain settings to full-text trained models.

References

SHOWING 1-10 OF 127 REFERENCES

Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP

This work argues that the causal direction of the data collection process bears nontrivial implications that can explain a number of published NLP findings, such as differences in semi-supervised learning and domain adaptation performance across different settings.

Causal Effects of Linguistic Properties

TextCause, an algorithm for estimating causal effects of linguistic properties, is introduced and it is shown that the proposed method outperforms related approaches when estimating the effect of Amazon review sentiment on semi-simulated sales figures.

Adapting Text Embeddings for Causal Inference

A method to estimate causal effects from observational text data, adjusting for confounding features of the text such as the subject or writing quality, and studies causally sufficient embeddings with semi-synthetic datasets and finds that they improve causal estimation over related embedding methods.

CausaLM: Causal Model Explanation Through Counterfactual Language Models

CausaLM is proposed, a framework for producing causal model explanations using counterfactual language representation models based on fine-tuning of deep contextualized embedding models with auxiliary adversarial tasks derived from the causal graph of the problem.

Challenges of Using Text Classifiers for Causal Inference

It is demonstrated how to conduct causal analyses using text classifiers on simulated and Yelp data, and the opportunities and challenges of future work that uses text data in causal inference are discussed.

Deconfounded Lexicon Induction for Interpretable Social Science

Two deep learning algorithms are introduced that are more predictive and less confound-related than those of standard feature weighting and lexicon induction techniques like regression and log odds and used to induce lexicons that are predictive of timely responses to consumer complaints, enrollment from course descriptions, and sales from product descriptions.

Causal Abstractions of Neural Networks

It is discovered that a BERT-based model with state-of-the-art performance successfully realizes parts of the natural logic model’s causal structure, whereas a simpler baseline model fails to show any such structure, demonstrating that BERT representations encode the compositional structure of MQNLI.

Annotation Artifacts in Natural Language Inference Data

It is shown that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI and 53% of MultiNLI, and that specific linguistic phenomena such as negation and vagueness are highly correlated with certain inference classes.

Explaining The Efficacy of Counterfactually-Augmented Data

A toy analog is explored, using linear Gaussian models, to explore interesting relationships between causal models, measurement noise, out-of-domain generalization, and reliance on spurious signals and yields interesting insights that help to explain the efficacy of counterfactually augmented data.

Learning the Difference that Makes a Difference with Counterfactually-Augmented Data

This paper focuses on natural language processing, introducing methods and resources for training models less sensitive to spurious patterns, and task humans with revising each document so that it accords with a counterfactual target label and retains internal coherence.
...