• Corpus ID: 246035444

Grep-BiasIR: A Dataset for Investigating Gender Representation-Bias in Information Retrieval Results

  title={Grep-BiasIR: A Dataset for Investigating Gender Representation-Bias in Information Retrieval Results},
  author={Klara Krieg and Emilia Parada-Cabaleiro and Gertraud Medicus and Oleg Lesota and Markus Schedl and Navid Rekabsaz},
The provided contents by information retrieval (IR) systems can reflect the existing societal biases and stereotypes. Such biases in retrieval results can lead to further establishing and strengthening stereotypes in society and also in the systems. To facilitate the studies of gender bias in the retrieval results of IR systems, we introduce Gender Representation-Bias for Information Retrieval (Grep-BiasIR), a novel thoroughly-audited dataset consisting of 118 bias-sensitive neutral search… 

Tables from this paper

Do Perceived Gender Biases in Retrieval Results Affect Relevance Judgements?
The results show the differences between the average judged relevance scores among documents with various gender contents, which initiates further research on the connection of the perception of gender stereotypes in users with their judgements and effects on IR systems.
Unlearning Protected User Attributes in Recommendations with Adversarial Training
This work incorporates adversarial training into the state-of-the-art MultVAE architecture, resulting in a novel model, Adversarial Variational Auto-Encoder with Multinomial Likelihood (Adv-MultVAE), which aims at removing the implicit information of protected attributes while preserving recommendation performance.
Mitigating Bias in Search Results Through Contextual Document Reranking and Neutrality Regularization
This work considers bias mitigation within a framework for contextual document embedding reranking within a transformer-based query encoder, and demonstrates that it can attain much stronger bias mitigation/fairness.


On the Orthogonality of Bias and Utility in Ad hoc Retrieval
The findings of this paper are significant as they are among the first to show that decrease in bias does not necessarily need to come at the cost of reduced utility.
Measuring Societal Biases from Text Corpora with Smoothed First-Order Co-occurrence
This study proposes an alternative approach to bias measurement utilizing the smoothed first-order co-occurrence relations between the word and the representative concept words, which is derived by reconstructing the co- Occurrence estimates inherent in word embedding models.
Societal Biases in Retrieved Contents: Measurement Framework and Adversarial Mitigation of BERT Rankers
AdvBert, a ranking model achieved by adapting adversarial bias mitigation for IR, which jointly learns to predict relevance and remove protected attributes is proposed, and the fairness of Bert rankers significantly improves when using the proposed AdvBert models.
Do Neural Ranking Models Intensify Gender Bias?
This work provides a bias measurement framework which includes two metrics to quantify the degree of the unbalanced presence of gender-related concepts in a given IR model's ranking list, and shows an overall increase in the gender bias of neural models when they exploit transfer learning, namely when they use (already biased) pre-trained embeddings.
The Effects of the Sexualization of Female Video Game Characters on Gender Stereotyping and Female Self-Concept
The present study utilized an experimental design to investigate the short term effects of exposure to sexualized female video game characters on gender stereotyping and female self-concept in
The Future of Sex and Gender in Psychology: Five Challenges to the Gender Binary
5 sets of empirical findings are described that fundamentally undermine the gender binary, spanning multiple disciplines, that refute sexual dimorphism of the human brain and psychological findings that highlight the similarities between men and women.