Large image datasets: A pyrrhic win for computer vision?

@article{Prabhu2021LargeID,
  title={Large image datasets: A pyrrhic win for computer vision?},
  author={Vinay Uday Prabhu and Abeba Birhane},
  journal={2021 IEEE Winter Conference on Applications of Computer Vision (WACV)},
  year={2021},
  pages={1536-1546}
}
  • Vinay Uday Prabhu, Abeba Birhane
  • Published 24 June 2020
  • Computer Science, Mathematics
  • 2021 IEEE Winter Conference on Applications of Computer Vision (WACV)
In this paper we investigate problematic practices and consequences of large scale vision datasets (LSVDs). We examine broad issues such as the question of consent and justice as well as specific concerns such as the inclusion of verifiably pornographic images in datasets. Taking the ImageNet-ILSVRC-2012 dataset as an example, we perform a cross-sectional model-based quantitative census covering factors such as age, gender, NSFW content scoring, class- wise accuracy, human-cardinality-analysis… 
ViBE: A Tool for Measuring and Mitigating Bias in Image Datasets
TLDR
The ViBE Tool allows for the machine learning bias problem to be addressed early in the pipeline at the dataset stage, allowing both dataset creators and users to gain a better understanding of what exactly is portrayed in their dataset.
A Study of Face Obfuscation in ImageNet
TLDR
This paper investigates how face blurring—a typical obfuscation technique—impacts classification accuracy, and investigates how features learned on face-blurred images are equally transferable to 4 downstream tasks.
To "See" is to Stereotype
TLDR
A controlled experiment is designed, to examine the interdependence between algorithmic recognition of context and the depicted person's gender, and to create a highly controlled dataset of people images, imposed on gender-stereotyped backgrounds.
REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets
TLDR
REVISE (REvealing VIsual biaSEs) is a tool that assists in the investigation of a visual dataset, surfacing potential biases currently along three dimensions: object- based, gender-based, and geography-based.
Understanding bias in facial recognition technologies
TLDR
This explainer examines how historical patterns of discrimination have made inroads into the design and implementation of FDRTs from their very earliest moments and explains the ways in which the use of biased FDRTs can lead distributional and recognitional injustices.
Privacy-preserving Object Detection
TLDR
This work shows that for object detection on COCO, both anonymizing the dataset by blurring faces, as well as swapping faces in a balanced manner along the gender and skin tone dimension, can retain object detection performances while preserving privacy and partially balancing bias.
Unlearnable Examples: Making Personal Data Unexploitable
TLDR
This work establishes an important first step towards making personal data unexploitable to deep learning models, and empirically verify the effectiveness of error-minimizing noise in both sample-wise and class-wise forms.
Towards decolonising computational sciences
TLDR
It is proposed that grappling with the authors' fields' histories and heritage holds the key to avoiding mistakes of the past and for these fields to progress away from their stagnant, sexist, and racist shared past into carving and maintaining an ecosystem where both a diverse demographics of researchers and scientific ideas that critically challenge the status quo are welcomed.
The Affective Growth of Computer Vision
The success of deep learning has led to intense growth and interest in computer vision, along with concerns about its potential impact on society. Yet we know little about how these changes have
The CREENDER Tool for Creating Multimodal Datasets of Images and Comments
TLDR
CREENDER, an annotation tool to create multimodal datasets with images associated with semantic tags and comments, which is made freely available under Apache 2.0 license.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 115 REFERENCES
Auditing ImageNet: Towards a Model-driven Framework for Annotating Demographic Attributes of Large-Scale Image Datasets
TLDR
This work introduces a model-driven framework for the automatic annotation of apparent age and gender attributes in large-scale image datasets and presents this work as the starting point for future development of unbiased annotation models and for the study of downstream effects of imbalances in the demographics of ImageNet.
Towards fairer datasets: filtering and balancing the distribution of the people subtree in the ImageNet hierarchy
TLDR
This paper examines ImageNet, a large-scale ontology of images that has spurred the development of many modern computer vision methods, and considers three key factors within the person subtree of ImageNet that may lead to problematic behavior in downstream computer vision technology.
From ImageNet to Image Classification: Contextualizing Progress on Benchmarks
TLDR
This work uses human studies to investigate the consequences of employing a noisy data collection pipeline and study how specific design choices in the ImageNet creation process impact the fidelity of the resulting dataset---including the introduction of biases that state-of-the-art models exploit.
80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition
TLDR
For certain classes that are particularly prevalent in the dataset, such as people, this work is able to demonstrate a recognition performance comparable to class-specific Viola-Jones style detectors.
ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases
TLDR
It is experimentally demonstrated that the accuracy and robustness of ConvNets measured on Imagenet are vastly underestimated and that explanations can mitigate the impact of misclassified adversarial examples from the perspective of the end-user.
Predictive Inequity in Object Detection
TLDR
This work annotates an existing large scale dataset which contains pedestrians with Fitzpatrick skin tones in ranges [1-3] or [4-6], and provides an in-depth comparative analysis of performance between these two skin tone groupings, finding that neither time of day nor occlusion explain this behavior.
Investigating the Impact of Inclusion in Face Recognition Training Data on Individual Face Identification
TLDR
This work audits ArcFace, a state-of-the-art, open source face recognition system, in a large-scale face identification experiment with more than one million distractor images and finds a Rank-1 face identification accuracy of 79.71% for individuals present in the model's training data and an accuracy of 75.73% for those not present.
Are we done with ImageNet?
TLDR
A significantly more robust procedure for collecting human annotations of the ImageNet validation set is developed, which finds the original ImageNet labels to no longer be the best predictors of this independently-collected set, indicating that their usefulness in evaluating vision models may be nearing an end.
How Computers See Gender
TLDR
It is found that FA services performed consistently worse on transgender individuals and were universally unable to classify non-binary genders, and that user perceptions about gender performance and identity contradict the way gender performance is encoded into the computer vision infrastructure.
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
TLDR
It is found that the performance on vision tasks increases logarithmically based on volume of training data size, and it is shown that representation learning (or pre-training) still holds a lot of promise.
...
1
2
3
4
5
...