Computing the Testing Error Without a Testing Set

@article{Corneanu2020ComputingTT,
  title={Computing the Testing Error Without a Testing Set},
  author={Ciprian Adrian Corneanu and Meysam Madadi and Sergio Escalera and Aleix M. Martinez},
  journal={2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2020},
  pages={2674-2682}
}
Deep Neural Networks (DNNs) have revolutionized computer vision. We now have DNNs that achieve top (accuracy) results in many problems, including object recognition, facial expression analysis, and semantic segmentation, to name but a few. The design of the DNNs that achieve top results is, however, non-trivial and mostly done by trail-and-error. That is, typically, researchers will derive many DNN architectures (\ie, topologies) and then test them on multiple datasets. However, there are no… 
Are Labels Always Necessary for Classifier Accuracy Evaluation?
  • Weijian Deng, Liang Zheng
  • Computer Science
    2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2021
TLDR
This work constructs a meta-dataset: a dataset comprised of datasets generated from the original images via various transformations such as rotation, background substitution, foreground scaling, etc, and reports a reasonable and promising prediction of the model accuracy.
Practical Assessment of Generalization Performance Robustness for Deep Networks via Contrastive Examples
TLDR
This work proposes a practical framework ContRE1 that uses Contrastive examples for DNN geneRalization performance Estimation that adopts classification errors and Fisher ratios on the generated contrastive examples to assess and analyze the generalization performance of DNN models in complement with a testing set.
Deep Neural Network Pruning Using Persistent Homology
  • Satoru Watanabe, H. Yamana
  • Computer Science
    2020 IEEE Third International Conference on Artificial Intelligence and Knowledge Engineering (AIKE)
  • 2020
TLDR
A novel method of network pruning, hereafter referred to as PHPM, using persistent homology (PH), which prunes DNNs in ascending order of magnitudes of the combinational effects among neurons, to prevent the deterioration of the accuracy.
Leveraging The Topological Consistencies of Learning in Deep Neural Networks
TLDR
This work defines a new class of topological features that accurately characterize the progress of learning while being quick to compute during running time and can induce learning in DNNs by actively constraining the DNN’s topological structure.
What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?
TLDR
This work trains semantic classification and rotation prediction in a multi-task way and reports an interesting finding, i.e., the semantic classification accuracy exhibits a strong linear relationship with the accuracy of the rotation prediction task (Pearson's Correlation r > 0.88).
When do GANs replicate? On the choice of dataset size
Do GANs replicate training images? Previous studies have shown that GANs do not seem to replicate training data without significant change in the training procedure. This leads to a series of
In Search of Probeable Generalization Measures
TLDR
This paper evaluates and compares measures, demonstrating effectiveness and robustness across model variations, dataset complexities, training hyperparameters, and training stages, and introduces a new dataset of trained models and performance metrics, GenProb, for testing generalization measures, model tuning algorithms and optimization algorithms.
How Can Increased Randomness in Stochastic Gradient Descent Improve Generalization?
TLDR
This work describes test set performance under a shift between the training and test data distributions for loss functions with multiple minima, and implies that SGD, including its stationary distribution, is largely invariant to changes in learning rate or batch size that leave its temperature constant.
Label-Free Model Evaluation with Semi-Structured Dataset Representations
TLDR
This work proposes a new semi-structured dataset representation that is manageable for regression learning while containing rich information for AutoEval, and integrates distribution shapes, clusters, and representative samples for a semi- structuring dataset representation.
Ranking Models in Unlabeled New Environments
TLDR
The problem of ranking models in unlabeled new environments is introduced, a proxy dataset that is fully labeled and well reflects the true model rankings in a given target environment is proposed, and a carefully constructed proxy set effectively captures relative performance ranking in new environments.
...
1
2
...

References

SHOWING 1-10 OF 33 REFERENCES
Deep Residual Learning for Image Recognition
TLDR
This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
Learning Multiple Layers of Features from Tiny Images
TLDR
It is shown how to train a multi-layer generative model that learns to extract meaningful features which resemble those found in the human visual cortex, using a novel parallelization algorithm to distribute the work among multiple machines connected on a network.
ImageNet classification with deep convolutional neural networks
TLDR
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
Very Deep Convolutional Networks for Large-Scale Image Recognition
TLDR
This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
Show and tell: A neural image caption generator
TLDR
This paper presents a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image.
What Does It Mean to Learn in Deep Networks? And, How Does One Detect Adversarial Attacks?
TLDR
A novel approach is derived to define what it means to learn in deep networks, and how to use this knowledge to detect adversarial attacks and define the ability of a network to generalize to unseen testing samples.
Predicting the Generalization Gap in Deep Networks with Margin Distributions
TLDR
This paper proposes a measure based on the concept of margin distribution, which are the distances of training points to the decision boundary, and finds that it is necessary to use margin distributions at multiple layers of a deep network.
What does it mean to learn in deep networks ? And , how does one detect adversarial attacks ?
The flexibility and high-accuracy of Deep Neural Networks (DNNs) has transformed computer vision. But, the fact that we do not know when a specific DNN will work and when it will fail has resulted in
The Pascal Visual Object Classes Challenge: A Retrospective
TLDR
A review of the Pascal Visual Object Classes challenge from 2008-2012 and an appraisal of the aspects of the challenge that worked well, and those that could be improved in future challenges.
Fully Convolutional Networks for Semantic Segmentation
TLDR
It is shown that convolutional networks by themselves, trained end- to-end, pixels-to-pixels, improve on the previous best result in semantic segmentation.
...
1
2
3
4
...