Corpus ID: 202540050

Testing Deep Learning Models for Image Analysis Using Object-Relevant Metamorphic Relations

  title={Testing Deep Learning Models for Image Analysis Using Object-Relevant Metamorphic Relations},
  author={Yongqiang Tian and Shiqing Ma and Ming Wen and Yepang Liu and S. C. Cheung and X. Zhang},
Deep learning models are widely used for image analysis. While they offer high performance in terms of accuracy, people are concerned about if these models inappropriately make inferences using irrelevant features that are not encoded from the target object in a given image. To address the concern, we propose a metamorphic testing approach that assesses if a given inference is made based on irrelevant features. Specifically, we propose two novel metamorphic relations to detect such… Expand
Exposing previously undetectable faults in deep neural networks
This paper introduces a novel DNN testing method that is able to find faults in DNNs that other methods cannot, and can generate fresh test inputs that vary in their high-level features by leveraging generative machine learning. Expand
Attack as defense: characterizing adversarial examples using robustness
This work proposes a novel defense framework, named attack as defense (A2D), to detect adversarial examples by effectively evaluating an example’s robustness, and shows that A2D is more effective than recent promising approaches. Expand
AUTOTRAINER: An Automatic DNN Training Problem Detection and Repair System
AUTOTRAINER is proposed, a DNN training monitoring and automatic repairing tool which supports detecting and auto repairing five commonly seen training problems and can effectively detect all potential problems with 100% detection rate and no false positives. Expand


Validating a Deep Learning Framework by Metamorphic Testing
The framework is first validated with a classifier built on support vector machine, and then it is tested using a metamorphic validation approach, demonstrated by validating a deep learning classifier for automated classification of biology cell images. Expand
Identifying implementation bugs in machine learning based image classifiers using metamorphic testing
This work presents the solution approach, based on the concept of Metamorphic Testing, which aims to identify implementation bugs in ML based image classifiers and developed metamorphic relations for an application based on Support Vector Machine and a Deep Learning based application. Expand
Deep Residual Learning for Image Recognition
This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. Expand
SSD: Single Shot MultiBox Detector
The approach, named SSD, discretizes the output space of bounding boxes into a set of default boxes over different aspect ratios and scales per feature map location, which makes SSD easy to train and straightforward to integrate into systems that require a detection component. Expand
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
This work introduces a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals and further merge RPN and Fast R-CNN into a single network by sharing their convolutionAL features. Expand
Bag of Tricks for Image Classification with Convolutional Neural Networks
This paper examines a collection of training procedure refinements and empirically evaluates their impact on the final model accuracy through ablation study, and shows that by combining these refinements together, they are able to improve various CNN models significantly. Expand
DeepRoad: GAN-Based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems
The experimental results demonstrate that DeepRoad can detect thousands of inconsistent behaviors for DNN-based autonomous driving systems, and effectively validate input images to potentially enhance the system robustness as well. Expand
What Makes for Effective Detection Proposals?
An in-depth analysis of twelve proposal methods along with four baselines regarding proposal repeatability, ground truth annotation recall on PASCAL, ImageNet, and MS COCO, and their impact on DPM, R-CNN, and Fast R- CNN detection performance shows that for object detection improving proposal localisation accuracy is as important as improving recall. Expand
Aggregated Residual Transformations for Deep Neural Networks
On the ImageNet-1K dataset, it is empirically show that even under the restricted condition of maintaining complexity, increasing cardinality is able to improve classification accuracy and is more effective than going deeper or wider when the authors increase the capacity. Expand
ImageNet classification with deep convolutional neural networks
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective. Expand