A Survey on Performance Metrics for Object-Detection Algorithms

@article{Padilla2020ASO,
  title={A Survey on Performance Metrics for Object-Detection Algorithms},
  author={Rafael Padilla and Sergio L. Netto and Eduardo A. B. da Silva},
  journal={2020 International Conference on Systems, Signals and Image Processing (IWSSIP)},
  year={2020},
  pages={237-242}
}
This work explores and compares the plethora of metrics for the performance evaluation of object-detection algorithms. Average precision (AP),for instance, is a popular metric for evaluating the accuracy of object detectors by estimating the area under the curve (AUC) of the precision × recall relationship. Depending on the point interpolation used in the plot, two different AP variants can be defined and, therefore, different results are generated. AP has six additional variants increasing the… 

Figures and Tables from this paper

Sensitivity of Average Precision to Bounding Box Perturbations
TLDR
This work quantifies the sensitivity of AP to bounding box perturbations and shows that AP is very sensitive to small translations, explaining why achieving higher mAP becomes increasingly harder as models get better.
Tools, techniques, datasets and application areas for object detection in an image: a review
TLDR
A systematic review has been followed to summarize the current research work’s findings and discuss seven research questions related to object detection.
VmAP: A Fair Metric for Video Object Detection
TLDR
This paper shows several disadvantages of mAP as a metric and suggests a novel evaluation metric (VmAP) which takes the focus away from evaluating detections on every frame, and shows that VmAP is able to address all the challenges with the mAP.
An Empirical Study and Comparison of Recent Few-Shot Object Detection Algorithms
TLDR
An empirical study and comparison has been conducted on the recent achievements of FSOD, and a new taxonomy is proposed based on the role of prior knowledge during object detection of novel classes.
Performance Indicator Survey for Object Detection
  • In-Deok Park, Sungho Kim
  • Computer Science
    2020 20th International Conference on Control, Automation and Systems (ICCAS)
  • 2020
TLDR
The main purpose of the survey is that researchers find the proper performance indicator for object detection, which can help to compare the detection result with a different algorithm result, exactly and effectively.
Object-level change detection with a dual correlation attention-guided detector
Detecting Object States vs Detecting Objects: A New Dataset and a Quantitative Experimental Study
TLDR
This paper introduces the Object State Detection Dataset (OSDD) , a new publicly available dataset consisting of more than 19,000 annotations for 18 object categories and 9 state classes and confirms that SD is harder than OD and that tailored SD methods need to be developed for addressing effectively this significant problem.
Camera Latency Review and Parameters Testing for Real-Time Object Detection Implementation
TLDR
This paper reviews latencies caused by IP Camera and USB Webcam, and data is used to compare those latencies to theoretical standard of real-time, imperceptible latency, to be compiled into a optimized parameters to be used in other implementations.
Object Detection using Deep Learning: A Review
TLDR
This paper reviews deep learning-based object detection models, and the performance evaluation of different detectors on different datasets based on mean Average Precision (mAP) is reviewed.
Continual Object Detection: A review of definitions, strategies, and challenges
TLDR
A short and systematic review of the methods that propose solutions to traditional incremental object detection scenarios, and a comprehensive evaluation of the existing approaches using a new metric to quantify the stability and plasticity of each technique in a standard way.
...
...

References

SHOWING 1-10 OF 48 REFERENCES
Empirical Upper Bound in Object Detection and More
TLDR
It is found that models generate boxes on empty regions and that context is more important for detecting small objects than larger ones and that classification error explains the largest fraction of errors and weighs more than localization and duplicate errors.
The Pascal Visual Object Classes Challenge: A Retrospective
TLDR
A review of the Pascal Visual Object Classes challenge from 2008-2012 and an appraisal of the aspects of the challenge that worked well, and those that could be improved in future challenges.
Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors
TLDR
A unified implementation of the Faster R-CNN, R-FCN and SSD systems is presented and the speed/accuracy trade-off curve created by using alternative feature extractors and varying other critical parameters such as image size within each of these meta-architectures is traced out.
Focal Loss for Dense Object Detection
TLDR
This paper proposes to address the extreme foreground-background class imbalance encountered during training of dense detectors by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples, and develops a novel Focal Loss, which focuses training on a sparse set of hard examples and prevents the vast number of easy negatives from overwhelming the detector during training.
You Only Look Once: Unified, Real-Time Object Detection
TLDR
Compared to state-of-the-art detection systems, YOLO makes more localization errors but is less likely to predict false positives on background, and outperforms other detection methods, including DPM and R-CNN, when generalizing from natural images to other domains like artwork.
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
TLDR
This paper proposes a simple and scalable detection algorithm that improves mean average precision (mAP) by more than 30% relative to the previous best result on VOC 2012 -- achieving a mAP of 53.3%.
SSD: Single Shot MultiBox Detector
TLDR
The approach, named SSD, discretizes the output space of bounding boxes into a set of default boxes over different aspect ratios and scales per feature map location, which makes SSD easy to train and straightforward to integrate into systems that require a detection component.
Microsoft COCO: Common Objects in Context
We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene
Optimization of CNN-Based Object Detection Algorithms for Embedded Systems
TLDR
This thesis proposes a technique to reduce the size and the computational requirements of CNN-based object detection by using post-training low bit-width quantization, and analyzes the possibility of using two consecutive inferences to obtain a highly-accurate result yet with an energy-efficient procedure.
To boost or not to boost? On the limits of boosted trees for object detection
TLDR
This study reveals the limited modeling capacity of the common boosted trees model, motivating a need for architectural changes in order to compete with multi-level and very deep architectures.
...
...