What Do Different Evaluation Metrics Tell Us About Saliency Models?

@article{Bylinskii2019WhatDD,
  title={What Do Different Evaluation Metrics Tell Us About Saliency Models?},
  author={Zoya Bylinskii and Tilke Judd and Aude Oliva and Antonio Torralba and Fr{\'e}do Durand},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2019},
  volume={41},
  pages={740-757}
}
How best to evaluate a saliency model's ability to predict where humans look in images is an open research question. [] Key Result Building off the differences in metric properties and behaviors, we make recommendations for metric selections under specific assumptions and for specific applications.
Saliency Benchmarking Made Easy: Separating Models, Maps and Metrics
TLDR
This work derives optimal saliency maps for the most commonly used saliency metrics and shows that they can be computed analytically or approximated with high precision and allows researchers to have their model compete on many different metrics with state-of-the-art in those metrics.
Learning a Saliency Evaluation Metric Using Crowdsourced Perceptual Judgments
TLDR
Experimental results validate that the learned metric can be generalized to the comparisons of saliency maps from new images, new datasets, new models and synthetic data, and it also can be used to facilitate the development of new models for fixation prediction.
Saliency Benchmarking: Separating Models, Maps and Metrics
TLDR
A principled approach to solve the benchmarking problem of saliency models and saliency maps is proposed: a probabilistic model of fixation density prediction and a metric-specific prediction derived from the model density which maximizes the expected performance on that metric.
Simple baselines can fool 360° saliency metrics
TLDR
This paper proposes a new probabilistic metric based on the independent Bernoullis assumption that is more suited to the 360° saliency task and shows that a simple constant predictor can fool existing metrics and achieve results on par with specialized models.
What Catches the Eye? Visualizing and Understanding Deep Saliency Models
TLDR
The inner representations of deep models for fixation prediction are revealed and evidence that saliency, as experienced by humans, is likely to involve high-level semantic knowledge in addition to low-level perceptual cues is provided.
Saliency Prediction in the Deep Learning Era: An Empirical Investigation
TLDR
A large number of image and video saliency models are reviewed and compared over two image benchmarks and two large scale video datasets and factors that contribute to the gap between models and humans are identified.
Predicting Visual Saliency : Where Do People Look ?
TLDR
This literature survey will first introduce and explore the visual saliency task, and survey some of the most relevant methods and datasets in the research area, including seminal methods from the early years as well as the most recent state-of-the-art.
Saliency for free: Saliency prediction as a side-effect of object recognition
A Comparison Study of Saliency Models for Fixation Prediction on Infants and Adults
TLDR
A comprehensive comparison study of eight state-of-the-art saliency models on predictions of experimentally captured fixations from infants and adults shows a consistent performance ofsaliency models predicting adult fixations over infant fixations in terms of overlap, center fitting, intersection, information loss of approximation, and spatial distance between the distributions of saliency map and fixation map.
Revisiting Saliency Metrics: Farthest-Neighbor Area Under Curve
  • Sen Jia, Neil D. B. Bruce
  • Computer Science
    2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2020
TLDR
This paper proposes a new metric to address the long-standing problem of center bias in saliency evaluation that is AUC-based because ROC curves are relatively robust to the standard deviation problem, and proposes a global smoothing function for the problem of few value degrees in predicted saliency output.
...
...

References

SHOWING 1-10 OF 108 REFERENCES
Where Should Saliency Models Look Next?
TLDR
It is argued that to continue to approach human-level performance, saliency models will need to discover higher-level concepts in images: text, objects of gaze and action, locations of motion, and expected locations of people in images.
Information-theoretic model comparison unifies saliency metrics
TLDR
This work brings saliency evaluation into the domain of information by framing fixation prediction models probabilistically and calculating information gain, and jointly optimize the scale, the center bias, and spatial blurring of all models within this framework.
How close are we to understanding image-based saliency?
TLDR
This work frames saliency models probabilistically as point processes, allowing the calculation of log-likelihoods and bringing saliency evaluation into the domain of information and provides a principled method to show where and how models fail to capture information in the fixations.
A Data-Driven Metric for Comprehensive Evaluation of Saliency Models
TLDR
Experimental results show that the data-driven metric performs the most consistently with the human-being in evaluating saliency maps as well as saliency models.
Analysis of Scores, Datasets, and Models in Visual Saliency Prediction
TLDR
A critical and quantitative look at challenges in saliency modeling and the way they affect model accuracy is pursued, providing a comprehensive high-level picture of the strengths and weaknesses of many popular models, and suggests future research directions in Saliency modeling.
Saliency and Human Fixations: State-of-the-Art and Study of Comparison Metrics
TLDR
This paper compares the ranking of 12 state-of-the art saliency models using 12 similarity metrics and shows that some of the metrics are strongly correlated leading to a redundancy in the performance metrics reported in the available benchmarks.
Selection of a best metric and evaluation of bottom-up visual saliency models
Spatially Binned ROC: A Comprehensive Saliency Metric
  • C. Wloka, John Tstotsos
  • Computer Science
    2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2016
TLDR
A baseline measure of inherent algorithm bias is used to show that Adaptive Whitening Saliency, Attention by Information Maximization, and Dynamic Visual Attention provide the least spatially biased results, suiting them for tasks in which there is no information about the underlying spatial bias of the stimuli.
A Benchmark of Computational Models of Saliency to Predict Human Fixations
TLDR
A benchmark data set containing 300 natural images with eye tracking data from 39 observers is proposed to compare model performances and it is shown that human performance increases with the number of humans to a limit.
...
...