Active Sampling for Pairwise Comparisons via Approximate Message Passing and Information Gain Maximization

  title={Active Sampling for Pairwise Comparisons via Approximate Message Passing and Information Gain Maximization},
  author={A. Mikhailiuk and Clifford Wilmot and M. P{\'e}rez-Ortiz and Dingcheng Yue and Rafał K. Mantiuk},
  journal={2020 25th International Conference on Pattern Recognition (ICPR)},
Pairwise comparison data arise in many domains with subjective assessment experiments, for example in image and video quality assessment. In these experiments observers are asked to express a preference between two conditions. However, many pairwise comparison protocols require a large number of comparisons to infer accurate scores, which may be unfeasible when each comparison is time-consuming (e.g. videos) or expensive (e.g. medical imaging). This motivates the use of an active sampling… Expand
Consolidated Dataset and Metrics for High-Dynamic-Range Image Quality
A Unified Photometric Image Quality dataset (UPIQ) with over 4,000 images is created by realigning and merging existing HDR and standard-dynamic-range (SDR) datasets and the utility of the dataset on brightness aware image compression is shown. Expand
A perceptual model of motion quality for rendering with adaptive refresh-rate and resolution
A perceptual visual model is proposed that predicts the quality of motion given an object velocity and predictability of motion, and an on-the-fly motion-adaptive rendering algorithm that adjusts the refresh rate of a G-Sync-capable monitor based on a given rendering budget and observed object motion is demonstrated. Expand


Pairwise ranking aggregation in a crowdsourced setting
This work proposes a new model to predict a gold-standard ranking that hinges on combining pairwise comparisons via crowdsourcing and formalizes this as an active learning strategy that incorporates an exploration-exploitation tradeoff and implements it using an efficient online Bayesian updating scheme. Expand
Active Sampling for Subjective Image Quality Assessment
  • Peng Ye, D. Doermann
  • Computer Science
  • 2014 IEEE Conference on Computer Vision and Pattern Recognition
  • 2014
A hybrid subjective test which combines MOS and PC tests via a unified probabilistic model and an active sampling method is presented which outperforms state-of-the-art subjective IQA tests in a crowdsourced setting. Expand
Random partial paired comparison for subjective video quality assessment via hodgerank
A novel framework of HodgeRank on Random Graphs (HRRG) is proposed to achieve efficient and reliable subjective Video Quality Assessment (VQA) to address the challenge of a potentially large number of combinations of videos to be assessed. Expand
Visual Quality Assessment for Motion Compensated Frame Interpolation
A subjective quality assessment study by crowdsourcing for the interpolated images provided in one of the optical flow benchmarks, the Middlebury benchmark, gives rise to a re-ranking that shows the necessity of visual quality assessment as another evaluation metric for optical flow and frame interpolation benchmarks. Expand
PieAPP: Perceptual Image-Error Assessment Through Pairwise Preference
A new learning-based method that is the first to predict perceptual image error like human observers, and significantly outperforms existing algorithms, beating the state-of-the-art by almost 3× on the authors' test set in terms of binary error rate, while also generalizing to new kinds of distortions, unlike previous learning- based methods. Expand
Bayesian Decision Process for Cost-Efficient Dynamic Ranking via Crowdsourcing
This work model the active sampling problem in crowdsourced ranking as a Bayesian Markov decision process, which dynamically selects item pairs and workers to improve the ranking accuracy under a budget constraint and develops a computationally efficient sampling policy based on knowledge gradient as well as a moment matching technique for posterior approximation. Expand
Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence
This work considers parametric ordinal models for pairwise comparison data involving a latent vector w* e Rd that represents the "qualities" of the d items being compared; this class of models includes the two most widely used parametric models|the Bradley-Terry-Luce (BTL) and the Thurstone models. Expand
HodgeRank With Information Maximization for Crowdsourced Pairwise Ranking Aggregation
The principle of information maximization for active sampling strategies in the framework of HodgeRank, an approach based on Hodge Decomposition of pairwise ranking data with multiple workers is studied. Expand
Active Ranking using Pairwise Comparisons
This paper proposes a robust, error-tolerant algorithm that only requires that the pairwise comparisons are probably correct and demonstrates an algorithm that can identify a randomly selected ranking using just slightly more than $d log n$ adaptively selected couplewise comparisons, on average. Expand
A practical guide and software for analysing pairwise comparison experiments
This paper improves on existing scaling methods by introducing outlier analysis, providing methods for computing confidence intervals and statistical testing and introducing a prior, which reduces estimation error when the number of observers is low. Expand