• Corpus ID: 254246754

Measure of Strength of Evidence for Visually Observed Differences between Subpopulations

@inproceedings{Yang2021MeasureOS,
  title={Measure of Strength of Evidence for Visually Observed Differences between Subpopulations},
  author={Xi Yang and Jan Hannig and Katherine A. Hoadley and Iain Carmichael and J. S. Marron},
  year={2021}
}

• Xi Yang, Jan Hannig, +2 authors J. S. Marron
• Published 2 January 2021
• Computer Science

An increasingly important data analytic challenge is understanding the relationships between subpopulations. Various visualization methods (PCA, tSNE, UMAP) that provide many useful insights into those relationships are popular, especially in high dimensional contexts such as bioinformatics. While visualization is often in-sightful, it can also be deceptive. This motivates the need for careful assessment of the strength of the evidence for differences between subpopulations. Because distributional models are dubious in many high dimensional contexts, permutation tests are considered. In the commonly encountered high signal (i.e large subpopulation difference) contexts classical permutation p-values frequently fail to give meaningful pairwise comparisons as they all tend to take on the same small value, determined only by the number of permutations performed. This paper addresses this challenge by proposing a novel and rigorous approach to quantifying such subpopulation relationships called the Population Difference Criterion (PDC). The PDC draws mo-tivation from classical permutation based hypothesis testing, but reformulates some methodology in a novel direction that is needed in contexts where the strength of the evidence in all comparisons is beyond the ability of p-values to reasonably quantify. This paper makes a number of other important contributions, besides proposal of the PDC. First a careful investigation finds that a balanced permutation approach is much more informative, in high signal contexts, than conventional approaches based on all permutations. That recommendation is diametrically opposite of that from related work done in low signal contexts. Another important contribution is mathematical resolution of this paradox, which also leads to further improvement of both permutation types. One more contribution is quantification of permutation variation via proposal of a bootstrap confidence interval. The practical usefulness of all of these ideas is illustrated using subpopulations in modern cancer data.