• DOI:10.2139/ssrn.4115866
• Corpus ID: 247922765

The Most Difference in Means: A Statistic for the Strength of Null and Near-Zero Results

@article{Corliss2022TheMD,
  title={The Most Difference in Means: A Statistic for the Strength of Null and Near-Zero Results},
  author={Bruce A. Corliss and Taylor R. Brown and Ting Zhang and Kevin A. Janes and Heman Shakeri and Philip E. Bourne},
  journal={SSRN Electronic Journal},
  year={2022}
}

• Bruce A. Corliss, Taylor R. Brown, +3 authors P. Bourne
• Published 4 January 2022
• Physics
• SSRN Electronic Journal

: Statistical insignificance does not suggest the absence of effect, yet scientists must often use null results as evidence of negligible (near-zero) effect size to falsify scientific hypotheses. Doing so must assess a result’s null strength, defined as the evidence for a negligible effect size. Such an assessment would differentiate high null strength results that suggest a negligible effect size from low null strength results that suggest a broad range of potential effect sizes. We propose the most difference in means (δ M ) as a two-sample statistic that can both quantify null strength and perform a hypothesis test for determining negligible effect size. Scientists will likely disagree on the value of the threshold for negligible effect. To facilitate consensus when interpreting results, our statistic allows scientists to conclude that a result has negligible effect size using different thresholds with no recalculation of the statistic required. This is accomplished with a post-hoc hypothesis test for negligibility. To assist with selecting a threshold, δ M can also compare null strength between related results. Both δ M and the relative form of δ M outperform other candidate statistics in identifying results with higher null strength. We compile results from broadly related experiments and use the relative δ M to compare null strength across different treatments, measurement methods, and experiment models. Reporting the relative δ M may provide a technical solution to the file drawer problem by encouraging the publication of null and near-zero results and allowing scientists with different perspectives to reach consensus for which results have negligible effect. highlight results with exceptionally strong null strength. All of this can be done without recalculation of the statistic. To test our statistics against previously developed candidates, we characterize the multidimensional problem of assessing null strength with various functions of population parameters. These functions serve as ground truth for simulation testing. We use an integrated risk assessment to test δ M and the relative form of δ M (rδ M ) against several candidate statistics by evaluating their error rates in comparing the null strength between simulated experiment results. Our statistics were the only candidates that demonstrated better than random error rates across all investigations. We illustrate with real data how rδ M can be used to test for negligibility and compare the null strength of results from broadly related experiments that have a combination of different experiment models, conditions, populations, species, timepoints, treatments, and measurement techniques. We propose that reporting rδ M of null and near-zero results will provide a more useful interpretation than alternative analysis techniques.