More Powerful Multiple Testing in Randomized Experiments with Non-Compliance

  title={More Powerful Multiple Testing in Randomized Experiments with Non-Compliance},
  author={Joseph J Lee and Laura Forastiere and Luke Miratrix and Natesh S. Pillai},
  journal={arXiv: Methodology},
Two common concerns raised in analyses of randomized experiments are (i) appropriately handling issues of non-compliance, and (ii) appropriately adjusting for multiple tests (e.g., on multiple outcomes or subgroups). Although simple intention-to-treat (ITT) and Bonferroni methods are valid in terms of type I error, they can each lead to a substantial loss of power; when employing both simultaneously, the total loss may be severe. Alternatives exist to address each concern. Here we propose an… Expand
Valid randomization-based p-values for partially post hoc subgroup analyses.
This work clarifies the source of partially post hoc subgroup analyses' invalidity, proposes a randomization-based approach for generating valid posterior predictive p-values, and investigates the approach's operating characteristics in a simple illustrative setting, showing that it can have desirable properties under both null and alternative hypotheses. Expand
Randomization Inference and Bayesian Inference in Regression Discontinuity Designs: An Application to Italian University Grants
This work adopts a probabilistic formulation of the assignment mechanism underling RD designs and selects suitable subpopulations around the cutoff point on the basis of observed covariates using both randomization tests and a Bayesian model-based approach both accounting for the problem of multiple testing. Expand
A randomization-based causal inference framework for uncovering environmental exposure effects on human gut microbiota
A causal inference framework based on the Rubin Causal Model is introduced that can help scientists to investigate such environment-host microbiome relationships, to capitalize on existing, possibly powerful, test statistics, and test plausible sharp null hypotheses. Expand
Local Randomization and Beyond for Regression Discontinuity Designs: Revisiting a Causal Analysis of the Effects of University Grants on Dropout Rates
Regression discontinuity designs (RDDs) are a common quasi-experiment in economics, education, medicine, and statistics. While the most popular methodologies for analyzing RDDs are local polynomialExpand


More powerful randomization-based p-values in double-blind trials with non-compliance.
  • D. Rubin
  • Computer Science, Medicine
  • Statistics in medicine
  • 1998
Bayesian analysis under this alternative 'exclusion' hypothesis leads to new estimates of the effect of receipt of treatment, and to a new randomization-based procedure that has frequentist validity yet can be substantially more powerful than the standard intent-to-treat procedure. Expand
Bayesian inference for causal effects in randomized experiments with noncompliance
For most of this century, randomization has been a cornerstone of scientific experimentation, especially when dealing with humans as experimental units. In practice, however, noncompliance isExpand
No Adjustments Are Needed for Multiple Comparisons
  • K. Rothman
  • Computer Science, Medicine
  • Epidemiology
  • 1990
A policy of not making adjustments for multiple comparisons is preferable because it will lead to fewer errors of interpretation when the data under evaluation are not random numbers but actual observations on nature. Expand
Estimating causal effects of treatments in randomized and nonrandomized studies.
A discussion of matching, randomization, random sampling, and other methods of controlling extraneous variation is presented. The objective is to specify the benefits of randomization in estimatingExpand
What's wrong with Bonferroni adjustments
This paper advances the view, widely held by epidemiologists, that Bonferroni adjustments are, at best, unnecessary and, at worst, deleterious to sound statistical inference. Expand
Exact significance levels for multiple binomial testing with application to carcinogenicity screens.
A method for calculating the exact permutational probability of at least one significant Fisher-Irwin test when only one treatment group and one control group is used and upper and lower bounds on the probability are provided. Expand
Causal Inference Using Potential Outcomes
Causal effects are defined as comparisons of potential outcomes under different treatments on a common set of units. Observed values of the potential outcomes are revealed by the assignmentExpand
Principal stratification in causal inference.
This work proposes a general framework for comparing treatments adjusting for posttreatment variables that yields principal effects based on principal stratification and attacks the problem of surrogate or biomarker endpoints, where all current definitions of surrogacy do not generally have the desired interpretation as causal effects of treatment on outcome. Expand
Identification of Causal Effects Using Instrumental Variables
It is shown that the instrumental variables (IV) estimand can be embedded within the Rubin Causal Model (RCM) and that under some simple and easily interpretable assumptions, the IV estimand is the average causal effect for a subgroup of units, the compliers. Expand
Likelihood-Based Analysis of Causal Effects of Job-Training Programs Using Principal Stratification
Government-sponsored job-training programs must be subject to evaluation to assess whether their effectiveness justifies their cost to the public. The evaluation usually focuses on employment andExpand