Experimental Design in Two-Sided Platforms: An Analysis of Bias

  title={Experimental Design in Two-Sided Platforms: An Analysis of Bias},
  author={Ramesh Johari and Hannah Li and Gabriel Y. Weintraub},
  journal={Proceedings of the 21st ACM Conference on Economics and Computation},
  • R. Johari, Hannah Li, G. Weintraub
  • Published 13 February 2020
  • Mathematics, Economics, Computer Science
  • Proceedings of the 21st ACM Conference on Economics and Computation
We develop an analytical framework to study experimental design in two-sided platforms. In the settings we consider, customers rent listings; rented listings are occupied for some amount of time, then become available. Platforms typically use two common designs to study interventions in such settings: customer-side randomization (CR), and listing-side randomization (LR), along with associated estimators. We develop a stochastic model and associated mean field limit to capture dynamics in such… 
Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms
A simple tractable market model is developed to study bias and variance in two common experiment types, where the platform randomizes individuals either on the supply side or on the demand side and the choice of treatment proportion can induce a bias-variance tradeoff.
Trustworthy Online Marketplace Experimentation with Budget-split Design
This paper proposes an experimental design that is unbiased in any marketplace where buyers have a defined budget, which could be finite or infinite, and shows that it is more powerful than all other unbiased designs in literature.
Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approach
Using a novel application of classical martingale analysis of Markov chains via Poisson's equation, efficient designs are characterized via a succinct convex optimization problem to propose a consistent, efficient online experimental design that adaptively samples the two Markov Chains.
The Limits of Centralized Pricing in Online Marketplaces and the Value of User Control∗
We report experimental and quasi-experimental evidence from a “sharing economy” marketplace that transitioned from decentralized to centralized pricing. Centralized pricing increased the utilization
Randomized Graph Cluster Randomization
This work proposes a randomized version of the GCR design, descriptively named randomized graph cluster randomization (RGCR), which uses a random clustering rather than a single fixed clustering, and proposes two randomized graph decomposition algorithms for use with RGCR, randomized 3-net and 1-hop-max, adapted from prior work on multiway graph cut problems.
Efficient Treatment Effect Estimation in Observational Studies under Heterogeneous Partial Interference
In many observational studies in social science and medical applications, subjects or individuals are connected, and one unit’s treatment and attributes may affect another unit’s treatment and
Applied Machine Learning in Operations Management
This work highlights how both supervised and unsupervised learning shape operations management research in both descriptive and prescriptive analyses, and identifies several exciting future directions at the intersection of machine learning and operations management.
Treatment Effects in Market Equilibrium
In evaluating social programs, it is important to measure treatment effects within a market economy, where interference arises due to individuals buying and selling various goods at the prevailing
Applied Machine Learning in Operations Management
Causal Inference from Network Data
This tutorial presents state-of-the-art research on causal inference from network data in the presence of interference. We start by motivating research in this area with real-world applications, such


Experimenting in Equilibrium
This paper introduces a new approach to experimental design in large-scale stochastic systems with considerable cross-unit interference, under an assumption that the interference is structured enough that it can be captured via mean-field modeling.
Limiting Bias from Test-Control Interference in Online Marketplace Experiments
Using a simulation built on top of data from Airbnb, the use of methods from the network interference literature for online marketplace experimentation are considered and suggest that experiment design and analysis techniques are promising tools for reducing bias due to test-control interference in marketplace experiments.
Why marketplace experimentation is harder than it seems: the role of test-control interference
The simple economics of this bias in a supply and demand framework is presented, showing that the bias is larger in magnitude where there is more inelastic supply, and is positive if demand is elastic.
Detecting Network Effects: Randomizing Over Randomized Experiments
A new experimental design is leverage for testing whether SUTVA holds, without making any assumptions on how treatment effects may spill over between the treatment and the control group, and the proposed methodology can be applied to settings in which a network is not necessarily observed but, if available, can be used in the analysis.
Randomization and The Pernicious Effects of Limited Budgets on Auction Experiments
If an A/B experiment affects only bids, then the observed treatment effect is unbiased when all the bidders in an auction are randomly assigned to A or B but it can be severely biased otherwise, even in the absence of throttling.
Graph cluster randomization: network exposure to multiple universes
It is shown that proper cluster randomization can lead to exponentially lower estimator variance when experimentally measuring average treatment effects under interference, and if a graph satisfies a restricted-growth condition on the growth rate of neighborhoods, then there exists a natural clustering algorithm, based on vertex neighborhoods, for which the variance of the estimator can be upper bounded by a linear function of the degrees.
Identification of Treatment Response with Social Interactions
This paper develops a formal language for study of treatment response with social interactions, and uses it to obtain new findings on identification of potential outcome distributions, and emphasizes that identification of possible outcome distributions differs from the longstanding econometric concern with identification of structural functions.
Exact p-Values for Network Interference
ABSTRACT We study the calculation of exact p-values for a large class of nonsharp null hypotheses about treatment effects in a setting with data from experiments involving members of a single
Search Frictions and the Design of Online Marketplaces
Online marketplaces increasingly act as intermediaries in labor, housing, dating, and other markets where traders match with each other. These marketplaces use novel data generated by users’
Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction
Most questions in social and biomedical sciences are causal in nature: what would happen to individuals, or to groups, if part of their environment were changed? In this groundbreaking text, two