# Experimental Design in Two-Sided Platforms: An Analysis of Bias

@article{Johari2020ExperimentalDI, title={Experimental Design in Two-Sided Platforms: An Analysis of Bias}, author={Ramesh Johari and Hannah Li and Gabriel Y. Weintraub}, journal={Proceedings of the 21st ACM Conference on Economics and Computation}, year={2020} }

We develop an analytical framework to study experimental design in two-sided platforms. In the settings we consider, customers rent listings; rented listings are occupied for some amount of time, then become available. Platforms typically use two common designs to study interventions in such settings: customer-side randomization (CR), and listing-side randomization (LR), along with associated estimators. We develop a stochastic model and associated mean field limit to capture dynamics in such…

## 19 Citations

Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms

- Computer Science, Mathematics
- 2021

A simple tractable market model is developed to study bias and variance in two common experiment types, where the platform randomizes individuals either on the supply side or on the demand side and the choice of treatment proportion can induce a bias-variance tradeoff.

Trustworthy Online Marketplace Experimentation with Budget-split Design

- Computer Science, Mathematics
- 2020

This paper proposes an experimental design that is unbiased in any marketplace where buyers have a defined budget, which could be finite or infinite, and shows that it is more powerful than all other unbiased designs in literature.

Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approach

- Computer Science, MathematicsNeurIPS
- 2020

Using a novel application of classical martingale analysis of Markov chains via Poisson's equation, efficient designs are characterized via a succinct convex optimization problem to propose a consistent, efficient online experimental design that adaptively samples the two Markov Chains.

The Limits of Centralized Pricing in Online Marketplaces and the Value of User Control∗

- 2021

We report experimental and quasi-experimental evidence from a “sharing economy” marketplace that transitioned from decentralized to centralized pricing. Centralized pricing increased the utilization…

Randomized Graph Cluster Randomization

- Computer Science, MathematicsArXiv
- 2020

This work proposes a randomized version of the GCR design, descriptively named randomized graph cluster randomization (RGCR), which uses a random clustering rather than a single fixed clustering, and proposes two randomized graph decomposition algorithms for use with RGCR, randomized 3-net and 1-hop-max, adapted from prior work on multiway graph cut problems.

Efficient Treatment Effect Estimation in Observational Studies under Heterogeneous Partial Interference

- Mathematics, Economics
- 2021

In many observational studies in social science and medical applications, subjects or individuals are connected, and one unit’s treatment and attributes may affect another unit’s treatment and…

Applied Machine Learning in Operations Management

- Computer Science
- 2020

This work highlights how both supervised and unsupervised learning shape operations management research in both descriptive and prescriptive analyses, and identifies several exciting future directions at the intersection of machine learning and operations management.

Treatment Effects in Market Equilibrium

- Economics, Mathematics
- 2021

In evaluating social programs, it is important to measure treatment effects within a market economy, where interference arises due to individuals buying and selling various goods at the prevailing…

Applied Machine Learning in Operations Management

- Innovative Technology at the Interface of Finance and Operations
- 2021

Causal Inference from Network Data

- Computer ScienceKDD
- 2021

This tutorial presents state-of-the-art research on causal inference from network data in the presence of interference. We start by motivating research in this area with real-world applications, such…

## References

SHOWING 1-10 OF 30 REFERENCES

Experimenting in Equilibrium

- Computer Science, MathematicsManag. Sci.
- 2021

This paper introduces a new approach to experimental design in large-scale stochastic systems with considerable cross-unit interference, under an assumption that the interference is structured enough that it can be captured via mean-field modeling.

Limiting Bias from Test-Control Interference in Online Marketplace Experiments

- Computer Science, Mathematics
- 2018

Using a simulation built on top of data from Airbnb, the use of methods from the network interference literature for online marketplace experimentation are considered and suggest that experiment design and analysis techniques are promising tools for reducing bias due to test-control interference in marketplace experiments.

Why marketplace experimentation is harder than it seems: the role of test-control interference

- Economics, Computer ScienceEC
- 2014

The simple economics of this bias in a supply and demand framework is presented, showing that the bias is larger in magnitude where there is more inelastic supply, and is positive if demand is elastic.

Detecting Network Effects: Randomizing Over Randomized Experiments

- Computer ScienceKDD
- 2017

A new experimental design is leverage for testing whether SUTVA holds, without making any assumptions on how treatment effects may spill over between the treatment and the control group, and the proposed methodology can be applied to settings in which a network is not necessarily observed but, if available, can be used in the analysis.

Randomization and The Pernicious Effects of Limited Budgets on Auction Experiments

- Computer Science, MathematicsAISTATS
- 2016

If an A/B experiment affects only bids, then the observed treatment effect is unbiased when all the bidders in an auction are randomly assigned to A or B but it can be severely biased otherwise, even in the absence of throttling.

Graph cluster randomization: network exposure to multiple universes

- Mathematics, Computer ScienceKDD
- 2013

It is shown that proper cluster randomization can lead to exponentially lower estimator variance when experimentally measuring average treatment effects under interference, and if a graph satisfies a restricted-growth condition on the growth rate of neighborhoods, then there exists a natural clustering algorithm, based on vertex neighborhoods, for which the variance of the estimator can be upper bounded by a linear function of the degrees.

Identification of Treatment Response with Social Interactions

- Computer Science
- 2012

This paper develops a formal language for study of treatment response with social interactions, and uses it to obtain new findings on identification of potential outcome distributions, and emphasizes that identification of possible outcome distributions differs from the longstanding econometric concern with identification of structural functions.

Exact p-Values for Network Interference

- Mathematics
- 2015

ABSTRACT We study the calculation of exact p-values for a large class of nonsharp null hypotheses about treatment effects in a setting with data from experiments involving members of a single…

Search Frictions and the Design of Online Marketplaces

- BusinessAMMA 2015
- 2015

Online marketplaces increasingly act as intermediaries in labor, housing, dating, and other markets where traders match with each other. These marketplaces use novel data generated by users’…

Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction

- Psychology
- 2015

Most questions in social and biomedical sciences are causal in nature: what would happen to individuals, or to groups, if part of their environment were changed? In this groundbreaking text, two…