Network Experimentation at Scale

@article{Karrer2020NetworkEA,
  title={Network Experimentation at Scale},
  author={Brian Karrer and Liang Shi and Monica Bhole and Matt Goldman and Tyrone Palmer and Charlie Gelman and Mikael Konutgan and Feng Sun},
  journal={Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery \& Data Mining},
  year={2020}
}
  • B. KarrerLiang Shi Feng Sun
  • Published 15 December 2020
  • Computer Science, Economics
  • Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining
We describe our network experimentation framework, deployed at Facebook, which accounts for interference between experimental units. We document this system, including the design and estimation procedures, and detail insights we have gained from the many experiments that have used this system at scale. In our estimation procedure, we introduce a cluster-based regression adjustment that substantially improves precision for estimating global treatment effects, as well as a procedure to test for… 
View on ACM
[PDF] Semantic Reader
10 Citations
Highly Influential Citations
1
Background Citations
4
Methods Citations
3

Figures and Tables from this paper

10 Citations

Model-Based Regression Adjustment with Model-Free Covariates for Network Interference

This work introduces a sequential procedure to generate and select graph- and treatment-based covariates for GATE estimation under regression adjustment and shows that it is possible to simultaneously achieve low bias and considerably reduce variance with such a procedure.

Experimental Design under Network Interference

This paper characterize conditions on the first and second wave experiments to guarantee unconfounded experimentation, showcases tradeoffs in the choice of the pilot's size, and derives asymptotic properties of estimators of interest under the proposed design mechanism and regret guarantees of the proposed method.

Policy design in experiments with unknown interference

An experimental design for estimation and inference on welfaremaximizing policies in the presence of spillover effects in a setting where units are organized into a finite number of large clusters and interact in unknown ways within each cluster is proposed.

Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology

This paper discusses the practice and culture of online experimentation, as well as its statistics literature, and discusses the current methodologies within their relevant statistical lineages, providing illustrative examples of OCE applications.

Representation-Aware Experimentation: Group Inequality Analysis for A/B Testing and Alerting

This work addresses the problem of unintended consequences in experimentation from two perspectives: namely, pre-specified vs. data-driven selection, and a method for simulating data that closely mimics observed data at LinkedIn, and the performance of DER statistics in simulations.

Learning Causal Effects on Hypergraphs

This work investigates high-order interference modeling, and proposes a new causality learning framework powered by hypergraph neural networks, which is verified over existing baselines on real-world hypergraphs.

Treatment Effects in Market Equilibrium

When randomized trials are run in a marketplace equilibriated by prices, interference arises. To analyze this, we build a stochastic model of treatment effects in equilibrium. We characterize the…

Auto-bidding Equilibrium in ROI-Constrained Online Advertising Markets

Most of the work in auction design literature assumes that bidders behave rationally based on the information available for each individual auction. However, in today's online advertising markets,…

Community informed experimental design

Algorithmic transparency and assessing effects of algorithmic ranking

References

SHOWING 1-10 OF 30 REFERENCES

Detecting Network Effects: Randomizing Over Randomized Experiments

A new experimental design is leverage for testing whether SUTVA holds, without making any assumptions on how treatment effects may spill over between the treatment and the control group, and the proposed methodology can be applied to settings in which a network is not necessarily observed but, if available, can be used in the analysis.

Network A/B Testing: From Sampling to Estimation

The existence of network effect is examined in a recent online experiment conducted at LinkedIn, an efficient and effective estimator for Average Treatment Effect (ATE) considering the interference between users in real online experiments is proposed, and the method is applied in both simulations and a real world online experiment.

Graph cluster randomization: network exposure to multiple universes

It is shown that proper cluster randomization can lead to exponentially lower estimator variance when experimentally measuring average treatment effects under interference, and if a graph satisfies a restricted-growth condition on the growth rate of neighborhoods, then there exists a natural clustering algorithm, based on vertex neighborhoods, for which the variance of the estimator can be upper bounded by a linear function of the degrees.

From Infrastructure to Culture: A/B Testing Challenges in Large Scale Social Networks

The experimentation platform at LinkedIn is described in depth and how it is built to handle each step of the A/B testing process at LinkedIn, from designing and deploying experiments to analyzing them.

Randomized Experimental Design via Geographic Clustering

This paper presents GeoCUTS, a novel algorithm that forms geographical clusters to minimize interference while preserving balance in cluster size, and uses a random sample of anonymized traffic from Google Search to form a graph representing user movements, then constructs a geographically coherent clustering of the graph.

Estimating average causal effects under general interference, with application to a social network experiment

This paper develops the case of estimating average unit-level causal effects from a randomized experiment with interference of arbitrary but known form and provides randomization-based variance estimators that account for the complex clustering that can occur when interference is present.

Exact p-Values for Network Interference

ABSTRACT We study the calculation of exact p-values for a large class of nonsharp null hypotheses about treatment effects in a setting with data from experiments involving members of a single…

Testing for arbitrary interference on experimentation platforms

An experimental design strategy for testing whether the classic assumption of no interference among users, under which the outcome of one user does not depend on the treatment assigned to other users, is rarely tenable on such platforms is introduced.

Improving the sensitivity of online controlled experiments by utilizing pre-experiment data

This work proposes an approach (CUPED) that utilizes data from the pre-experiment period to reduce metric variability and hence achieve better sensitivity in experiments, applicable to a wide variety of key business metrics.

Finding and evaluating community structure in networks.

  • M. NewmanM. Girvan
  • Computer Science
    Physical review. E, Statistical, nonlinear, and soft matter physics
  • 2004
It is demonstrated that the algorithms proposed are highly effective at discovering community structure in both computer-generated and real-world network data, and can be used to shed light on the sometimes dauntingly complex structure of networked systems.