When Less is More: Data and Power in Advertising Experiments

  title={When Less is More: Data and Power in Advertising Experiments},
  author={Garrett A. Johnson and Randall A. Lewis and David H. Reiley},
  journal={Innovation \& Management Science eJournal},
Yahoo! Research partnered with a nationwide retailer to study the effects of online display advertising on both online and in-store purchases. We use a randomized field experiment on 3 million Yahoo! users who are also past customers of the retailer. We find statistically significant evidence that the retailer ads increase sales 3.6% relative to the control group. We show that control ads boost measurement precision by identifying and removing the half of in-campaign sales data that is… 

Figures and Tables from this paper

Location, Location, Location: Repetition and Proximity Increase Advertising Effectiveness

We examine a large-scale field experiment that measures the effect of online display advertising. Yahoo! Research and an American apparel retailer collaborated on the experiment that tracked the

Ghost Ads: Improving the Economics of Measuring Online Ad Effectiveness

The authors show that, relative to public service announcement and intent-to-treat A/B tests, ghost ads can reduce the cost of experimentation, improve measurement precision, deliver the relevant strategic baseline, and work with modern ad platforms that optimize ad delivery in real time.

Principal Stratification for Advertising Experiments

Advertising experiments often suffer from noisy responses making precise estimation of the average treatment effect (ATE) and evaluating ROI difficult. We develop a principal stratification model

Increasing Mobile Search Ad Spend: Cross-device Effects

Quantifying the causal effect of displaying product price in search ads on performance metrics, we partner with an online retailer to run a large-scale field experiment involving multiple products by

Are Consumers Averse to Sponsored Messages? The Role of Search Advertising in Information Discovery

We analyze a large-scale randomized field experiment in which a search engine varied the prominence of search ads for 3.3 million US users: one group of users saw the status quo, while the other saw

An Experimental Investigation of the Effects of Retargeted Advertising: The Role of Frequency and Timing

In collaboration with an online seller of home-improvement products, the authors conduct a large-scale randomized field experiment to study the effects of retargeted advertising, a form of internet advertising that targets specific consumers with a specific brand.

Inferno: A Guide to Field Experiments in Online Display Advertising

In this guide, challenges and best practices in online display ads are reviewed to place renewed focus on old topics like statistical power and compliance as well as on new issues like identity fragmentation and market spillovers.

Incentive Misalignments in Programmatic Advertising: Evidence from a Randomized Field Experiment

In programmatic advertising, firms outsource the bidding for ad impressions to ad platforms. Although firms are interested in targeting consumers that respond positively to advertising, ad platforms

From Mad Men to Maths Men: Concentration and Buyer Power in Online Advertising

This paper analyzes the impact of intermediary concentration on the allocation of revenue in online platforms. We study sponsored search documenting how advertisers increasingly bid through a handful



Online ads and offline sales: measuring the effect of retail advertising via a controlled experiment on Yahoo!

A randomized experiment with 1.6 million customers measures positive causal effects of online advertising for a major retailer. The advertising profitably increases purchases by 5%. 93% of the

The Unfavorable Economics of Measuring the Returns to Advertising

Twenty-five large field experiments with major U.S. retailers and brokerages, most reaching millions of customers and collectively representing $2.8 million in digital advertising expenditure, reveal

Ghost Ads: Improving the Economics of Measuring Ad Effectiveness

To measure the effects of advertising, marketers must know how consumers would behave had they not seen the ads. We develop a methodology we call ‘Ghost Ads,’ which facilitates this comparison by

Dynamics of Retail Advertising: Evidence from a Field Experiment

"We use a controlled field experiment to investigate the dynamic effects of retail advertising. The experimental design overcomes limitations hindering previous investigations of this issue. Our

Here, there, and everywhere: correlated online behaviors can lead to overestimates of the effects of advertising

Using three controlled experiments, it is shown that observational data frequently lead to incorrect estimates of adFX, and how and why observational methods lead to a massive overestimate of adfx in such circumstances.

Advertising Spillovers: Evidence from Online Field Experiments and Implications for Returns on Advertising

The author analyzes the impact of online ads on the advertiser's competitors, using data from randomized field experiments on a restaurant-search website. He finds that ads increase the chances of

Online Display Advertising: Targeting and Obtrusiveness

It is found that matching an ad to website content and increasing an ad's obtrusiveness independently increase purchase intent, however, in combination, these two strategies are ineffective.

Advertising Experiments at the Campbell Soup Company

In the mid 1970's, 19 controlled experiments in the marketplace were conducted by the Campbell Soup Company to evaluate the sales impact of advertising changes for various well established packaged

Effects of Internet Display Advertising in the Purchase Funnel: Model-Based Insights from a Randomized Field Experiment

This study examines the effects of Internet display advertising using cookie-level data from a field experiment at a financial tools provider. The experiment randomized assignment of cookies to

How T.V. Advertising Works: A Meta-Analysis of 389 Real World Split Cable T.V. Advertising Experiments

The authors analyze results of 389 BehaviorScan® matched household, consumer panel, split cable, real world T.V. advertising weight, and copy tests. Additionally, study sponsors—packaged goods