A comparison of approaches to advertising measurement: Evidence from big field experiments at facebook

Brett R Gordon, Florian Zettelmeyer, Neha Bhargava, Dan Chapsky

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

Measuring the causal effects of digital advertising remains challenging despite the availability of granular data. Unobservable factors make exposure endogenous, and advertising’s effect on outcomes tends to be small. In principle, these concerns could be addressed using randomized controlled trials (RCTs). In practice, few online ad campaigns rely on RCTs and instead use observational methods to estimate ad effects. We assess empirically whether the variation in data typically available in the advertising industry enables observational methods to recover the causal effects of online advertising. Using data from 15 U.S. advertising experiments at Facebook comprising 500 million userexperiment observations and 1.6 billion ad impressions, we contrast the experimental results to those obtained from multiple observational models. The observational methods often fail to produce the same effects as the randomized experiments, even after conditioning on extensive demographic and behavioral variables. In our setting, advances in causal inference methods do not allow us to isolate the exogenous variation needed to estimate the treatment effects. We also characterize the incremental explanatory power our data would require to enable observational methods to successfully measure advertising effects. Our findings suggest that commonly used observational approaches based on the data usually available in the industry often fail to accurately measure the true effect of advertising.

Original languageEnglish (US)
Pages (from-to)193-205
Number of pages13
JournalMarketing Science
Volume38
Issue number2
DOIs
StatePublished - Mar 1 2019

Fingerprint

Field experiment
Facebook
Industry
Randomized controlled trial
Causal effect
Advertising effects
Randomized experiments
Online advertising
Experiment
Treatment effects
Demographics
Incremental
Factors
Conditioning
Causal inference

Keywords

  • Advertising measurement
  • Causal inference
  • Digital advertising
  • Field experiments
  • Observational methods

ASJC Scopus subject areas

  • Business and International Management
  • Marketing

Cite this

@article{f4f85126d4344c7e8a8a7a4ce2dc31bb,
title = "A comparison of approaches to advertising measurement: Evidence from big field experiments at facebook",
abstract = "Measuring the causal effects of digital advertising remains challenging despite the availability of granular data. Unobservable factors make exposure endogenous, and advertising’s effect on outcomes tends to be small. In principle, these concerns could be addressed using randomized controlled trials (RCTs). In practice, few online ad campaigns rely on RCTs and instead use observational methods to estimate ad effects. We assess empirically whether the variation in data typically available in the advertising industry enables observational methods to recover the causal effects of online advertising. Using data from 15 U.S. advertising experiments at Facebook comprising 500 million userexperiment observations and 1.6 billion ad impressions, we contrast the experimental results to those obtained from multiple observational models. The observational methods often fail to produce the same effects as the randomized experiments, even after conditioning on extensive demographic and behavioral variables. In our setting, advances in causal inference methods do not allow us to isolate the exogenous variation needed to estimate the treatment effects. We also characterize the incremental explanatory power our data would require to enable observational methods to successfully measure advertising effects. Our findings suggest that commonly used observational approaches based on the data usually available in the industry often fail to accurately measure the true effect of advertising.",
keywords = "Advertising measurement, Causal inference, Digital advertising, Field experiments, Observational methods",
author = "Gordon, {Brett R} and Florian Zettelmeyer and Neha Bhargava and Dan Chapsky",
year = "2019",
month = "3",
day = "1",
doi = "10.1287/mksc.2018.1135",
language = "English (US)",
volume = "38",
pages = "193--205",
journal = "Marketing Science",
issn = "0732-2399",
publisher = "INFORMS Inst.for Operations Res.and the Management Sciences",
number = "2",

}

A comparison of approaches to advertising measurement : Evidence from big field experiments at facebook. / Gordon, Brett R; Zettelmeyer, Florian; Bhargava, Neha; Chapsky, Dan.

In: Marketing Science, Vol. 38, No. 2, 01.03.2019, p. 193-205.

Research output: Contribution to journalArticle

TY - JOUR

T1 - A comparison of approaches to advertising measurement

T2 - Evidence from big field experiments at facebook

AU - Gordon, Brett R

AU - Zettelmeyer, Florian

AU - Bhargava, Neha

AU - Chapsky, Dan

PY - 2019/3/1

Y1 - 2019/3/1

N2 - Measuring the causal effects of digital advertising remains challenging despite the availability of granular data. Unobservable factors make exposure endogenous, and advertising’s effect on outcomes tends to be small. In principle, these concerns could be addressed using randomized controlled trials (RCTs). In practice, few online ad campaigns rely on RCTs and instead use observational methods to estimate ad effects. We assess empirically whether the variation in data typically available in the advertising industry enables observational methods to recover the causal effects of online advertising. Using data from 15 U.S. advertising experiments at Facebook comprising 500 million userexperiment observations and 1.6 billion ad impressions, we contrast the experimental results to those obtained from multiple observational models. The observational methods often fail to produce the same effects as the randomized experiments, even after conditioning on extensive demographic and behavioral variables. In our setting, advances in causal inference methods do not allow us to isolate the exogenous variation needed to estimate the treatment effects. We also characterize the incremental explanatory power our data would require to enable observational methods to successfully measure advertising effects. Our findings suggest that commonly used observational approaches based on the data usually available in the industry often fail to accurately measure the true effect of advertising.

AB - Measuring the causal effects of digital advertising remains challenging despite the availability of granular data. Unobservable factors make exposure endogenous, and advertising’s effect on outcomes tends to be small. In principle, these concerns could be addressed using randomized controlled trials (RCTs). In practice, few online ad campaigns rely on RCTs and instead use observational methods to estimate ad effects. We assess empirically whether the variation in data typically available in the advertising industry enables observational methods to recover the causal effects of online advertising. Using data from 15 U.S. advertising experiments at Facebook comprising 500 million userexperiment observations and 1.6 billion ad impressions, we contrast the experimental results to those obtained from multiple observational models. The observational methods often fail to produce the same effects as the randomized experiments, even after conditioning on extensive demographic and behavioral variables. In our setting, advances in causal inference methods do not allow us to isolate the exogenous variation needed to estimate the treatment effects. We also characterize the incremental explanatory power our data would require to enable observational methods to successfully measure advertising effects. Our findings suggest that commonly used observational approaches based on the data usually available in the industry often fail to accurately measure the true effect of advertising.

KW - Advertising measurement

KW - Causal inference

KW - Digital advertising

KW - Field experiments

KW - Observational methods

UR - http://www.scopus.com/inward/record.url?scp=85065961167&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065961167&partnerID=8YFLogxK

U2 - 10.1287/mksc.2018.1135

DO - 10.1287/mksc.2018.1135

M3 - Article

AN - SCOPUS:85065961167

VL - 38

SP - 193

EP - 205

JO - Marketing Science

JF - Marketing Science

SN - 0732-2399

IS - 2

ER -