Targeting prospective customers: Robustness of machine-learning methods to typical data challenges

Duncan Simester*, Artem Timoshenko, Spyros I. Zoumpoulis

*Corresponding author for this work

Research output: Contribution to journalArticle

1 Scopus citations

Abstract

We investigate how firms can use the results of field experiments to optimize the targeting of promotions when prospecting for new customers. We evaluate seven widely used machine-learning methods using a series of two large-scale field experiments. The first field experiment generates a common pool of training data for each of the seven methods. We then validate the seven optimized policies provided by each method together with uniform benchmark policies in a second field experiment. The findings not only compare the performance of the targeting methods, but also demonstrate how well the methods address common data challenges. Our results reveal that when the training data are ideal, model-driven methods perform better than distance-driven methods and classification methods. However, the performance advantage vanishes in the presence of challenges that affect the quality of the training data, including the extent to which the training data captures details of the implementation setting. The challenges we study are covariate shift, concept shift, information loss through aggregation, and imbalanced data. Intuitively, the model-driven methods make better use of the information available in the training data, but the performance of these methods is more sensitive to deterioration in the quality of this information. The classification methods we tested performed relatively poorly. We explain the poor performance of the classification methods in our setting and describe how the performance of these methods could be improved.

Original languageEnglish (US)
Pages (from-to)2495-2522
Number of pages28
JournalManagement Science
Volume66
Issue number6
DOIs
StatePublished - Jun 2020

Keywords

  • Concept shift
  • Covariate shift
  • Field experiments
  • Machine learning
  • Targeting

ASJC Scopus subject areas

  • Strategy and Management
  • Management Science and Operations Research

Fingerprint Dive into the research topics of 'Targeting prospective customers: Robustness of machine-learning methods to typical data challenges'. Together they form a unique fingerprint.

  • Cite this