Performance-based variable selection for scoring models

The performance of a direct marketing scoring model at a particular mailing depth, d, is usually measured by the total amount of revenue generated by sending an offer to the customers with the 100d% largest scores (predicted values). Commonly used variable selection algorithms optimize some function of model fit (squared difference between training and predicted values). This article (1) discusses issues involved in selecting a mailing depth, d, and (2) proposes a variable selection algorithm that optimizes the performance as the primary objective. The relationship between fit and performance is discussed. The performance-based algorithm is compared with fit-based algorithms using two real direct marketing data sets. These experimental results indicate that performance-based variable selection is 3-4% better than corresponding fit-based models, on average, when the mailing depth is between 20% and 40%.