Predicting the Performance of Measures in a Confirmatory Factor Analysis With a Pretest Assessment of Their Substantive Validities

A pretest methodology for predicting the performance of measures in a confirmatory factor analysis is presented. A pretest item-sort task draws on the concept of substantive validity, and two indices of substantive validity in a theory-testing context are proposed: the proportion of substantive agreement, P_sa, and the substantive-validity coefficient, c_sv. The utility of this method is empirically illustrated with a postdictive study of impulsivity measures. Results from two pretest samples of 20 respondents provided significant support for the use of substantive-validity coefficient values to discriminate measures that would be retained in a subsequent confirmatory factor analysis from those that would not. In addition, significant evidence was found for the reproducibility of each substantive-validity index across the two samples. Issues to be considered when using the pretest methodology and some benefits of assessing the substantive validity of measures for construct definitions and delineation of content domains are discussed.