TY - JOUR
T1 - How bias reduction is affected by covariate choice, unreliability, and mode of data analysis
T2 - Results from two types of within-study comparisons
AU - Cook, Thomas D.
AU - Steiner, Peter M.
AU - Pohl, Steffi
N1 - Funding Information:
Thomas D. Cook and Peter M. Steiner were supported in part by Grant R305U070003 from the Institute for Educational Sciences, U.S. Department of Education. Peter M. Steiner was also supported by grants from the W. T. Grant Foundation and Spencer Foundation. Thanks to David Kenny for a critical and helpful reading.
PY - 2009
Y1 - 2009
N2 - This study uses within-study comparisons to assess the relative importance of covariate choice, unreliability in the measurement of these covariates, and whether regression or various forms of propensity score analysis are used to analyze the outcome data. Two of the within-study comparisons are of the four-arm type, and many more are of the three-arm type. To examine unreliability, simulations of differences in reliability are deliberately introduced into the 2 four-arm studies. Results are similar across the samples of studies reviewed with their wide range of non-experimental designs and topic areas. Covariate choice counts most, unreliability next most, and the mode of data analysis hardly matters at all. Unreliability has larger effects the more important a covariate is for bias reduction, but even so the very best covariates measured with a reliability of only. 60 still do better than substantively poor covariates that are measured perfectly. Why regression methods do as well as propensity score methods used in several different ways is a mystery still because, in theory, propensity scores would seem to have a distinct advantage in many practical applications, especially those where functional forms are in doubt.
AB - This study uses within-study comparisons to assess the relative importance of covariate choice, unreliability in the measurement of these covariates, and whether regression or various forms of propensity score analysis are used to analyze the outcome data. Two of the within-study comparisons are of the four-arm type, and many more are of the three-arm type. To examine unreliability, simulations of differences in reliability are deliberately introduced into the 2 four-arm studies. Results are similar across the samples of studies reviewed with their wide range of non-experimental designs and topic areas. Covariate choice counts most, unreliability next most, and the mode of data analysis hardly matters at all. Unreliability has larger effects the more important a covariate is for bias reduction, but even so the very best covariates measured with a reliability of only. 60 still do better than substantively poor covariates that are measured perfectly. Why regression methods do as well as propensity score methods used in several different ways is a mystery still because, in theory, propensity scores would seem to have a distinct advantage in many practical applications, especially those where functional forms are in doubt.
UR - http://www.scopus.com/inward/record.url?scp=77951656304&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=77951656304&partnerID=8YFLogxK
U2 - 10.1080/00273170903333673
DO - 10.1080/00273170903333673
M3 - Review article
C2 - 26801798
AN - SCOPUS:77951656304
VL - 44
SP - 828
EP - 847
JO - Multivariate Behavioral Research
JF - Multivariate Behavioral Research
SN - 0027-3171
IS - 6
ER -