TY - JOUR
T1 - Using performance standards to evaluate social programs with incomplete outcome data
T2 - General issues and application to a higher education block grant program
AU - Manski, Charles F.
AU - Newman, John
AU - Pepper, John V.
PY - 2002/8
Y1 - 2002/8
N2 - The idea of program evaluation is both simple and appealing. Program outcomes are measured and compared to some minimum performance standard or threshold. In practice, however, evaluation is difficult. Two fundamental problems of outcome measurement must be addressed. The first, which we call the problem of auxiliary outcomes, is that we do not observe outcome of interest. The second, which we call the problem of counterfactual outcomes, is that we do not observe the threshold standard. This article examines how performance standards should be set and applied in the face of these problems in measuring outcomes. The central message is that the proper way to implement standards varies with the prior information an evaluator can credibly bring to bear to compensate for incomplete outcome data. By combining available data with credible assumptions on treatments and outcomes, the performance of a program may be deemed acceptable, unacceptable, or indeterminate.
AB - The idea of program evaluation is both simple and appealing. Program outcomes are measured and compared to some minimum performance standard or threshold. In practice, however, evaluation is difficult. Two fundamental problems of outcome measurement must be addressed. The first, which we call the problem of auxiliary outcomes, is that we do not observe outcome of interest. The second, which we call the problem of counterfactual outcomes, is that we do not observe the threshold standard. This article examines how performance standards should be set and applied in the face of these problems in measuring outcomes. The central message is that the proper way to implement standards varies with the prior information an evaluator can credibly bring to bear to compensate for incomplete outcome data. By combining available data with credible assumptions on treatments and outcomes, the performance of a program may be deemed acceptable, unacceptable, or indeterminate.
UR - http://www.scopus.com/inward/record.url?scp=0036675771&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=0036675771&partnerID=8YFLogxK
U2 - 10.1177/0193841X02026004001
DO - 10.1177/0193841X02026004001
M3 - Article
C2 - 12174536
AN - SCOPUS:0036675771
VL - 26
SP - 355
EP - 381
JO - Evaluation Review
JF - Evaluation Review
SN - 0193-841X
IS - 4
ER -