Comparison of Outlier Identification Methods in Hospital Surgical Quality Improvement Programs

Karl Y. Bilimoria, Mark E. Cohen, Ryan P. Merkow, Xue Wang, David J. Bentrem, Angela M. Ingraham, Karen Richards, Bruce L. Hall, Clifford Y. Ko

Research output: Contribution to journalArticle

14 Citations (Scopus)

Abstract

Background: Surgeons and hospitals are being increasingly assessed by third parties regarding surgical quality and outcomes, and much of this information is reported publicly. Our objective was to compare various methods used to classify hospitals as outliers in established surgical quality assessment programs by applying each approach to a single data set. Methods: Using American College of Surgeons National Surgical Quality Improvement Program data (7/2008-6/2009), hospital risk-adjusted 30-day morbidity and mortality were assessed for general surgery at 231 hospitals (cases = 217,630) and for colorectal surgery at 109 hospitals (cases = 17,251). The number of outliers (poor performers) identified using different methods and criteria were compared. Results: The overall morbidity was 10.3% for general surgery and 25.3% for colorectal surgery. The mortality was 1.6% for general surgery and 4.0% for colorectal surgery. Programs used different methods (logistic regression, hierarchical modeling, partitioning) and criteria (P<0.01, P<0.05, P<0.10) to identify outliers. Depending on outlier identification methods and criteria employed, when each approach was applied to this single dataset, the number of outliers ranged from 7 to 57 hospitals for general surgery morbidity, 1 to 57 hospitals for general surgery mortality, 4 to 27 hospitals for colorectal morbidity, and 0 to 27 hospitals for colorectal mortality. Conclusions: There was considerable variation in the number of outliers identified using different detection approaches. Quality programs seem to be utilizing outlier identification methods contrary to what might be expected, thus they should justify their methodology based on the intent of the program (i.e., quality improvement vs. reimbursement). Surgeons and hospitals should be aware of variability in methods used to assess their performance as these outlier designations will likely have referral and reimbursement consequences.

Original languageEnglish (US)
Pages (from-to)1600-1607
Number of pages8
JournalJournal of Gastrointestinal Surgery
Volume14
Issue number10
DOIs
StatePublished - Sep 8 2010

Fingerprint

Quality Improvement
Colorectal Surgery
Morbidity
Mortality
Hospital Mortality
Referral and Consultation
Logistic Models

Keywords

  • Complications
  • Deaths
  • Hospital
  • Morbidity
  • Mortality
  • National surgery quality improvement program
  • Outlier
  • Quality
  • Surgery

ASJC Scopus subject areas

  • Surgery
  • Gastroenterology

Cite this

Bilimoria, Karl Y. ; Cohen, Mark E. ; Merkow, Ryan P. ; Wang, Xue ; Bentrem, David J. ; Ingraham, Angela M. ; Richards, Karen ; Hall, Bruce L. ; Ko, Clifford Y. / Comparison of Outlier Identification Methods in Hospital Surgical Quality Improvement Programs. In: Journal of Gastrointestinal Surgery. 2010 ; Vol. 14, No. 10. pp. 1600-1607.
@article{66d92aec6034443d8694846ea8ea89cd,
title = "Comparison of Outlier Identification Methods in Hospital Surgical Quality Improvement Programs",
abstract = "Background: Surgeons and hospitals are being increasingly assessed by third parties regarding surgical quality and outcomes, and much of this information is reported publicly. Our objective was to compare various methods used to classify hospitals as outliers in established surgical quality assessment programs by applying each approach to a single data set. Methods: Using American College of Surgeons National Surgical Quality Improvement Program data (7/2008-6/2009), hospital risk-adjusted 30-day morbidity and mortality were assessed for general surgery at 231 hospitals (cases = 217,630) and for colorectal surgery at 109 hospitals (cases = 17,251). The number of outliers (poor performers) identified using different methods and criteria were compared. Results: The overall morbidity was 10.3{\%} for general surgery and 25.3{\%} for colorectal surgery. The mortality was 1.6{\%} for general surgery and 4.0{\%} for colorectal surgery. Programs used different methods (logistic regression, hierarchical modeling, partitioning) and criteria (P<0.01, P<0.05, P<0.10) to identify outliers. Depending on outlier identification methods and criteria employed, when each approach was applied to this single dataset, the number of outliers ranged from 7 to 57 hospitals for general surgery morbidity, 1 to 57 hospitals for general surgery mortality, 4 to 27 hospitals for colorectal morbidity, and 0 to 27 hospitals for colorectal mortality. Conclusions: There was considerable variation in the number of outliers identified using different detection approaches. Quality programs seem to be utilizing outlier identification methods contrary to what might be expected, thus they should justify their methodology based on the intent of the program (i.e., quality improvement vs. reimbursement). Surgeons and hospitals should be aware of variability in methods used to assess their performance as these outlier designations will likely have referral and reimbursement consequences.",
keywords = "Complications, Deaths, Hospital, Morbidity, Mortality, National surgery quality improvement program, Outlier, Quality, Surgery",
author = "Bilimoria, {Karl Y.} and Cohen, {Mark E.} and Merkow, {Ryan P.} and Xue Wang and Bentrem, {David J.} and Ingraham, {Angela M.} and Karen Richards and Hall, {Bruce L.} and Ko, {Clifford Y.}",
year = "2010",
month = "9",
day = "8",
doi = "10.1007/s11605-010-1316-6",
language = "English (US)",
volume = "14",
pages = "1600--1607",
journal = "Journal of Gastrointestinal Surgery",
issn = "1091-255X",
publisher = "Springer New York",
number = "10",

}

Comparison of Outlier Identification Methods in Hospital Surgical Quality Improvement Programs. / Bilimoria, Karl Y.; Cohen, Mark E.; Merkow, Ryan P.; Wang, Xue; Bentrem, David J.; Ingraham, Angela M.; Richards, Karen; Hall, Bruce L.; Ko, Clifford Y.

In: Journal of Gastrointestinal Surgery, Vol. 14, No. 10, 08.09.2010, p. 1600-1607.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Comparison of Outlier Identification Methods in Hospital Surgical Quality Improvement Programs

AU - Bilimoria, Karl Y.

AU - Cohen, Mark E.

AU - Merkow, Ryan P.

AU - Wang, Xue

AU - Bentrem, David J.

AU - Ingraham, Angela M.

AU - Richards, Karen

AU - Hall, Bruce L.

AU - Ko, Clifford Y.

PY - 2010/9/8

Y1 - 2010/9/8

N2 - Background: Surgeons and hospitals are being increasingly assessed by third parties regarding surgical quality and outcomes, and much of this information is reported publicly. Our objective was to compare various methods used to classify hospitals as outliers in established surgical quality assessment programs by applying each approach to a single data set. Methods: Using American College of Surgeons National Surgical Quality Improvement Program data (7/2008-6/2009), hospital risk-adjusted 30-day morbidity and mortality were assessed for general surgery at 231 hospitals (cases = 217,630) and for colorectal surgery at 109 hospitals (cases = 17,251). The number of outliers (poor performers) identified using different methods and criteria were compared. Results: The overall morbidity was 10.3% for general surgery and 25.3% for colorectal surgery. The mortality was 1.6% for general surgery and 4.0% for colorectal surgery. Programs used different methods (logistic regression, hierarchical modeling, partitioning) and criteria (P<0.01, P<0.05, P<0.10) to identify outliers. Depending on outlier identification methods and criteria employed, when each approach was applied to this single dataset, the number of outliers ranged from 7 to 57 hospitals for general surgery morbidity, 1 to 57 hospitals for general surgery mortality, 4 to 27 hospitals for colorectal morbidity, and 0 to 27 hospitals for colorectal mortality. Conclusions: There was considerable variation in the number of outliers identified using different detection approaches. Quality programs seem to be utilizing outlier identification methods contrary to what might be expected, thus they should justify their methodology based on the intent of the program (i.e., quality improvement vs. reimbursement). Surgeons and hospitals should be aware of variability in methods used to assess their performance as these outlier designations will likely have referral and reimbursement consequences.

AB - Background: Surgeons and hospitals are being increasingly assessed by third parties regarding surgical quality and outcomes, and much of this information is reported publicly. Our objective was to compare various methods used to classify hospitals as outliers in established surgical quality assessment programs by applying each approach to a single data set. Methods: Using American College of Surgeons National Surgical Quality Improvement Program data (7/2008-6/2009), hospital risk-adjusted 30-day morbidity and mortality were assessed for general surgery at 231 hospitals (cases = 217,630) and for colorectal surgery at 109 hospitals (cases = 17,251). The number of outliers (poor performers) identified using different methods and criteria were compared. Results: The overall morbidity was 10.3% for general surgery and 25.3% for colorectal surgery. The mortality was 1.6% for general surgery and 4.0% for colorectal surgery. Programs used different methods (logistic regression, hierarchical modeling, partitioning) and criteria (P<0.01, P<0.05, P<0.10) to identify outliers. Depending on outlier identification methods and criteria employed, when each approach was applied to this single dataset, the number of outliers ranged from 7 to 57 hospitals for general surgery morbidity, 1 to 57 hospitals for general surgery mortality, 4 to 27 hospitals for colorectal morbidity, and 0 to 27 hospitals for colorectal mortality. Conclusions: There was considerable variation in the number of outliers identified using different detection approaches. Quality programs seem to be utilizing outlier identification methods contrary to what might be expected, thus they should justify their methodology based on the intent of the program (i.e., quality improvement vs. reimbursement). Surgeons and hospitals should be aware of variability in methods used to assess their performance as these outlier designations will likely have referral and reimbursement consequences.

KW - Complications

KW - Deaths

KW - Hospital

KW - Morbidity

KW - Mortality

KW - National surgery quality improvement program

KW - Outlier

KW - Quality

KW - Surgery

UR - http://www.scopus.com/inward/record.url?scp=77957018168&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77957018168&partnerID=8YFLogxK

U2 - 10.1007/s11605-010-1316-6

DO - 10.1007/s11605-010-1316-6

M3 - Article

C2 - 20824379

AN - SCOPUS:77957018168

VL - 14

SP - 1600

EP - 1607

JO - Journal of Gastrointestinal Surgery

JF - Journal of Gastrointestinal Surgery

SN - 1091-255X

IS - 10

ER -