Reliable and Computationally Efficient Maximum-Likelihood Estimation of "Proper" Binormal ROC Curves

Lorenzo L. Pesce, Charles E. Metz*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

106 Scopus citations

Abstract

Rationale and Objectives: Estimation of ROC curves and their associated indices from experimental data can be problematic, especially in multireader, multicase (MRMC) observer studies. Wilcoxon estimates of area under the curve (AUC) can be strongly biased with categorical data, whereas the conventional binormal ROC curve-fitting model may produce unrealistic fits. The "proper" binormal model (PBM) was introduced by Metz and Pan to provide acceptable fits for both sturdy and problematic datasets, but other investigators found that its first software implementation was numerically unstable in some situations. Therefore, we created an entirely new algorithm to implement the PBM. Materials and Methods: This paper describes in detail the new PBM curve-fitting algorithm, which was designed to perform successfully in all problematic situations encountered previously. Extensive testing was conducted also on a broad variety of simulated and real datasets. Windows, Linux, and Apple Macintosh OS X versions of the algorithm are available online at http://xray.bsd.uchicago.edu/krl/. Results: Plots of fitted curves as well as summaries of AUC estimates and their standard errors are reported. The new algorithm never failed to converge and produced good fits for all of the several million datasets on which it was tested. For all but the most problematic datasets, the algorithm also produced very good estimates of AUC standard error. The AUC estimates compared well with Wilcoxon estimates for continuously distributed data and are expected to be superior for categorical data. Conclusion: This implementation of the PBM is reliable in a wide variety of ROC curve-fitting tasks.

Original languageEnglish (US)
Pages (from-to)814-829
Number of pages16
JournalAcademic radiology
Volume14
Issue number7
DOIs
StatePublished - Jul 2007

Keywords

  • Receiver operating characteristic (ROC) analysis
  • maximum likelihood estimation (MLE)
  • multireader, multicase (MRMC) analysis
  • proper binormal model
  • receiver operating characteristic (ROC) curves

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging

Fingerprint

Dive into the research topics of 'Reliable and Computationally Efficient Maximum-Likelihood Estimation of "Proper" Binormal ROC Curves'. Together they form a unique fingerprint.

Cite this