The international workshop on osteoarthritis imaging knee mri segmentation challenge: A multi-institute evaluation and analysis framework on a standardized dataset

IWOAI Segmentation Challenge Writing Group

Research output: Contribution to journalArticlepeer-review

12 Scopus citations

Abstract

Purpose: To organize a multi-institute knee MRI segmentation challenge for characterizing the semantic and clinical efficacy of automatic segmentation methods relevant for monitoring osteoarthritis progression. Materials and Methods: A dataset partition consisting of three-dimensional knee MRI from 88 retrospective patients at two time points (baseline and 1-year follow-up) with ground truth articular (femoral, tibial, and patellar) cartilage and meniscus segmentations was standardized. Challenge submissions and a majority-vote ensemble were evaluated against ground truth segmentations using Dice score, average symmetric surface distance, volumetric overlap error, and coefficient of variation on a holdout test set. Similarities in automated segmentations were measured using pairwise Dice coefficient correlations. Articular cartilage thickness was computed longitudinally and with scans. Correlation between thickness error and segmentation metrics was measured using the Pearson correlation coefficient. Two empirical upper bounds for ensemble performance were computed using combinations of model outputs that consolidated true positives and true negatives. Results: Six teams (T1 –T6) submitted entries for the challenge. No differences were observed across any segmentation metrics for any tissues (P = .99) among the four top-performing networks (T2, T3, T4, T6). Dice coefficient correlations between network pairs were high (> 0.85). Per-scan thickness errors were negligible among networks T1 –T4 (P = .99), and longitudinal changes showed minimal bias (< 0.03 mm). Low correlations (r < 0.41) were observed between segmentation metrics and thickness error. The majority-vote ensemble was comparable to top-performing networks (P = .99). Empirical upper-bound performances were similar for both combinations (P = .99). Conclusion: Diverse networks learned to segment the knee similarly, where high segmentation accuracy did not correlate with cartilage thickness accuracy and voting ensembles did not exceed individual network performance.

Original languageEnglish (US)
Article numbere200078
JournalRadiology: Artificial Intelligence
Volume3
Issue number3
DOIs
StatePublished - Mar 2021

Keywords

  • Cartilage
  • Knee
  • MR-Imaging
  • Segmentation

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging
  • Artificial Intelligence
  • Radiological and Ultrasound Technology

Fingerprint

Dive into the research topics of 'The international workshop on osteoarthritis imaging knee mri segmentation challenge: A multi-institute evaluation and analysis framework on a standardized dataset'. Together they form a unique fingerprint.

Cite this