Joint region and nucleus segmentation for characterization of tumor infiltrating lymphocytes in breast cancer

Mohamed Amgad, Anindya Sarkar, Chukka Srinivas, Rachel Redman, Simrath Ratra, Charles J. Bechert, Benjamin C. Calhoun, Karen Mrazeck, Uday Kurkure, Lee Alex Donald Cooper*, Michael Barnes

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Scopus citations


Histologic assessment of stromal tumor infiltrating lymphocytes (sTIL) as a surrogate of the host immune response has been shown to be prognostic and potentially chemo-predictive in triple-negative and HER2-positive breast cancers. The current practice of manual assessment is prone to intra-and inter-observer variability. Furthermore, the inter-play of sTILs, tumor cells, other microenvironment mediators, their spatial relationships, quantity, and other image-based features have yet to be determined exhaustively and systemically. Towards analysis of these aspects, we developed a deep learning based method for joint region-level and nucleus-level segmentation and classification of breast cancer H&E tissue whole slide images. Our proposed method simultaneously identifies tumor, fibroblast, and lymphocyte nuclei, along with key histologic region compartments including tumor and stroma. We also show how the resultant segmentation masks can be combined with seeding approaches to yield accurate nucleus classifications. Furthermore, we outline a simple workflow for calibrating computational scores to human scores for consistency. The pipeline identifies key compartments with high accuracy (Dice= overall: 0.78, tumor: 0.83, and fibroblasts: 0.77). ROC AUC for nucleus classification is high at 0.89 (micro-average), 0.89 (lymphocytes), 0.90 (tumor), and 0.78 (fibroblasts). Spearman correlation between computational sTIL and pathologist consensus is high (R=0.73, p<0.001) and is higher than inter-pathologist correlation (R=0.66, p<0.001). Both manual and computational sTIL scores successfully stratify patients by clinical progression outcomes.

Original languageEnglish (US)
Title of host publicationMedical Imaging 2019
Subtitle of host publicationDigital Pathology
EditorsJohn E. Tomaszewski, Aaron D. Ward
ISBN (Electronic)9781510625594
StatePublished - 2019
EventMedical Imaging 2019: Digital Pathology - San Diego, United States
Duration: Feb 20 2019Feb 21 2019

Publication series

NameProgress in Biomedical Optics and Imaging - Proceedings of SPIE
ISSN (Print)1605-7422


ConferenceMedical Imaging 2019: Digital Pathology
CountryUnited States
CitySan Diego


  • Computational pathology
  • Convolutional networks
  • Deep learning
  • Tumor infiltrating lymphocytes

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Biomaterials
  • Atomic and Molecular Physics, and Optics
  • Radiology Nuclear Medicine and imaging

Fingerprint Dive into the research topics of 'Joint region and nucleus segmentation for characterization of tumor infiltrating lymphocytes in breast cancer'. Together they form a unique fingerprint.

Cite this