Journal of Pathology Informatics

TECHNICAL NOTE
Year
: 2021  |  Volume : 12  |  Issue : 1  |  Page : 45-

A pathologist-annotated dataset for validating artificial intelligence: A project description and pilot study


Sarah N Dudgeon1, Si Wen1, Matthew G Hanna2, Rajarsi Gupta3, Mohamed Amgad4, Manasi Sheth5, Hetal Marble6, Richard Huang6, Markus D Herrmann6, Clifford H Szu7, Darick Tong7, Bruce Werness7, Evan Szu7, Denis Larsimont8, Anant Madabhushi9, Evangelos Hytopoulos10, Weijie Chen1, Rajendra Singh11, Steven N Hart6, Ashish Sharma12, Joel Saltz3, Roberto Salgado13, Brandon D Gallas1 
1 Division of Imaging Diagnostics and Software Reliability, Office of Science and Engineering Laboratories, Center for Devices and Radiologic Health, United States Food and Drug Administration, White Oak, MD, USA
2 Memorial Sloan Kettering Cancer Center, New York, NY, USA
3 Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
4 Department of Pathology, Northwestern University, Chicago, IL, USA
5 Division of Biostatistics, Center for Devices and Radiologic Health, United States Food and Drug Administration, White Oak, MD, USA
6 Department of Pathology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA
7 Arrive Bio, San Francisco, CA, USA
8 Department of Pathology, Institute Jules Bordet, Brussels, Belgium
9 Louis Stokes Cleveland Veterans Administration Medical Center, Cleveland, OH, USA
10 iRhythm Technologies Inc., San Francisco, CA, USA
11 Northwell Health and Zucker School of Medicine, New York, NY, USA
12 Department of Biomedical Informatics, Emory University, Atlanta, GA, USA
13 Division of Research, Peter Mac Callum Cancer Centre, Melbourne, Australia; Department of Pathology, GZA-ZNA Hospitals, Antwerp, Belgium

Correspondence Address:
Dr. Brandon D Gallas
Division of Imaging, Diagnostics, and Software Reliability, Office of Science and Engineering Laboratories, Center for Devices and Radiological Health, US Food and Drug Administration, 10903 New Hampshire Ave, Silver Spring, Rm 4104, White Oke-62, MD 20993
USA

Purpose: Validating artificial intelligence algorithms for clinical use in medical images is a challenging endeavor due to a lack of standard reference data (ground truth). This topic typically occupies a small portion of the discussion in research papers since most of the efforts are focused on developing novel algorithms. In this work, we present a collaboration to create a validation dataset of pathologist annotations for algorithms that process whole slide images. We focus on data collection and evaluation of algorithm performance in the context of estimating the density of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer. Methods: We digitized 64 glass slides of hematoxylin- and eosin-stained invasive ductal carcinoma core biopsies prepared at a single clinical site. A collaborating pathologist selected 10 regions of interest (ROIs) per slide for evaluation. We created training materials and workflows to crowdsource pathologist image annotations on two modes: an optical microscope and two digital platforms. The microscope platform allows the same ROIs to be evaluated in both modes. The workflows collect the ROI type, a decision on whether the ROI is appropriate for estimating the density of sTILs, and if appropriate, the sTIL density value for that ROI. Results: In total, 19 pathologists made 1645 ROI evaluations during a data collection event and the following 2 weeks. The pilot study yielded an abundant number of cases with nominal sTIL infiltration. Furthermore, we found that the sTIL densities are correlated within a case, and there is notable pathologist variability. Consequently, we outline plans to improve our ROI and case sampling methods. We also outline statistical methods to account for ROI correlations within a case and pathologist variability when validating an algorithm. Conclusion: We have built workflows for efficient data collection and tested them in a pilot study. As we prepare for pivotal studies, we will investigate methods to use the dataset as an external validation tool for algorithms. We will also consider what it will take for the dataset to be fit for a regulatory purpose: study size, patient population, and pathologist training and qualifications. To this end, we will elicit feedback from the Food and Drug Administration via the Medical Device Development Tool program and from the broader digital pathology and AI community. Ultimately, we intend to share the dataset, statistical methods, and lessons learned.


How to cite this article:
Dudgeon SN, Wen S, Hanna MG, Gupta R, Amgad M, Sheth M, Marble H, Huang R, Herrmann MD, Szu CH, Tong D, Werness B, Szu E, Larsimont D, Madabhushi A, Hytopoulos E, Chen W, Singh R, Hart SN, Sharma A, Saltz J, Salgado R, Gallas BD. A pathologist-annotated dataset for validating artificial intelligence: A project description and pilot study.J Pathol Inform 2021;12:45-45


How to cite this URL:
Dudgeon SN, Wen S, Hanna MG, Gupta R, Amgad M, Sheth M, Marble H, Huang R, Herrmann MD, Szu CH, Tong D, Werness B, Szu E, Larsimont D, Madabhushi A, Hytopoulos E, Chen W, Singh R, Hart SN, Sharma A, Saltz J, Salgado R, Gallas BD. A pathologist-annotated dataset for validating artificial intelligence: A project description and pilot study. J Pathol Inform [serial online] 2021 [cited 2021 Nov 29 ];12:45-45
Available from: https://www.jpathinformatics.org/article.asp?issn=2153-3539;year=2021;volume=12;issue=1;spage=45;epage=45;aulast=Dudgeon;type=0