Journal of Pathology Informatics Journal of Pathology Informatics
Contact us | Home | Login   |  Users Online: 8022  Print this pageEmail this pageSmall font sizeDefault font sizeIncrease font size 




 
Table of Contents    
ORIGINAL ARTICLE
J Pathol Inform 2021,  12:39

Testing of actual scanner performance in a high-loaded UNIM laboratory environment


1 UNIM LLC, Moscow, Russia
2 Department of Pathology, Faculty of Pediatric, Pirogov Russian National Research Medical University (Pirogov Medical University), Moscow, Russia

Date of Submission13-Jan-2021
Date of Decision27-Apr-2021
Date of Acceptance04-Jul-2021
Date of Web Publication01-Nov-2021

Correspondence Address:
Mikhail Yurevich Genis
Podsosenskaya Lane, 23, b. 6., Moscow 105062
Russia
Login to access the Email id

Source of Support: None, Conflict of Interest: None


DOI: 10.4103/jpi.jpi_4_21

Rights and Permissions
   Abstract 


Background: Scanners are the main tool in digital pathology. The technical abilities of scanners determine the workflow logic in the pathology laboratory. Its performance can be restricted by the divergence between the scanning time presented by the manufacturer and the actual scanning time. This could lead to critical deviations from the established business processes in a 24/7 laboratory. Aim: Our investigation is focused in exploring the performance of three main models of high-performance scanners available on the Russian market: 3DHistech, Hamamatsu и Leica. Objectives: We compared the performance of the scanners on the samples of a given size with the manufacturer's stated specifications and evaluated the speed of the scanners on the reference and routine laboratory material. Subjects and Methods: We examined 3DHistech Pannoramic 1000, Hamamatsu NanoZoomer s360 and Leica AT2 with default settings and automatic mode. Two sets of glasses were used (glass slide): Group 1 included 120 slides with 15 mm × 15 mm slices, Group 2 included 120 workflow slides. Results: The average slide scan times in Groups 1 and 2 for the C13220 (156 ± 1.25 s and 117 ± 4.17 s) and Pannoramic 1000 (210 ± 1.64 s and 183 ± 3.78 s) differ statistically significantly (P < 0.0001). Total scanning time including rack reloading was shorter for the workflow slide set group for the modern C13220 and Pannoramic 1000 scanner models. Conclusions: The scanner specifications provided by manufacturers are not sufficient to evaluate the performance. The guidelines and regulations concerning scanner selection should be consented by the digital pathology community. We suggest discussing criteria for evaluating scanner performance.

Keywords: Digital pathology, high-throughput scanner, scanning speed,Whole Slide Images in Pathology.


How to cite this article:
Genis MY, Remez AI, Untesco MI, Zhakota DA. Testing of actual scanner performance in a high-loaded UNIM laboratory environment. J Pathol Inform 2021;12:39

How to cite this URL:
Genis MY, Remez AI, Untesco MI, Zhakota DA. Testing of actual scanner performance in a high-loaded UNIM laboratory environment. J Pathol Inform [serial online] 2021 [cited 2021 Dec 5];12:39. Available from: https://www.jpathinformatics.org/text.asp?2021/12/1/39/329880




   Introduction Top


Successful implementation of whole slide imaging for routine clinical practice and COVID-19 epidemic crisis demonstrates the need to introduce digital technology into pathology for rapid integration into the chain of telemedicine consultations. The area of digital pathology was introduced in the clinical practice quite recently. As a result, the choice the right technical solution in this area is even more urgent. Most guidelines address the topic generally and suggestively, and it needs further investigation.[1],[2],[3],[4],[5] General problem of scanner performance in the workflow was explored by Zarell et al.[2] and García-Rojo.[5]

The UNIM laboratory, founded in 2014, works as a fully digital pathology laboratory. The lab is ISO 15189:2012 certified. During the work process, it was found that the throughput of the scanners differed from the manufacturer's stated capacity. Therefore, it was crucial to evaluate the real capabilities of the equipment. We have had two Aperio AT2 scanners in service 24/7 since 2018. They are serviced once a year. The average equipment utilization rate in our laboratory is 80%. For this reason, we are constantly looking for relevant solutions on the market and regularly request equipment from all manufacturers represented in Russia for testing.


   Subjects and Methods Top


Three models of scanners participated in the study: Aperio AT2, 3DHistech Pannoramic 1000 and Hamamatsu NanoZoomer s360 C13220. It should be noted that these scanners have been on the market for different number of years. To compare them as identical models would be rather inappropriate. This was the reason for simplifying the comparison method.

Scanner settings

Manufacturers equip their scanners with a wide range of settings. In this study, we only used the default settings offered by the manufacturer. We are primarily interested in the performance of the equipment with preconfigured settings to minimize startup and maintenance times during the scanning process.

To equalize the working conditions of the scanners, we chose automatic mode without prescan checks. We were interested not only in the speed of scanning one slide or one rack, but also in the time it takes to rack reloading. All scanners had different rack capacities. We chose the smallest multiple of 120 slides for all scanners. General technical parameters are presented in [Table 1].
Table 1: Basic characteristics of the scan modes

Click here to view


Samples for reference scanning

In the characteristics of scanners manufacturers indicate the scanning speed of one sample of a given size, 15 mm × 15 mm.[6],[7],[8] As a rule, pure scanning time of a tissue sample of this size is indicated. That is, without taking into account focusing, recording time, moving slides in the magazine, changing magazines, etc. These are justified minimum parameters to estimate the scanning speed. However, it is not enough to understand the throughput of high-performance scanners for 24/7 operation. We decided to expand the testing conditions.

Manufacturers do not specify the histological affiliation of the tissue. In our study, we used myocardial tissue. A 15 mm × 15 mm × 5 mm tissue sample from autopsy without cardiac pathology was taken to make Group 1 preparations. This is a sufficiently homogeneous tissue, which is minimally prone to artifacts when making slides. The myocardial sample was fixed in 10% buffered formalin, and the blocks and slides were made according to the standard laboratory process chain. The technological chain included devices Tissue-Tek VIP Jr, Excelsior AS, Leica R2235, Leica BOND, Tissue-Tek Prisma, and Tissue-Tek Film.

For group 1, 120 slides with a slice thickness of 2–3 μ were made. In routine work, there are usually several slices on a single slide. Based on this, we decided to simulate the maximum load when scanning surgical biopsies. For this purpose, we placed two 15 mm × 15 mm slices on each slide. Hematoxylin and eosin staining was used. For the study, we needed native format files of scanner manufacturers. The slides were numbered manually from 1 to 120, because in the technological chain of the laboratory slides with barcode are loaded immediately after scanning to the server with LIS UNIM. Saving the scans in LIS UNIM would have affected the purity of the experiment. The slides were moved between scanners with strict sequencing.

Scanning a 15 mm × 15 mm reference sample

To make a comparison with the manufacturers' stated characteristics, we took one slide with reference slices of 15 mm × 15 mm. We manually selected a 15 mm × 15 mm area [Figure 1]. Afterward, we measured the time between pressing the start button and getting the finished virtual slide for the AT2. For other scanners, we have set the full scanning time according to the manufacturer's software.
Figure 1: Scanner software interface in single area 15 mm × 15 mm scanning mode ([a] - AT2, [b] - C13220, [c] - Panoramic 1000)

Click here to view


Scanning 120 reference slides

In the next step, we studied the scanning of 120 slides loaded at a time with two 15 mm × 15 mm slices. The slides were distributed in racks. Racks were placed sequentially in the scanner. Afterward, the scanning process was started [Figure 2].
Figure 2: Software interface in the automatic scanning mode with two sections 15x15 mm ([a] - AT2, [b] - C13220, [c] - Panoramic 1000). The scanning area tends to the cover glass area.

Click here to view


To calculate the average scanning time of a slide in each group, we took the scanning time from each file for the C13220 and Panoramic 1000 models using the manufacturers' software. Since the AT2 manufacturer's software does not provide this metric, this scanner was excluded from this step. Statistical analysis of the data obtained was performed with Statistica 10.0 software (StatSoft, Tulsa, OK, USA). Data were presented as M ± standard deviation (SD). Significance of differences was assessed by the Student's t-test. The results were considered significant at 95.5% probability (P < 0.05).

The formula was used to calculate the total scanning time including the rack reloading: Tt = (T120-T1) +Td, where Tt is the total scan time in format (HH: MM: SS), T120 is the start time of the last slide scan in format (DD: MM: YYYY) (HH: MM: SS), T1 is the start time of the first slide scan in format (DD: MM: YYY) (HH: MM: SS) and Td is the duration of the last slide scan time in format (HH: MM: SS). Last slide scan time was measured with a stopwatch for AT2. The results are presented in [Table 2].
Table 2: Comparison of scanning speed of 15×15 mm object with manufacturer's data

Click here to view


Scanning of workflow slide set

For group 2, we selected glasses from the workflow of our laboratory, which had already been scanned without errors on AT2. The characteristics of the materials are presented in [Table 3].
Table 3: Structure of Group 2, the flow material for the study

Click here to view


Selected workflow slides were used for the examination on all three scanners. Each workflow slide stained with H and E contained at least two slices (two or more tissue samples for surgical biopsy, at least four slices for gastric biopsy, and at least two slices for fine-needle biopsy). For cases stained with ICH/HC, two slices were placed on the slide, the specimen to be examined and the control. This is an important point for estimating the scanning area of workflow slides.

Scanning errors

Accounting for scan failures is definitely an important metric for workflow scans. In this study, we eliminated the possibility of errors in the scanning process as much as possible. We selected glasses for Group 2 (workflow) that had already been scanned without errors on AT2. There were no scanning failures in Group 1 (reference).

We assume that a large study with a representative sample or a long scan cycle (24–72 h) is needed to estimate the errors. In addition, the homogeneity of the histological material (surgical, trepan, gastro, etc.) and the technical conditions of the preparation must be considered. The design of such a study requires careful preparation.


   Results Top


Scanning a reference slide 15 mm × 15 mm

Of course, the area of one 15 mm × 15 mm slice is much larger than one gastric or fine-needle biopsy. However, in workflow conditions, it is very rare to have one slice of small samples on the same slide. Thus, the scanning area in the automatic mode with default settings tends to the area of the coverslip and when scanning such slides. Therefore, we allow a scanning speed metric of one slice of 15 mm × 15 mm as the minimum necessary but insufficient.

The results presented in [Table 2] show the scanning speed of one area 15 mm × 15 mm for one slide compared to the manufacturer's data.

The manufacturers specify in the technical specifications the scanning time without the focus and recording time. The official websites of 3DHistech and Hamamatsu indicate the parameter-scanning speed. Leica for the AT2 scanner says “.... scans and moves the images in record time with eSlides ready in 60 s or less for remote pathologist review....based on Aperio AT2 scanning a 15 mm × 15 mm area at ×20.”[7] The “scan speed” parameter for the AT2 is listed for the comparison with other models on the manufacturer's website. With the ×40 objective lens, the scan speed for this model is stated as “~155 s.”[9] In our study, we used only ×20 lenses. For comparison, we left the value of 60 s from the official page of the AT2 model.

Scanning 120 reference and workflow slides

For the C13220, the mean scan time (M ± SD) of the slide in Group 1 was 156 ± 1.25 s and 117 ± 4.17 s in Group 2. For the Pannoramic 1000, the average scan time (M ± SD) of a slide in Group 1 was 210 ± 1.64 s and 183 ± 3.78 s in Group 2. The average scan time per slide with two 15 × 15 slices (Group 1) and a workflow slide set (Group 2) differed statistically significantly (P < 0.0001) for both scanners.

AT2 was excluded from this comparison because the manufacturer's software does not provide scan time data for each slide.

The average scan time data does not take into account the time to reload the rack. Our main goal was to compare the performance of the scanners with regard to rack reloading. The total scan times calculated by the formula are shown in [Table 4].
Table 4: Results of calculated total scanning time with rack reloading of 120 slides with two 15×15 mm slides (Group 1) and workflow slide set (Group 2)

Click here to view



   Discussion Top


We assume that the methodology for scanner testing should be approximated to real-world usage scenarios in heavily loaded digital laboratories. The “scan speed” and “throughput” characteristics without detailing the methodology can lead to the errors in business process planning.

The average scan time in Group 1 is longer than in Group 2 for the C13220 and Pannoramic 1000 models. The significant difference between Groups 1 and 2 can be explained by the heterogeneity of the slice sizes on all 120 slides in Group 2 [Table 3].

The difference between the groups in [Table 5] for the C13220 and Pannoramic 1000 models was approximately 1 h. For AT2, it was approximately, 11 min. This discrepancy is most likely due to the different age of these models and the features of the software settings. The new models use improved algorithms to adapt the scanning process in the automatic mode.
Table 5: Comparison of scanner throughput data

Click here to view


Scanning speed depends on many factors: Sample size, number of focus points, threshold values, etc. If the laboratory has the ability to rank the material and apply presets for different types of material, it can significantly reduce the scanning time. This scheme can be implemented if several scanners are available in the laboratory, or if there is a scheduled inflow of the same type of material. Unfortunately, such conditions are rare. Thus, users are forced to allocate a specialist to scan and experiment with the settings.

Scanner throughput

Based on the measurements from [Table 4], we calculated the estimated throughput of the scanners, taking into account the time to rack reloading. The formula of the proportion x = n × 3600/t, where × is the number of slides per hour, n is the number of scanned slides, 3600 is 1 h converted to seconds and t is the total scanning time for n in seconds. All calculated values were rounded to a whole value.

One can disregard such notations on the manufacturers' website as “~”, “more than”, “up to,”[6],[7],[8] because a spread of 10 slides could be considered insignificant. However, if we take the average time of scanning one slide 2–3 min in real circumstances, then in the workflow condition, the shift in the time of readiness of scans for viewing by a pathologist will be significant. These nuances must be taken into account if the entire laboratory material is digitized in 24/7 mode and there are routine consultations with specialists working in different time zones.

It is generally known that this model of operation (full-scale digitization) is currently affordable to a limited number of laboratories. However, we would like to share our vision of potential problems in the process chain. They can lead to errors in business processes and consequently cause reputation and/or economic loss.

On the one hand, each laboratory needs to carefully study scanner settings and adjust them to their own needs. This will help significantly reduce scanning time and the amount of information stored. In this case, a repository of settings that other laboratories have tested under their own conditions can be a useful solution for the community. On the other hand, digital-only laboratories should not have to spend time searching for such settings on their own. The challenge for manufacturers is to provide ready-made set of settings for different scanning scenarios.


   Conclusions Top


With our article, we want to start discussing the methodology for testing scanners in a real workflow, using metrics that will be helpful for choosing a scanner for laboratories of different levels. Our goal is not to identify the absolute leader and focus on it in the future, but to evaluate the features of different models as fully as possible under load testing, close to the real work of a digital laboratory, working in the continuous mode.

In the absence of a uniform standard and mass experience with 24/7 scanning microscopes, it makes sense to consider a multivendor approach. This would increase flexibility and reduce dependence on one vendor.

Hopefully that a consensus will emerge from the digital pathology community to evaluate the scanner performance. Moreover, these characteristics will be specified by scanner manufacturers in the official documentation.

Our suggestions for a basic testing methodology:

  1. Use a single 15 mm × 15 mm tissue type as the minimum reference set (similar to the manufacturer)
  2. To use a reference set with two 15 mm × 15 mm slices on a single slide as the basic set
  3. To use the smallest total multiple of slides for one series of slides. At the moment, 120 is suitable for most high-performance scanners
  4. To use an additional set that simulates workflow slides (10 trepanobiopsy slides, 10 gastric biopsy slides, 10 skin slides, etc.)
  5. To specify the number of slices per slide to simulate workflow slide set
  6. To evaluate two scanning modes: Full automatic and manual. The automatic mode is designed for fully digital laboratories with minimal involvement of an employee in the scanning process (“walk away” scanning). Manual mode is designed for small laboratories or for material, requiring attention during scanning. It helps to increase scanning accuracy and reduce slide size
  7. In both modes to take into account, the total time taken to get one virtual slide or a series of slides
  8. To specify file recording characteristics (hdd/ssd, internal/external, lan/thunderbolt, raid massive). We believe that the next generation of scanners will use NAS and cloud technologies as storage.


To conclude this part of our study of scanner characteristics, we want to emphasize that our goal was not to identify a good or bad scanner. We suggest that the digital pathology community discuss objective criteria for scanner specifications that affect laboratory performance.

Acknowledgment

We express gratitude and appreciation to the manufacturers of scanners and their representatives in Russia for the samples provided for testing.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.



 
   References Top

1.
Arnold MA, Chenever E, Baker PB, Boué DR, Fung B, Hammond S, et al. The College of American Pathologists guidelines for whole slide imaging validation are feasible for pediatric pathology: A pediatric pathology practice experience. Pediatr Dev Pathol 2015;18:109-16.  Back to cited text no. 1
    
2.
Zarella MD, Bowman D, Aeffner F, Farahani N, Xthona A, Absar SF, et al. A practical guide to whole slide imaging: A white paper from the digital pathology association. Arch Pathol Lab Med 2019;143:222-34.  Back to cited text no. 2
    
3.
Williams BJ, Knowles C, Treanor D. Maintaining quality diagnosis with digital pathology: A practical guide to ISO 15189 accreditation. J Clin Pathol 2019;72:663-8.  Back to cited text no. 3
    
4.
Pantanowitz L, Sinard JH, Henricks WH, Fatheree LA, Carter AB, Contis L, et al. Validating whole slide imaging for diagnostic purposes in pathology: Guideline from the College of American Pathologists Pathology and Laboratory Quality Center. Arch Pathol Lab Med 2013;137:1710-22.  Back to cited text no. 4
    
5.
García-Rojo M. International clinical guidelines for the adoption of digital pathology: A review of technical aspects. Pathobiology 2016;83:99-109.  Back to cited text no. 5
    
6.
NanoZoomer S360 Digital Slide Scanner C13220-01 | Hamamatsu Photonics. Available from: https://nanozoomer.hamamatsu.com/jp/en/scanner/search/C13220-01/index.html. [Last accessed on 2020 Sep 18].  Back to cited text no. 6
    
7.
Aperio AT2 - Digital Whole Slide Scanner, Automated: Leica Biosystems. Available from: https://www.leicabiosystems.com/digital-pathology/scan/aperio-at2/. [Last accessedon 2020 Sep 18].  Back to cited text no. 7
    
8.
PANNORAMIC ® 1000 Introducing the Ultimate Whole-Slide Scanning Solution with Market Leading Performance Unprecedented Capacity Highest Throughput Flexibility. Available from: https://www. 3dhistech.com/wp-content/uploads/2019/09/bp_p1000_022019.pdf. [Last accessed on 2020 Sep 18].  Back to cited text no. 8
    
9.
Aperio GT 450 - Automated, High Capacity Digital Pathology Scanner: Leica Biosystems. Available from: https://www.leicabiosystems.com/digital-pathology/scan/aperio-gt-450/#compare. [Last accessed on 2020 Nov 03].  Back to cited text no. 9
    


    Figures

  [Figure 1], [Figure 2]
 
 
    Tables

  [Table 1], [Table 2], [Table 3], [Table 4], [Table 5]



 

 
Top
  

    

 
  Search
 
   Browse articles
  
    Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
    Access Statistics
    Email Alert *
    Add to My List *
* Registration required (free)  

 
  In this article
    Abstract
   Introduction
   Subjects and Methods
   Results
   Discussion
   Conclusions
    References
    Article Figures
    Article Tables

 Article Access Statistics
    Viewed422    
    Printed6    
    Emailed0    
    PDF Downloaded49    
    Comments [Add]    

Recommend this journal