|
 |
| ORIGINAL ARTICLE |
|
|
|
| J Pathol Inform 2021,
12:39 |
Testing of actual scanner performance in a high-loaded UNIM laboratory environment
Mikhail Yurevich Genis1, Alexey Igorevich Remez1, Maksim Ivanovich Untesco1, Dmitrii Anatolevich Zhakota2
1 UNIM LLC, Moscow, Russia
2 Department of Pathology, Faculty of Pediatric, Pirogov Russian National Research Medical University (Pirogov Medical University), Moscow, Russia
| Date of Submission | 13-Jan-2021 |
| Date of Decision | 27-Apr-2021 |
| Date of Acceptance | 04-Jul-2021 |
| Date of Web Publication | 01-Nov-2021 |
Correspondence Address:
Mikhail Yurevich Genis
Podsosenskaya Lane, 23, b. 6., Moscow 105062
Russia
 Source of Support: None, Conflict of Interest: None  | Check |
DOI: 10.4103/jpi.jpi_4_21
 Abstract | | |
Background: Scanners are the main tool in digital pathology. The technical abilities of scanners determine the workflow logic in the pathology laboratory. Its performance can be restricted by the divergence between the scanning time presented by the manufacturer and the actual scanning time. This could lead to critical deviations from the established business processes in a 24/7 laboratory. Aim: Our investigation is focused in exploring the performance of three main models of high-performance scanners available on the Russian market: 3DHistech, Hamamatsu и Leica. Objectives: We compared the performance of the scanners on the samples of a given size with the manufacturer's stated specifications and evaluated the speed of the scanners on the reference and routine laboratory material. Subjects and Methods: We examined 3DHistech Pannoramic 1000, Hamamatsu NanoZoomer s360 and Leica AT2 with default settings and automatic mode. Two sets of glasses were used (glass slide): Group 1 included 120 slides with 15 mm × 15 mm slices, Group 2 included 120 workflow slides. Results: The average slide scan times in Groups 1 and 2 for the C13220 (156 ± 1.25 s and 117 ± 4.17 s) and Pannoramic 1000 (210 ± 1.64 s and 183 ± 3.78 s) differ statistically significantly (P < 0.0001). Total scanning time including rack reloading was shorter for the workflow slide set group for the modern C13220 and Pannoramic 1000 scanner models. Conclusions: The scanner specifications provided by manufacturers are not sufficient to evaluate the performance. The guidelines and regulations concerning scanner selection should be consented by the digital pathology community. We suggest discussing criteria for evaluating scanner performance.
Keywords: Digital pathology, high-throughput scanner, scanning speed,Whole Slide Images in Pathology.
How to cite this article:
Genis MY, Remez AI, Untesco MI, Zhakota DA. Testing of actual scanner performance in a high-loaded UNIM laboratory environment. J Pathol Inform 2021;12:39 |
| Introduction | |  |
Successful implementation of whole slide imaging for routine clinical practice and COVID-19 epidemic crisis demonstrates the need to introduce digital technology into pathology for rapid integration into the chain of telemedicine consultations. The area of digital pathology was introduced in the clinical practice quite recently. As a result, the choice the right technical solution in this area is even more urgent. Most guidelines address the topic generally and suggestively, and it needs further investigation.[1],[2],[3],[4],[5] General problem of scanner performance in the workflow was explored by Zarell et al.[2] and García-Rojo.[5]
The UNIM laboratory, founded in 2014, works as a fully digital pathology laboratory. The lab is ISO 15189:2012 certified. During the work process, it was found that the throughput of the scanners differed from the manufacturer's stated capacity. Therefore, it was crucial to evaluate the real capabilities of the equipment. We have had two Aperio AT2 scanners in service 24/7 since 2018. They are serviced once a year. The average equipment utilization rate in our laboratory is 80%. For this reason, we are constantly looking for relevant solutions on the market and regularly request equipment from all manufacturers represented in Russia for testing.
| Subjects and Methods | | |
Three models of scanners participated in the study: Aperio AT2, 3DHistech Pannoramic 1000 and Hamamatsu NanoZoomer s360 C13220. It should be noted that these scanners have been on the market for different number of years. To compare them as identical models would be rather inappropriate. This was the reason for simplifying the comparison method.
Scanner settings
Manufacturers equip their scanners with a wide range of settings. In this study, we only used the default settings offered by the manufacturer. We are primarily interested in the performance of the equipment with preconfigured settings to minimize startup and maintenance times during the scanning process.
To equalize the working conditions of the scanners, we chose automatic mode without prescan checks. We were interested not only in the speed of scanning one slide or one rack, but also in the time it takes to rack reloading. All scanners had different rack capacities. We chose the smallest multiple of 120 slides for all scanners. General technical parameters are presented in [Table 1].
Samples for reference scanning
In the characteristics of scanners manufacturers indicate the scanning speed of one sample of a given size, 15 mm × 15 mm.[6],[7],[8] As a rule, pure scanning time of a tissue sample of this size is indicated. That is, without taking into account focusing, recording time, moving slides in the magazine, changing magazines, etc. These are justified minimum parameters to estimate the scanning speed. However, it is not enough to understand the throughput of high-performance scanners for 24/7 operation. We decided to expand the testing conditions.
Manufacturers do not specify the histological affiliation of the tissue. In our study, we used myocardial tissue. A 15 mm × 15 mm × 5 mm tissue sample from autopsy without cardiac pathology was taken to make Group 1 preparations. This is a sufficiently homogeneous tissue, which is minimally prone to artifacts when making slides. The myocardial sample was fixed in 10% buffered formalin, and the blocks and slides were made according to the standard laboratory process chain. The technological chain included devices Tissue-Tek VIP Jr, Excelsior AS, Leica R2235, Leica BOND, Tissue-Tek Prisma, and Tissue-Tek Film.
For group 1, 120 slides with a slice thickness of 2–3 μ were made. In routine work, there are usually several slices on a single slide. Based on this, we decided to simulate the maximum load when scanning surgical biopsies. For this purpose, we placed two 15 mm × 15 mm slices on each slide. Hematoxylin and eosin staining was used. For the study, we needed native format files of scanner manufacturers. The slides were numbered manually from 1 to 120, because in the technological chain of the laboratory slides with barcode are loaded immediately after scanning to the server with LIS UNIM. Saving the scans in LIS UNIM would have affected the purity of the experiment. The slides were moved between scanners with strict sequencing.
Scanning a 15 mm × 15 mm reference sample
To make a comparison with the manufacturers' stated characteristics, we took one slide with reference slices of 15 mm × 15 mm. We manually selected a 15 mm × 15 mm area [Figure 1]. Afterward, we measured the time between pressing the start button and getting the finished virtual slide for the AT2. For other scanners, we have set the full scanning time according to the manufacturer's software. | Figure 1: Scanner software interface in single area 15 mm × 15 mm scanning mode ([a] - AT2, [b] - C13220, [c] - Panoramic 1000)
Click here to view |
Scanning 120 reference slides
In the next step, we studied the scanning of 120 slides loaded at a time with two 15 mm × 15 mm slices. The slides were distributed in racks. Racks were placed sequentially in the scanner. Afterward, the scanning process was started [Figure 2]. | Figure 2: Software interface in the automatic scanning mode with two sections 15x15 mm ([a] - AT2, [b] - C13220, [c] - Panoramic 1000). The scanning area tends to the cover glass area.
Click here to view |
To calculate the average scanning time of a slide in each group, we took the scanning time from each file for the C13220 and Panoramic 1000 models using the manufacturers' software. Since the AT2 manufacturer's software does not provide this metric, this scanner was excluded from this step. Statistical analysis of the data obtained was performed with Statistica 10.0 software (StatSoft, Tulsa, OK, USA). Data were presented as M ± standard deviation (SD). Significance of differences was assessed by the Student's t-test. The results were considered significant at 95.5% probability (P < 0.05).
The formula was used to calculate the total scanning time including the rack reloading: Tt = (T120-T1) +Td, where Tt is the total scan time in format (HH: MM: SS), T120 is the start time of the last slide scan in format (DD: MM: YYYY) (HH: MM: SS), T1 is the start time of the first slide scan in format (DD: MM: YYY) (HH: MM: SS) and Td is the duration of the last slide scan time in format (HH: MM: SS). Last slide scan time was measured with a stopwatch for AT2. The results are presented in [Table 2]. | Table 2: Comparison of scanning speed of 15×15 mm object with manufacturer's data
Click here to view |
Scanning of workflow slide set
For group 2, we selected glasses from the workflow of our laboratory, which had already been scanned without errors on AT2. The characteristics of the materials are presented in [Table 3].
Selected workflow slides were used for the examination on all three scanners. Each workflow slide stained with H and E contained at least two slices (two or more tissue samples for surgical biopsy, at least four slices for gastric biopsy, and at least two slices for fine-needle biopsy). For cases stained with ICH/HC, two slices were placed on the slide, the specimen to be examined and the control. This is an important point for estimating the scanning area of workflow slides.
Scanning errors
Accounting for scan failures is definitely an important metric for workflow scans. In this study, we eliminated the possibility of errors in the scanning process as much as possible. We selected glasses for Group 2 (workflow) that had already been scanned without errors on AT2. There were no scanning failures in Group 1 (reference).
We assume that a large study with a representative sample or a long scan cycle (24–72 h) is needed to estimate the errors. In addition, the homogeneity of the histological material (surgical, trepan, gastro, etc.) and the technical conditions of the preparation must be considered. The design of such a study requires careful preparation.
| Results | | |
Scanning a reference slide 15 mm × 15 mm
Of course, the area of one 15 mm × 15 mm slice is much larger than one gastric or fine-needle biopsy. However, in workflow conditions, it is very rare to have one slice of small samples on the same slide. Thus, the scanning area in the automatic mode with default settings tends to the area of the coverslip and when scanning such slides. Therefore, we allow a scanning speed metric of one slice of 15 mm × 15 mm as the minimum necessary but insufficient.
The results presented in [Table 2] show the scanning speed of one area 15 mm × 15 mm for one slide compared to the manufacturer's data.
The manufacturers specify in the technical specifications the scanning time without the focus and recording time. The official websites of 3DHistech and Hamamatsu indicate the parameter-scanning speed. Leica for the AT2 scanner says “.... scans and moves the images in record time with eSlides ready in 60 s or less for remote pathologist review....based on Aperio AT2 scanning a 15 mm × 15 mm area at ×20.”[7] The “scan speed” parameter for the AT2 is listed for the comparison with other models on the manufacturer's website. With the ×40 objective lens, the scan speed for this model is stated as “~155 s.”[9] In our study, we used only ×20 lenses. For comparison, we left the value of 60 s from the official page of the AT2 model.
Scanning 120 reference and workflow slides
For the C13220, the mean scan time (M ± SD) of the slide in Group 1 was 156 ± 1.25 s and 117 ± 4.17 s in Group 2. For the Pannoramic 1000, the average scan time (M ± SD) of a slide in Group 1 was 210 ± 1.64 s and 183 ± 3.78 s in Group 2. The average scan time per slide with two 15 × 15 slices (Group 1) and a workflow slide set (Group 2) differed statistically significantly (P < 0.0001) for both scanners.
AT2 was excluded from this comparison because the manufacturer's software does not provide scan time data for each slide.
The average scan time data does not take into account the time to reload the rack. Our main goal was to compare the performance of the scanners with regard to rack reloading. The total scan times calculated by the formula are shown in [Table 4]. | Table 4: Results of calculated total scanning time with rack reloading of 120 slides with two 15×15 mm slides (Group 1) and workflow slide set (Group 2)
Click here to view |
| Discussion | | |
We assume that the methodology for scanner testing should be approximated to real-world usage scenarios in heavily loaded digital laboratories. The “scan speed” and “throughput” characteristics without detailing the methodology can lead to the errors in business process planning.
The average scan time in Group 1 is longer than in Group 2 for the C13220 and Pannoramic 1000 models. The significant difference between Groups 1 and 2 can be explained by the heterogeneity of the slice sizes on all 120 slides in Group 2 [Table 3].
The difference between the groups in [Table 5] for the C13220 and Pannoramic 1000 models was approximately 1 h. For AT2, it was approximately, 11 min. This discrepancy is most likely due to the different age of these models and the features of the software settings. The new models use improved algorithms to adapt the scanning process in the automatic mode.
Scanning speed depends on many factors: Sample size, number of focus points, threshold values, etc. If the laboratory has the ability to rank the material and apply presets for different types of material, it can significantly reduce the scanning time. This scheme can be implemented if several scanners are available in the laboratory, or if there is a scheduled inflow of the same type of material. Unfortunately, such conditions are rare. Thus, users are forced to allocate a specialist to scan and experiment with the settings.
Scanner throughput
Based on the measurements from [Table 4], we calculated the estimated throughput of the scanners, taking into account the time to rack reloading. The formula of the proportion x = n × 3600/t, where × is the number of slides per hour, n is the number of scanned slides, 3600 is 1 h converted to seconds and t is the total scanning time for n in seconds. All calculated values were rounded to a whole value.
One can disregard such notations on the manufacturers' website as “~”, “more than”, “up to,”[6],[7],[8] because a spread of 10 slides could be considered insignificant. However, if we take the average time of scanning one slide 2–3 min in real circumstances, then in the workflow condition, the shift in the time of readiness of scans for viewing by a pathologist will be significant. These nuances must be taken into account if the entire laboratory material is digitized in 24/7 mode and there are routine consultations with specialists working in different time zones.
It is generally known that this model of operation (full-scale digitization) is currently affordable to a limited number of laboratories. However, we would like to share our vision of potential problems in the process chain. They can lead to errors in business processes and consequently cause reputation and/or economic loss.
On the one hand, each laboratory needs to carefully study scanner settings and adjust them to their own needs. This will help significantly reduce scanning time and the amount of information stored. In this case, a repository of settings that other laboratories have tested under their own conditions can be a useful solution for the community. On the other hand, digital-only laboratories should not have to spend time searching for such settings on their own. The challenge for manufacturers is to provide ready-made set of settings for different scanning scenarios.
| Conclusions | | |
With our article, we want to start discussing the methodology for testing scanners in a real workflow, using metrics that will be helpful for choosing a scanner for laboratories of different levels. Our goal is not to identify the absolute leader and focus on it in the future, but to evaluate the features of different models as fully as possible under load testing, close to the real work of a digital laboratory, working in the continuous mode.
In the absence of a uniform standard and mass experience with 24/7 scanning microscopes, it makes sense to consider a multivendor approach. This would increase flexibility and reduce dependence on one vendor.
Hopefully that a consensus will emerge from the digital pathology community to evaluate the scanner performance. Moreover, these characteristics will be specified by scanner manufacturers in the official documentation.
Our suggestions for a basic testing methodology:
- Use a single 15 mm × 15 mm tissue type as the minimum reference set (similar to the manufacturer)
- To use a reference set with two 15 mm × 15 mm slices on a single slide as the basic set
- To use the smallest total multiple of slides for one series of slides. At the moment, 120 is suitable for most high-performance scanners
- To use an additional set that simulates workflow slides (10 trepanobiopsy slides, 10 gastric biopsy slides, 10 skin slides, etc.)
- To specify the number of slices per slide to simulate workflow slide set
- To evaluate two scanning modes: Full automatic and manual. The automatic mode is designed for fully digital laboratories with minimal involvement of an employee in the scanning process (“walk away” scanning). Manual mode is designed for small laboratories or for material, requiring attention during scanning. It helps to increase scanning accuracy and reduce slide size
- In both modes to take into account, the total time taken to get one virtual slide or a series of slides
- To specify file recording characteristics (hdd/ssd, internal/external, lan/thunderbolt, raid massive). We believe that the next generation of scanners will use NAS and cloud technologies as storage.
To conclude this part of our study of scanner characteristics, we want to emphasize that our goal was not to identify a good or bad scanner. We suggest that the digital pathology community discuss objective criteria for scanner specifications that affect laboratory performance.
Acknowledgment
We express gratitude and appreciation to the manufacturers of scanners and their representatives in Russia for the samples provided for testing.
Financial support and sponsorship
Nil.
Conflicts of interest
There are no conflicts of interest.
| References | | |
| 1. | Arnold MA, Chenever E, Baker PB, Boué DR, Fung B, Hammond S, et al. The College of American Pathologists guidelines for whole slide imaging validation are feasible for pediatric pathology: A pediatric pathology practice experience. Pediatr Dev Pathol 2015;18:109-16.  |
| 2. | Zarella MD, Bowman D, Aeffner F, Farahani N, Xthona A, Absar SF, et al. A practical guide to whole slide imaging: A white paper from the digital pathology association. Arch Pathol Lab Med 2019;143:222-34. |
| 3. | Williams BJ, Knowles C, Treanor D. Maintaining quality diagnosis with digital pathology: A practical guide to ISO 15189 accreditation. J Clin Pathol 2019;72:663-8. |
| 4. | Pantanowitz L, Sinard JH, Henricks WH, Fatheree LA, Carter AB, Contis L, et al. Validating whole slide imaging for diagnostic purposes in pathology: Guideline from the College of American Pathologists Pathology and Laboratory Quality Center. Arch Pathol Lab Med 2013;137:1710-22. |
| 5. | García-Rojo M. International clinical guidelines for the adoption of digital pathology: A review of technical aspects. Pathobiology 2016;83:99-109. |
| 6. | |
| 7. | |
| 8. | |
| 9. | |
[Figure 1], [Figure 2]
[Table 1], [Table 2], [Table 3], [Table 4], [Table 5]
|
