Abstract
Background: It is anticipated that many licensing examination centres for pathology will begin fully digitizing the certification examinations. The objective of our study was to test the feasibility of a fully digital examination and to assess the needs, concerns and expectations of pathology residents in moving from a glass slide-based examination to a fully digital examination.
Methods: We conducted a mixed methods study that compared, after randomization, the performance of senior residents (postgraduate years 4 and 5) in 7 accredited anatomical pathology training programs across Canada on a pathology examination using either glass slides or digital whole-slide scanned images of the slides. The pilot examination was followed by a post-test survey. In addition, pathology residents from all levels of training were invited to participate in an online survey.
Results: A total of 100 residents participated in the pilot examination; 49 were given glass slides instead of digital images. We found no significant difference in examination results between the 2 groups of residents (estimated marginal mean 8.23/12, 95% confidence interval [CI] 7.72-8.87, for glass slides; 7.84/12, 95% CI 7.28-8.41, for digital slides). In the post-test survey, most of the respondents expressed concerns with the digital examination, including slowly functioning software, blurring and poor detail of images, particularly nuclear features. All of the respondents of the general survey (n = 179) agreed that additional training was required if the examination were to become fully digital.
Interpretation: Although the performance of residents completing pathology examinations with glass slides was comparable to that of residents using digital images, our study showed that residents were not comfortable with the digital technology, especially given their current level of exposure to it. Additional training may be needed before implementing a fully digital examination, with consideration for a gradual transition.
The evolution of information technology has revolutionized the practice of pathology. After years of traditional practice using glass slides, we are stepping into a new era of digital images and telepathology.1 Digital pathology can be defined as an image-based information environment enabled by computer technology that allows for the management of information generated from a digital slide. Digital pathology is enabled in part by virtual microscopy, which involves the conversion of glass slides into digital slides that can be viewed, managed and analyzed.2,3 The range of applications of digital pathology is wide and includes primary diagnosis, intraoperative and remote consultation through telepathology,1,4-6 quality assurance, archiving, education and conferences, examinations, automated image analysis, research and publishing.1,7
Traditionally, education and training in pathology have been delivered using glass slides and conventional microscopy. Recently, there has been a gradual switch to use of digital images and Web-based pathology resources at the different levels of medical education, from teaching medical students to residents.8 Whole-slide imaging technology allows glass slides to be scanned and viewed on a computer screen at different magnifications as an exact replica of the glass slide. This technology has created enormous opportunities in pathology training and education.9 Several major initiatives are also underway that introduce online competency and diagnostic decision analysis using virtual microscopy, which have important future roles in accreditation and recertification.10
For examination purposes, replacing glass slides with digital whole-slide scanned images offers a number of advantages, including a wider range of case selection and the availability of rare cases and small biopsy specimens, from which it is difficult to obtain multiple identical copies for a glass slide-based examination. Use of digital slides also allows easier handling and storage and ensures uniformity of the examination for all candidates. Furthermore, it eliminates the need to carry a microscope to the examination centre. There are, however, challenges experienced with digital pathology in general, including quality of images,11 ease of navigation and the unfamiliarity of many pathologists and pathology residents with the technology.1,12 Another concern is the authenticity of digital images and the ability to alter their details.11,13,14
Residency training in anatomical pathology in Canada is a 5-year program, with 4 years devoted to pathology. Training is conducted mainly through the traditional approach of the resident shadowing an attendant pathologist in evaluating glass slides using conventional light microscopes. Exposure to digital pathology varies among the different centres.
At the end of training, residents are required to pass the Royal College of Physicians and Surgeons of Canada's certification examination to obtain a licence for independent practice as a specialist in pathology. In the past, the practical component of the examination involved assessment of glass slides. Currently, certain components of the examination, including slides used for oral discussion, gross pathology and cytopathology, have involved digital whole-slide scanned images.
It is anticipated that the Royal College certification examination for pathology will be fully digitized. Before implementing this change, it is essential to test the feasibility of this examination method and to understand fully the needs, concerns and reaction of residents toward this emerging technology. The aim of our study was to test the feasibility of a fully digital examination by comparing the performance of senior pathology residents in completing the examination using glass slides versus digital images and to test whether more time is needed to navigate through digital images. We also assessed the needs, concerns and expectations of pathology residents in moving to a fully digital examination.
Methods
Study design
The mixed methods study consisted of 2 parts. First, we compared the diagnostic performance of senior residents (postgraduate years 4 and 5) in anatomical pathology using glass slides versus digital images through a pilot examination in 7 accredited anatomical pathology training programs across Canada.
The second portion of the study consisted of an online survey with 2 parts. The first part (general) was open to residents from all levels of training (postgraduate years 1to 5), and the second part (post-test) was restricted to the residents who participated in the pilot examination.
The study was approved by the Research Ethics Committee of St. Michael's Hospital.
Participants
There are 9 accredited anatomical pathology training programs in Canada. We recruited residents at 7 of the programs (University of British Columbia, University of Calgary, Western University, McMaster University, University of Toronto, University of Montréal and Dalhousie University). Two centres were excluded from the study because of an insufficient number of participants for randomization purposes.
Participants in the pilot examination (residents in postgraduate years 4 and 5) were recruited through a general email sent by the director of each participating program. The residents were asked to respond directly to the study coordinator; their responses were not available to the program directors, and the program directors did not influence the residents to participate in the study.
Invitations to participate in the online survey were emailed to residents in anatomical pathology in all years of training (postgraduate years 1 to 5) at the 7 participating centres. A separate section of the survey (post-test) was open only to residents who had participated in the pilot examination.
Residents were not compensated for their participation.
Slide examination
In the pilot examination, we compared the diagnostic performance of the senior residents using glass slides versus digital images of the slides. The residents from each centre were randomly assigned into 2 groups. Half of the residents from each program were given glass slides to assess, and the other half from the same program performed the test using a digital version of the identical set of slides. The examination was administered under the supervision of a staff pathologist, and the digital slides were accessible through a central server.
In an attempt to simulate the Royal College examination as much as possible, 24 slides were carefully selected from a large database of examination-style slides, provided by participating examination committee members. Case variety included resections, in addition to biopsy specimens. Slides represented a variety of cases covering the range of subspecialties of anatomical pathology, with different diagnostic entities (e.g., inflammation, cancer, normal).
Recuts were ordered to standardize the glass-slide portion of the test for all centres. The glass-slide portion of the test was divided into 2 groups (A and B), each consisting of a different set of slides. Each slide in both groups was allocated the standard 2 minutes per slide for diagnosis. The total duration of the glass-slide portion of the test was 48 minutes. Microscopes used for the pilot examination were equipped with 4 diagnostic powers (magnification × 2.5, × 10, × 20 and × 40).
For the digital examination, the same 24 slides used for the glass-slide portion were scanned at magnification × 40 for high resolution using an Aperio slide scanner and were uploaded on a University of Calgary server, along with an answer sheet.
The participating residents were asked to provide the most probable diagnosis for each slide. Only 1 answer was allowed for each slide.
There were concerns about the time needed for diagnosis using digital images. It was thought that more time might be required to navigate through a digital image compared with a traditional glass slide. Accordingly, the digital slides used in the pilot examination were split into 2 groups (matching the same glass slides in groups A and B in the glass-slide portion of the examination) to test whether more time allocated to the digital images would improve accuracy of the diagnosis. For group A, each of the 12 digital slides was allocated the standard 2 minutes, for a total duration of 24 minutes. For group B, each slide was allocated 3 minutes, for a total duration of 36 minutes. The total duration of the digital examination was 60 minutes.
Survey
All of us met by teleconference to develop a preliminary list of questions for the survey. A group of residents also met via teleconference to identify an additional set of questions. The final draft of the survey was designed with the help of a professional biostatistician. The final proposed survey was presented and discussed at the academic half-day for anatomical pathology residents at the University of Toronto and was further refined during the Pathology Informatics Group annual meeting of the Canadian Association of Pathologists, to ensure optimum quality and clarity of the questions.
The survey was administered online and consisted of 18 questions (closed, multiple-choice and open-ended). A copy of the questionnaire is included in Appendix 1 (available at www.cmajopen.ca/content/4/1/E88/suppl/DC1). The first part (general) consisted of 13 questions and was open to all residents (postgraduate years 1 to 5); the second part (post-test) consisted of 5 questions and was restricted to the residents who participated in the pilot examination.
Statistical analysis
Statistical analyses were conducted with SPSS for Windows, version 20. We assessed differences between group A and group B slides, and between glass and digital slides, using a 2-factor analysis of variance (factor 1 = glass v. digital; factor 2 = group A slides v. group B slides). Estimated marginal means are presented to show the independent contributions of these 2 factors.
Results
Pilot examination
For the pilot examination, 105 senior residents were invited to participate, of whom 100 agreed, for a participation rate of 95% (Figure 1).
Selection of senior residents (postgraduate years 4 and 5) for participation in the pilot examination. R = randomization.
For group A slides, 24 residents completed the examination using glass slides. The average score was 61%. The same slides were interpreted in digital format by 26 different residents, with an average score of 57%. For group B slides, 25 residents assessed glass slides, with an average score of 78%. The same slides were interpreted in digital format by 25 different residents, with an average score of 74%. Performance was comparable between the participating centres.
As shown in Table 1, there was a significant effect on individual scores associated with group A slides compared with group B slides (F = 24.92, p = 0.001). Scores were lower with group A slides (estimated marginal mean 7.06/12, 95% confidence interval [CI] 6.49-7.63) than with group B slides (estimated marginal mean 9.08/12, 95% CI 8.51-9.65). There was no significant difference in scores between glass and digital slides (F = 1.25, p = 0.3). The estimated marginal mean was 8.23/12 (95% CI 7.72-8.87) for glass slides and 7.84/12 (95% CI 7.28-8.41) for digital slides. There was no apparent advantage in allowing candidates additional time with the digital images (p > 0.9).
Post-test survey
All 100 residents who participated in the pilot examination completed the post-test survey. Eighty percent (n = 80) were more comfortable with glass slides than with digital slides in examination settings; 15% (n = 15) had no preference (Appendix 2, available at www.cmajopen.ca/content/4/1/E88/suppl/DC1). Half of the respondents (n = 50) thought that 3 minutes were necessary for diagnosis of biopsy specimens, whereas 65% (n = 65) thought that 3 minutes were needed only for large resection specimens. Most respondents (95%, n = 95) reported encountering problems during the digital portion of the test, specifically software functioning too slowly, blurring and poor detail of images, and nuclear features being unsatisfactory. Some participants had difficulty navigating through the entire slides and thought that they were missing important diagnostic features. One resident was unable to take the examination because the system froze and the digital slides did not upload.
General survey
The participation rate for the general part of the survey was 81% (179/220), with 59% of the respondents in postgraduate years 4 and 5 (105/179). Half of the respondents (n = 89) were rarely or never exposed to digital slides during their training. Most exposure to digital slides occurred during academic half-days and in-house examinations (60%, n = 107). None of the respondents were exposed to digital slides on a daily basis for routine sign-out purposes (Figure 2).
Exposure of survey respondents (all levels of training, n = 179) to digital pathology during training.
Twenty percent (n = 36) of all respondents (at all levels of training) stated that they were very uncomfortable using digital slides, whereas 10% (n = 18) were very comfortable with using digital slides. Most of the respondents were undecided (Figure 3).
Comfort level of survey respondents (all levels of training, n = 179) with digital whole-slide scanned images.
The main advantages and disadvantages of digital slides, as perceived by the survey respondents, are summarized in Box 1. When comparing the overall advantages and disadvantages of digital versus glass slides, respondents thought that the use of glass slides for routine pathology practice was more cost-effective than digital slides. Most residents agreed that it was much faster (90%, n = 161) and easier (80%, n = 143) to navigate through glass slides, as compared with digital slides. Seventy percent (n = 125) of respondents thought that glass slides were more practical to use during training and in preparation for real practice, whereas 80% (n = 143) stated that the quality of images would be better using glass slides, especially for cytology cases. Anticipated advantages of digital slides included easier transfer between centres for consultation, conservation of storage space and, for examination purposes, no need to carry a microscope to the examination (Table 2).
Box 1: Advantages and disadvantages of digital whole-slide scanned images reported by survey participants (n = 179)
Advantages
• Flexibility of exchange of material between centres (n = 107)
• Conservation of storage space (n = 161)
• Consistency of appearance of tissue (no fading slides) (n = 81)
• Availability of rare cases for examination purposes (n = 80)
• No need to carry a microscope to the examination (n = 161)
Disadvantages
• Additional training needed (n = 160)
• High cost of processing (n = 134)
• Inappropriate for cytology cases (n = 143)
• More time required to examine cases (n = 160)
• Poor nuclear details (n = 150)
• Technical difficulties (n = 160)
In questions about preparation for the Royal College examination, all of the respondents (n = 179) agreed that additional training was necessary to become familiar with digital imaging; 90% (n = 161) suggested that in-house examinations should be digitized starting in the first postgraduate year. Nearly half (45%, n = 80) suggested that a 1-month rotation in digital pathology could help in familiarization with digital slides.
When asked how they would feel if the Royal College fully digitized the examination, 50% (n = 90) of the respondents were not in favour, 25% (n = 45) were undecided, and 25% (n = 45) were in favour. If the Royal College examination were to be fully digitized, 85% (n = 152) of the respondents suggested that the idea should be gradually implemented over a period of at least 2 to 3 years.
Interpretation
The results of our pilot examination showed that the mean score was 4% higher with glass slides than with digital images of the slides; however, the difference was not statistically significant. This finding is in keeping with recent literature, indicating that digital whole-slide scanned images are of comparable efficiency to glass slides for diagnostic purposes.2,8,15,16 There were no apparent advantages in allowing candidates more time with the digital images (3 min v. 2 min).
Interestingly, although the results of our pilot examination showed that the residents' performance was similar between those given glass slides and those using digital images, many residents who completed the general survey expressed concerns about the Royal College examination being fully digitized. This finding reflects the need for more communication and evidence-based discussions about the digital examination with residents. The lack of understanding of the nature and limitations of digital pathology has been highlighted as an important challenge in previous studies.8,12
The residents who underwent the digital portion of the examination faced a number of challenges. They had issues with uploading slides and changing resolutions. In some cases, the images were blurry and nuclear features were not appreciated. Some residents had difficulty navigating through the entire set of slides. Overall, many of the issues raised were related to resolution and are likely to decrease as the technology improves.17-19
The use of digital pathology for educational purposes is not unprecedented. There are a growing number of digital pathology initiatives that are being pursued. About a third of American medical schools have incorporated digital whole-slide images into their pathology training, with promising results.20 Moreover, the American Board of Pathology examination is partially administered with the use of digital slides.21 Some pathology informatics rotations are now in place in several countries, including the United States and Canada.22-24
An important challenge highlighted in the survey was balancing the need to develop skills for real practice as independent pathologists (mainly assessing glass slides) and the need to deve lop skills to pass the certification examination (through digital pathology training). Although digital whole-slide images are currently emerging as a tool for consultation and education, most reports suggest that these images are not yet ready for routine use in Canada.25 More studies are needed to investigate whether a digital-based examination may compromise the diagnostic ability of residents as future pathologists. Alternative approaches include a partially digital and partially glass slide-based examination. Our results are in agreement with a recent survey showing positive attitudes of pathologists and residents toward digital pathology in Canada,11 especially for consultation-type applications (frozen section or second opinion).
Limitations
Although our study provides strong evidence about the feasibility of a fully digital Royal College examination, it has several limitations. A wider variety of cases, including cytology cases, would have added value to the pilot examination. There was also no standardization of hardware or software specifications for computers used in the pilot exam.
Areas of future research include comparing different resolutions when creating digital images and evaluating how they affect diagnostic accuracy, as well as comparing the quality of digital images from different scanning providers. If the Royal College examination is fully digitized, a post-implementation assessment would be important to identify new issues that may arise. Finally, as we approach an era of digital pathology, future research should be directed toward the feasibility of digital pathology for other applications, for example, digital consultations.
Conclusion
Our results showed comparable performance between residents using glass slides and those using digital images in the pilot examination. However, our study highlights some concerns that need to be addressed before a fully digital examination can be implemented. The need for more training was an important issue raised by the participants. Additional training through digitizing in-house examinations and a block rotation in digital pathology were thought to be important preparatory steps. A gradual transition to a fully digital examination should be considered so that residents will become more comfortable with using the technology.26
Our study also highlights many of the advantages of a digital examination and shows that, with adequate training, the idea of digitizing the Royal College examination for pathologists may be possible.
Supplemental information
For reviewer comments and the original submission of this manuscript, please see www.cmajopen.ca/content/4/1/E88/suppl/DC1
Footnotes
Competing interests: None declared.
Contributors: George Yousef was the principal investigator. Lorna Mirham coordinated the study. Christopher Naugler, Malcolm Hayes, Nadia Ismiil, Annie Belisle, Shachar Sade, Snezana Popovic, Mariamma Joseph, Manal Gabril and Penny Barnes randomly assigned the senior residents for the pilot examination, administered the examination and collected the results. Malcolm Hayes, Nadia Ismiil, Annie Belisle, Shachar Sade, Catherine Streutker, Christina MacMillan and Golnar Rasty conducted the quality assessment of the digital and glass slides and prepared the recuts. Christopher Naugler created digital slides. Christopher Naugler, Malcolm Hayes, Nadia Ismiil, Annie Belisle, Shachar Sade, Catherine Streutker, Christina MacMillan, Golnar Rasty, Snezana Popovic, Mariamma Joseph, Manal Gabril and Penny Barnes designed the survey. Christopher Naugler and Richard Hegele conducted the statistical analysis. Lorna Mirham and Richard Hegele drafted the manuscript; Christopher Naugler, Malcolm Hayes, Nadia Ismiil, Annie Belisle, Shachar Sade, Catherin Streutker, Christina MacMillan, Golnar Rasty, Snezana Popovic, Mariamma Joseph, Manal Gabril, Penny Barnes, Beverley Carter and George Yousef revised it for important intellectual content. All of the authors accepted the version to be published and agreed to act as guarantors of the work.
Funding: This study was supported by the Institute for Clinical Evaluative Sciences (ICES), which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). The opinions, results and conclusions reported in this article are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred.
References
- Copyright 2016, 8872147 Canada Inc.
In this issue
Article tools
