The use of computer-aided detection (CAD) significantly alters the search and identification of polyps among readers of CT colonography (CTC) data, according to a new study in the June issue of European Radiology. In particular, CAD drew radiologists' attention to polyps more quickly, but it also prompted them to spend more time interpreting scans.
The multireader study from the U.K. comparing CAD to no CAD was unique in its use of eye-tracking technology to determine exactly what CAD users were looking at, a question that has remained largely unanswered throughout the development of CAD software applications.
Armed with tracking technology, the investigators found that CAD software drew readers' attention to polyps faster, accelerating the identification time from more than 0.5 second to less than 0.5 second overall. CAD led to significant increases in the number of correct polyp identifications (87% with CAD versus 47% without), with larger effects on more inexperienced readers. But CAD also showed the potential to distract readers.
"CAD was associated with an increase in polyp identifications in both inexperienced and experienced readers," wrote Drs. Emma Helbren, Thomas Fanshawe, Stuart Taylor, Steve Halligan, and colleagues from University College London, the University of Oxford, and three other U.K. centers. "However, particularly for inexperienced readers, CAD can act as a major distractor, drawing readers' attention both before and during polyp visibility" (Eur Radiol, June 2015, Vol. 25:6, pp. 1570-1578).
CT colonography images with polyp identified with CAD mark (left) and without (right). Images courtesy of Dr. Steve Halligan.CAD improves
In more than a decade of studies, CAD has shown that it can improve sensitivity on both a per-patient and per-polyp basis for radiologists who use it, even outperforming some individual readers when the algorithms are applied to datasets on their own. Experienced readers are needed mostly to reject false-positive CAD prompts, the group wrote.
But what's not known is how different CAD systems affect the behavior of the radiologists who are using them. The use of eye-tracking technology can help: By recording a reader's gaze as they interpret images, researchers can evaluate how visual attention and search behavior operate. But tracking technology has rarely been applied to CTC reading, especially in the 3D data that encompasses a larger volume of studies each year.
"We believed that CAD marks would draw readers' attention since we know this from 2D static work, but wanted to investigate the effect when pathology was moving and changing in size, as happens during real-time 3D interpretation," Halligan told AuntMinnieEurope.com in an e-mail.
The stakes are high, because increasing understanding of how visual CAD prompts affect visual searches could enable optimization of diagnostic performance for both clinical reading and training. The study aimed to determine the effect of CAD markers on readers' visual search behavior using eye-tracking technology and determine the relative effects of CAD on readers with different prior experience, the team wrote.
The investigators recorded 15 endoluminal CTC examinations, generating two videos each -- one with and one without CAD marks. Each case had a single polyp. A total of 42 readers, including 17 experienced and 25 inexperienced, interpreted the videos during a visual search, and their pupil eye movement was recorded using eye-tracking technology (Tobii X50 or X120 eye tracker, Tobii Technology) that was placed beneath their viewing screens.
Faster, longer attention to polyps
The following results were identified:
- The results showed that CAD use drew readers' attention to polyps faster, speeding up polyp identification times to 0.48 seconds with CAD versus 0.58 seconds without.
- In the case of inexperienced readers, CAD held their visual attention for longer, leading to a higher average assessment rate for pursuit of the CAD marker and longer average assessment pursuit times.
- The shorter average time to the first pursuit with CAD was evidenced by a higher hazard ratio of 1.42 (95% confidence interval [CI]: 1.19 to 1.69), and the lower median time of 0.48 seconds (interquartile range [IQR]: 0.27 to 0.87 seconds).
- In addition, the percentage of eye pursuits that immediately fell on polyps was higher with CAD: 29% for inexperienced readers and 27% for inexperienced readers using CAD, versus 16% for inexperienced readers and 12% for experienced readers without CAD.
- The average values for time to first pursuit, identification time span (time to click), and last assessment time span were significantly shorter in the reads with CAD. This phenomenon was evidenced by hazard ratios (HR) that were higher for the identification time span (HR: 1.35, CI: 1.19 to 1.53) and last assessment time span (HR: 1.28, CI: 1.12 to 1.46).
- Finally, a significant increase in the number of correct polyp identifications across all readers was seen with CAD (74 % without CAD, 87 % with CAD; p < 0.001).
"CAD significantly alters visual search and polyp identification in readers viewing three-dimensional endoluminal CTC. For polyp and CAD marker pursuit times, CAD generally exerted a larger effect on inexperienced readers," the team concluded.
Broader meaning?
The study results answer several questions, but leave several others for future research. For example, because CAD certainly misses some polyps, unmarked pathology could be missed if the reader is paying too much attention to the CAD marks, Halligan noted. On the other hand, not seeing a polyp because it's not marked is a risk too, which is where CAD is an important aid.
"When viewing in an endoluminal flythrough mode, there is a 'window of opportunity' during which you have time to see a polyp because when it's moved off the screen, it's gone," Halligan wrote. "That's unlike, say, CAD work in 2D mammography."
It was also interesting to confirm that experienced readers are less prone to distraction than those with less experience, he said. It could be that experienced readers are more aware that CAD algorithms produce false negatives, and so are more concerned with scanning the entire endoluminal surface. Because that's what is needed to optimize detection.
"We hope that appreciation of the distracting effect of CAD will allow readers to focus more on active search strategies that cover other regions of the endoluminal surface," the study team wrote. "For example, it is possible that a synchronous unmarked polyp may be missed if a CAD-marked polyp is simultaneously present onscreen?"
Also, false-positive CAD markers could distract readers from true-positive polyps that are present elsewhere, and future studies should investigate these aspects, the group noted.
"Importantly, all of this has to be balanced against the fact that, overall, readers are more sensitive with CAD than without," Halligan wrote in his email. "In my opinion, our data serve more as a warning to be vigilant when using CAD rather than an exhortation not to use CAD; that would be wrong on the basis of existing reader studies. Our work with discrete choice experiments has shown repeatedly that patients are most concerned with diagnostic sensitivity; specificity is less important to them."
