Radiologists examining data from virtual colonoscopy (also known as CT colonography or CTC) studies can easily dismiss increasing numbers of false-positive detections generated by computer-aided detection (CAD) systems, researchers from the U.K. report. But the presence of more false positives makes reading less efficient.
"For all computer-aided detection ... systems, there is a trade-off between algorithm sensitivity and specificity (false-positive rate)," explained Dr. Stuart Taylor and colleagues in the American Journal of Roentgenology. "Although some CAD software is interactive, allowing the user to define the sensitivity-specificity profile according to preference, most operate at a fixed point chosen by software developers to maximize sensitivity while giving an 'acceptable' false-positive rate," (AJR, June 2009, Vol. 192:6, pp. 1682-1689).
However, what constitutes an acceptable false-positive rate is poorly defined, wrote the researchers from University College Hospital in London and five other U.K. hospitals. Taylor, along with Dr. John Brittenden, Dr. James Lenton, and colleagues, sought to determine whether an increasing number of false-positive CAD prompts harms reader sensitivity and diagnostic confidence for trained but inexperienced readers.
Fifty endoscopically proven CTC datasets (29 men, 21 women; mean age, 65 years) were selected from a research database; each had been verified for technical adequacy and were read by an expert with full knowledge of the endoscopic results. Half of the cases were positive (containing 35 polyps ≥ 5 mm) and the other 25 were negative for colorectal polyps.
Before imaging, all patients underwent cathartic bowel cleansing with 45 mL of sodium phosphosoda or received sodium picosulfate with magnesium citrate, in addition to fecal tagging (2% barium suspension).
Five fully trained but relatively inexperienced readers examined each dataset using both sphericity settings, chosen randomly, with readings of the same dataset separated by at least six weeks of time to minimize recall bias. As expected, lowering the sphericity setting increased sensitivity and the number of false positives (i.e., decreasing specificity).
Results for all readers for polyps ≥ 5 mm
|
|||||||||||||||||||||||||||||||||||||||||||||
At both sphericity settings, the use of CAD increased per-patient sensitivity from 82% to 87% (p = 0.03) and increased per-polyp sensitivity by 8% and 10% for sphericity settings of 0 and 75, respectively (p < 0.001), the authors reported. Specificity decreased from 84% to 79% (sphericity 0 and 75, p = 0.03 and 0.07, respectively) with the use of CAD.
But there were no differences in sensitivity, specificity, or reader confidence between sphericity settings (p = 1.0, 1.0, and 0.11, respectively).
Although the use of CAD reduced specificity to some extent, the decrease in specificity was not related to the number of CAD false positives, the group reported. In addition, the use of CAD increased reading time as sphericity decreased, adding a median of 4.4 minutes and 2.2 minutes for sphericity settings of 0 and 75, respectively (p < 0.001).
"Per-patient and per-polyp sensitivities improved significantly when readers used CAD at either sphericity setting, with no significant difference between them," Taylor and colleagues wrote. "This result might be expected because the standalone CAD sensitivity was 100% at both settings. However, the standalone specificity varied widely, and our data suggest that even a mean of 42 false-positive marks per case did not negatively impact reader sensitivity."
The presence of numerous false positives didn't distract readers from CAD true-positive detections or diminish their trust in CAD detections (which might increase any tendency to incorrectly dismiss true-positive CAD detections).
However, they added, "it is generally preferable to minimize CAD-generated false positives because of the additional time devoted to their assessment, as confirmed in our study and other studies."
The causes of CAD false positives were similar between the two settings, with the main differences being the overall number, the authors wrote.
The researchers found no relationship between the number of CAD false positives and additional reading time after five minutes, a finding that suggests readers weren't prepared to "invest" more than five minutes of reading with CAD assistance.
"Perhaps the main point to be taken from our data is that CAD developers could relax their operating point, allowing more false positives in exchange for better sensitivity," they wrote.
Study limitations include the use of a single CAD system, the inexperience and lack of a uniform training protocol for the readers, and adjustment of the system to achieve 100% sensitivity at both sphericity settings.
The data suggest that a low number of false positives is necessary for a CAD system to be beneficial, the authors noted. "A particularly relevant example of the benefit of such a sensitivity-specificity trade-off is for reliable CAD detection of flat neoplasia, which generally necessitates a highly sensitive but less specific operating point," they wrote.
CAD also has the potential to increase the sensitivity of inexperienced readers for detecting polyps at virtual colonoscopy.
By Eric Barnes
AuntMinnie.com staff writer
June 12, 2009
Related Reading
VC's high positive predictive value important for colonoscopy follow-up, April 30, 2009
VC CAD plus 3D improves sensitivity for novice readers, April 21, 2009
VC CAD plus electronic cleaning boosts sensitivity, false positives, April 10, 2009
VC CAD improves results for readers at all levels, April 7, 2006
Copyright © 2009 AuntMinnie.com
