Should breast radiologists undergo self-assessment test?

A national breast cancer screening self-evaluation test implemented in Italy is showing signs of success, and the developers are convinced readers should undergo such a test. However, some experts warn the Italian initiative needs to be improved.

In January 2018, the Italian Society of Medical Radiology (SIRM) introduced a national online test for the self-evaluation of screening mammography interpretation. A group of authors led by Dr. Beniamino Brancato, from the Unit of Breast Imaging at the Institute for the Study and Prevention of Cancer (ISPRO) in Florence, analyzed the results from the SIRM test, assessing the diagnostic performance of Italian breast radiologists and its potential correlations with several individual characteristics.

User interface of the self-evaluation test. Initial case presentation shows the four standard mammographic projections side by side. The left column allows the reader to browse through the 132 cases, identified with coded names. The toolbar at the top provides standard tools for image interpretation (brightness and contrast, magnification, drag) and classification (region of interest drawing and editing, case forward and back buttons). Figures courtesy of Dr. Beniamino Brancato and European Radiology.

During the first 13 months, a total of 685 radiologists registered on the test organizers' website. Of these people, 502 (73.3%) started the test, and 342 (49.9%) completed it, Brancato and his colleagues reported in an article posted on 4 September by European Radiology.

A total of 236 (69%) of the 342 radiologists who completed the test were women, and they had a mean age of 46 ± 12 years (range, 30-78 years) and a median of eight years of breast imaging experience.

Participants who completed the test interpreted a median of 1,501 mammographic exams each year and had interpreted a median of 1,372 mammographic exams the year before taking the test. Most of these radiologists (235/342, 68.7%) worked in a public hospital, while involvement in the interpretation of population-based organized screening mammography was reported by 169 (49.4%) out of the 342 radiologists.

On their first attempt, 98 (28.7%) of the 342 radiologists passed the test. After the first failure, 138 (56.6%) of the 244 remaining radiologists tried the test a second time, and 49 (35.5%) out of 138 passed it. Of the 89 (64.5%) out of 138 radiologists with an unsuccessful second attempt, only 41 (46.1%) tried a third time, with a 9.8% pass rate.

Considering the best result of the first two attempts, 43% (147/342) of radiologists passed the test. Taking into account the best result of all three attempts, 44.2% (151/342) of radiologists completed the test successfully. Given the small number of radiologists with three attempts, further analyses focused only on the first two attempts, considering the one with the best result, according to Brancato and his colleagues.

Example of mediolateral oblique projection magnification. Region-of-interest delineation in a magnified craniocaudal projection. No more than one region of interest can be placed on each projection. If a case is left with a region of interest placed on a mammographic projection, it will be classified as true positive if the region of interest matches the one previously delineated during dataset building for the actual presence of carcinoma; otherwise, the case will be classified as a false positive. To store the case as a negative one and move on to the next one, the reader can click on the button with white arrowheads on green background (shown at the right end in top figure [Example of mediolateral oblique projection magnification]). The case will be classified as true negative if carcinoma is not actually present, otherwise as a false negative.

"This study confirmed that having specific breast imaging experience improves, but does not guarantee, diagnostic accuracy in screening reading," they wrote. "This national self-evaluation test, due to its easy use and the validity of the obtained results, can represent a useful tool to assess and foster the improvement of breast radiologists' skills, regardless of their degree of experience, and could be included as a criterion for the accreditation process of breast units."

Feedback from experts

Two breast imaging specialists have given a cautious welcome to the Italian initiative, but they suggested that improvements were needed.

Dr. László Tabár, professor emeritus of Uppsala University in Sweden, told AuntMinnieEurope.com he was pleased to see that the Italian authors used the recommended reading protocol of enlarging every mammogram while reading and if necessary, using window leveling (a viewer setting affecting the range of grayscale in the image).

However, he expressed concern that "the number of cancer cases is not sufficient to cover all the important imaging biomarkers." Most importantly, the authors did not include any cases of extensive architectural distortion, which is a particularly tricky and dangerous disease.

"The cases named 'calcifications' are not good enough," Tabár said. "Further specification is needed -- and definitely further detailed description of the proportion of stellate and circular tumors. We do not know the size of those tumors. What if the tumor mass is 2.0 cm?"

Overall, he thinks the Italian project represents a good attempt, but it now needs to be fine-tuned, particularly concerning the quality and number of the cases.

"How many of the cancer cases were hidden in dense breast tissue, how dense? I totally miss the diffuse cancers," he said. "Their 'test-case kit' needs to have a proper proportion of aggressive cancers that must be detected early, and it should also include the very deceptive huge architectural distortion with high risk of fatality."

When asked by AuntMinnieEurope.com about the Italian initiative, another source said it was a positive development but not an original one. For instance, the U.K. has the Performs test, which is mandatory for reporting of screening mammograms. Arguably, the U.K. test is superior to the Italian scheme because it is carried out on a real workstation rather than on a low-resolution PC monitor, the source said.