Commercially available chest x-ray algorithms require more research, an influential government-backed U.K. health technology agency has recommended. AI developers must conduct prospective studies to address gaps in evidence, it says.
Prof. Fergus Gleeson is one of four U.K. radiologists on the specialist committee for the NICE report. Photo courtesy of National Consortium of Intelligent Medical Imaging.The National Institute for Health and Clinical Excellence (NICE) issued a 24-page report, "Artificial intelligence-derived software to analyze chest x-rays (CXRs) for suspected lung cancer in primary care referrals," on 28 September. NICE provides national guidance and advice to improve health and social care, and is an executive public body supported by the Department of Health and Social Care.
The authors noted that more work is needed on 14 products: AI-Rad Companion Chest X-ray (Siemens Healthineers), Annalise CXR (Annalise ai), Auto Lung Nodule Detection (Samsung), ChestLink (Oxipit), ChestView (Gleamer), Chest X-ray (Rayscape), ClearRead Xray (Riveraintech), InferRead DR Chest (Infervision), Lunit INSIGHT CXR (Lunit), Milvue Suite (Milvue), qXR (Qure.ai), and Red dot (Behold.ai), SenseCare-Chest DR PRO (SenseTime), and VUNO Med-Chest X-Ray (VUNO).
NICE thinks more research should be done on these outcomes:
- The impact of the software on clinical decision-making and the number of people referred to have a chest CT scan, as well as the impact on review and reporting time, and time to CT referral and diagnosis
- How using the software affects healthcare costs and resource use, plus the technical failure and rejection rates of the products and patient perceptions of using AI-derived software
- The diagnostic accuracy of AI-derived software alongside clinician review in detecting nodules or other abnormal lung features that suggest lung cancer
- The diagnostic accuracy of using AI-derived software alongside clinician review for identifying normal x-rays with high confidence and the impact of this on work prioritization and patient flow
- Whether the software works in cases when it is hard to get high-quality images (e.g., scoliosis and morbid obesity), and whether it works in groups that could particularly benefit, including people with multiple conditions, people from high-risk family backgrounds, and younger women who do not smoke.
Access to the technology should be through company, research, or non-core NHS funding, they recommend. "Centres already using AI-derived software to review chest X-rays in adults referred from primary care may continue to do so but only under an appropriate evaluation framework and only alongside clinician review. They should do more research on the outcomes."
Lack of data and evidence
The NICE report has already brought a strong response from radiology AI experts.
Dr. Hugh Harvey.Dr. Hugh Harvey, managing director of Hardian Health, the clinical digital consultancy firm, said he wasn't surprised by the main findings, given the lack of human-AI prospective evaluation. "There is very little health economic evidence for cost-effectiveness to support the adoption of these technologies -- something which hopefully vendors will now start to produce," he told AuntMinnieEurope.com.
Harvey wrote a blog on this topic in April 2022 ("Autonomous AI - too good to be true?"), in which he argues that "one of the 'low-hanging fruits' of radiology AI has long been assumed to be separating normal from abnormal CXR studies."
Prof. Erik Ranschaert, PhD, past president of the European Society of Medical Imaging Informatics (EuSoMII) and visiting professor at Ghent University in Belgium, is concerned that the goals, intentions, and phrasing of the NICE report are unclear.
"What do they mean by the title? Does it mean that only patients with suspicion of lung cancer should be analyzed with AI? Or is this for using AI in the context of lung cancer screening with CXR?" he asked.
Ranschaert also points out that all studies referred to in the report were retrospective. "Few studies thus far have compared AI to a radiologist in a setting where the radiologist has access to the patient's clinical information, as would be the case in real life; this always gives the radiologist an 'advantage' over the self-contained AI solution."
There is an unmet need to further test these applications in clinical scenarios to evaluate AI solutions, he continued. "Such studies should be performed on a large population group in a PROSPECTIVE manner. The retrospective studies are usually conducted in a research-type environment."
Optimum use of AI
It would be more efficient to use the AI solutions for "filtering out" the normal CXR exams instead of detecting and showing the radiologists the abnormalities, according to Ranschaert.
"This is the best way to go if CXRs are used for lung cancer screening purposes (detection of nodules)," he said. "In the context of lung cancer screening, this could help for analyzing huge volumes of CXRs. In screening populations, the normals are usually much higher in number. There's already a huge backlog of CXRs in the U.K., so imagine if this had to be done manually for screening purposes."
Reading examinations at very high speed also causes errors, he stated. "CT would probably be more efficient (and accurate) in detection small nodules for cancer screening, but the disadvantage would be the cost, and you would need automated AI to facilitate the reading of all those scans."
Additionally, Ranschaert wonders whether the main goal of the authors of the report is to improve the service to patients and to get rid of the large backlogs of tests -- "Or is it more about cost reduction than service improvement?"
Input of specialist committee
The value and extent of the contribution of the experts to the NICE report might also be questioned, Ranschaert noted. Also, the inclusion of health economists, informatics specialists, and IT managers would have helped. "Multidisciplinary perspectives usually are more useful and realistic," he commented.
The seven-member specialist committee included four radiologists: Dr. Alberto Alonso, from Manchester University NHS Foundation; Dr. Christopher Johns, from Northern General Hospital, Sheffield; Dr. Robin Alcock, from Salisbury District Hospital; and Prof. Fergus Gleeson, from Oxford University Hospitals. The rest of the committee comprised two respiratory medicine specialists and a lay member.
According to another source, the important thing to remember is that chest x-rays are of very limited value for diagnosing lung cancer. "I always say, if you suspect lung cancer clinically, then do a CT; if you want to miss cancers, then do a CXR."
You can download a free copy of the NICE report here.
