What issues should you consider when buying artificial intelligence (AI) software for radiology? A multinational team of authors has produced new guidelines -- including a list of top 10 questions to ask -- aimed at helping guide prospective purchasers.
The framework aims to help radiologists assess and choose commercial radiology AI software that will best fit their needs, and it was published in an open-access article online on 5 March in European Radiology by a group led by Dr. Patrick Omoumi, PhD, of the University of Lausanne in Switzerland. The team dubbed the guidelines the evaluating commercial AI solutions in radiology (ECLAIR).
Thanks to unprecedented investment and activity in private and public companies, a wide range of commercial AI-based offerings are now available for sale, according to the authors. Regulation of the software has also evolved.
As a result, radiology practices need to learn how to properly evaluate these tools, they said.
"While several guidelines describing good practices for conducting and reporting AI-based research in medicine and radiology have been published, fewer efforts have focused on recommendations addressing the key questions to consider when critically assessing AI solutions before purchase," Omoumi et al noted.
Many factors need to be included in the evaluation of these complicated software applications, such as assessment of technical and financial considerations, as well as quality and safety factors, according to the authors. Input from key stakeholders must also be included.
"Topics to consider in the evaluation include the relevance of the solution from the point of view of each stakeholder, issues regarding performance and validation, usability and integration, regulatory and legal aspects, and financial and support services," the group wrote.
In their article, the authors provided a detailed checklist of points to consider when assessing commercial radiology AI software, as well as a top 10 list of questions:
- What problem is the application intended to solve, and who is the application designed for?
- What are the potential benefits and risks, and for whom?
- Has the algorithm been rigorously and independently validated?
- How can the application be integrated into your clinical workflow and is the solution interoperable with your existing software?
- What are the IT infrastructure requirements?
- Does the application conform to the medical device and the personal data protection regulations of the target country, and what class of regulation does it conform to?
- Have return on investment (RoI) analyses been performed?
- How is the maintenance of the product ensured?
- How are user training and follow-up handled?
- How will potential malfunctions or erroneous results be handled?
"Although some assessment criteria presented here may not apply to every situation, we hope to have developed a framework that will allow all stakeholders to conduct relevant discussions with manufacturers and reach an informed decision," the authors concluded.
