FDA pushes makers of AI devices to disclose more details on testing, performance

STAT News – Thursday, January 9, 2025
By Casey Ross

The Food and Drug Administration wants the developers of medical devices that rely on artificial intelligence to disclose much more detail about how their devices were developed and tested, and what must be done to guard against safety risks in medical settings.

In a new draft guidance, the FDA calls on makers of AI devices to describe the sources and demographics of data used to train and validate their products, and disclose blindspots and potential biases that might impair performance. The information would be included in applications for approvals from the agency.

Although the document is advisory in nature and does not impose new rules on device makers, it aims to set a higher bar for companies that, until now, have gained approvals for AI products without fully describing their training, testing, and limitations. It remains to be seen whether its recommendations will be endorsed by the incoming Trump administration, or what level of cooperation the agency will ultimately get from industry.

The FDA has approved more than 1,000 medical devices that rely on artificial intelligence since the late 1990s, with the vast majority of those approvals coming in the last several years. A STAT investigation published in 2021 found that approvals were based on widely divergent amounts of clinical data, and that public summaries of approved products often lacked key details about their testing, such as how they performed in different racial subgroups.

Stanford researchers who examined AI devices cleared by the agency also spotlighted that most approvals were based on retrospective testing and were evaluated at a single site, raising concerns about hidden biases and other vulnerabilities that might hurt their performance in real-world medical settings. As a result of these and other challenges, many AI devices approved by the agency have failed to gain widespread adoption by doctors and provider networks, despite device makers’ promises that AI would help improve many aspects of medical care.

James Zou, a Stanford scientist who led that research, said makers of AI devices have begun to more regularly report sample sizes and the results of testing at multiple sites and within demographic subgroups. But he noted that evaluation of AI devices remains almost entirely retrospective, meaning that performance is evaluated on historical data, but not within a live setting where other factors can bear on the AI’s utility and effects on patient care.

“If it’s supposed to be a decision support tool what you really care about is does this actually help the clinicians to make better decisions,” Zou said. “And that’s something that you really cannot easily infer from a standalone retrospective analysis.”

The FDA guidance does not specifically call for more testing in live clinical settings as a condition of getting approvals. But it does recommend that device makers submit a plan for monitoring the performance of their devices in clinical settings after they are cleared for commercial use. The plan would identify potential changes in performance and establish protocols for monitoring the device’s accuracy and intervening if its outputs become faulty.

The draft guidance, published earlier this week, lists recommendations for disclosures across the stages of a product’s development and use. It calls on device makers to describe the sources of their data, sample sizes and methods of clinical studies, and the limitations and risks of their products. It also asks for those details to be included on an AI device label that would include additional information on the product’s performance across different demographic groups and types of patients.

Additionally, the document calls for much of the above information to be included in a public summary to be posted on the agency’s web site. The FDA noted in its draft guidance that the public is clamoring for more information about how AI devices are being evaluated and applied in their care.

“The public has consistently called for additional information about how FDA makes authorization decisions about AI-enabled devices, as well as more information about the design and validation of these devices,” the draft guidance states, adding that such information would improve the public’s understanding of AI and its overall trust in the technology.

Zou said improving education around the design and use of AI is especially important as it gets embedded into different aspects of medical decision-making and patient communication. The draft guidance is primarily focused on machine learning and predictive tools designed to help caregivers detect and diagnose illnesses.

But Zou said the FDA also must grapple with the advent of generative AI, an increasingly powerful subtype of the technology that can analyze multiple forms of data and produce responses to a range of prompts. The technology, increasingly used to communicate with patients, poses much different challenges than the task-specific tools the FDA is used to regulating.

“If I have a chatbot, and more generally an AI agent, it’s actually very different from a tool,” Zou said, noting that it can be used to analyze several different sources of information at once. “It’s really much more of a teammate, in the form of an AI agent. All the things in this guidance need to be revamped for this coming wave of agentic AI.”

The FDA’s digital health advisory committee recently held a meeting to discuss the uptake of generative AI and the regulatory implications of its expanded use. “It’s a whole new world that I believe is going to be coming to us very rapidly,” Zou said.

Access this article at its original source.