GenAI in Regulated Healthcare: The FDA and EMA Draw the Rules

TL;DR

On 18 August the FDA opened a discussion on evaluating generative AI-enabled medical devices, while EMA's draft Annex 22 keeps generative models out of critical manufacturing for now. The FDA paper proposes a competency-based approach: benchmark what a system knows and can do, then confirm it clinically. Both regulators are circling one idea: judge these systems by the performance they can demonstrate for a defined use. I like that approach: it rewards the teams doing the hard validation work.

The next phase of generative AI regulation is taking shape on both sides of the Atlantic.

On 18 August the FDA opened a discussion on how it might evaluate generative AI-enabled medical devices. In parallel, Europe is building draft Annex 22, which keeps generative models out of another regulated area, critical manufacturing, for now.

What EMA draft Annex 22 covers today

EMA's draft Annex 22 would set the rules for AI in medicines manufacturing. It covers the "traditional" machine learning pharma has used for years: a model trained on data, then held static in use, giving the same answer to the same input. Predictable, and checkable.

Dynamic, probabilistic and generative models, i.e pretty much everything we consider GenAI/agentic today, sit outside the scope for critical use. EMA is already considering how to bring them in. It held a two-day workshop this summer on the safeguards that might make it possible.

The FDA's device center took a different route. Its discussion paper considers a competency-based approach: evaluate the system the way you would credential a doctor. Benchmark what it knows and can do, then confirm it clinically. The paper even takes on agentic systems directly, asking how to handle the reduced room for human review.

Where the two approaches converge

These are different domains. Annex 22 covers the factory floor, while the FDA paper covers medical device software used in patient care. What links them is one idea both are circling: judge these systems by the performance they can demonstrate for a defined use. I really like that approach. It rewards the teams doing the hard validation work, and it gives newer models a real path into regulated settings.

Regulators have a great opportunity here. The next step is to define what good evidence looks like when outputs can vary in nondeterministic systems like GenAI: which competencies matter, how they should be tested, and where human oversight remains essential. Get that right, and newer systems have a credible path into regulated use without lowering the bar on safety.