World's First Regulatory Clearance for a Large Language Model Medical Device
Recently, we hosted an exclusive webinar featuring Dr Vera Roedel (CEO) and Professor Heinz Wiendl (Co-Founder) from Prof.Valmed®. They have achieved the significant milestone of securing the world's first regulatory clearance for a large language model (LLM) as a medical device.
We know how tricky it can be for healthcare companies to get their heads around regulatory compliance, especially when you're dealing with AI. So when we heard that Prof.Valmed® had been successful and secured CE marking as a Class IIb device, we knew we had to get them to share exactly how they achieved it.
The regulatory puzzle that nobody had solved before
AI medical devices present a fundamental challenge. The regulations simply weren't written with them in mind. Traditional medical device frameworks assume you're dealing with static code and fixed datasets, but AI systems are constantly learning and adapting.
Prof.Valmed® had to work incredibly closely with their notified body, MDC, to write the rulebook as they went along. The whole process took 18 months of back-and-forth discussions, with thousands of pages of documentation and countless meetings to convince regulators that their approach was robust.
"We had to convince them that this process makes sense... this was the first example of how you would do such a new class of products." – Dr Vera Roedel
So what exactly did they build?
Prof. Valmed is a highly intelligent "medical copilot" for healthcare professionals that can instantly find relevant information to help with diagnostic and treatment decisions.
They've structured their system around a closed database containing 2.5 million carefully curated medical documents. This includes guidelines, regulatory documents, handbooks and the full spectrum of medical literature. They've also integrated PubMed and Cochrane databases.
The system uses GPT 4.0 as its underlying model. The LLM functions as a search and summary tool that pulls information from their curated database. As Heinz explained, "the LLM is only used for the conversational and the combinatorial power, not for the quality of the documents."
This approach was necessary for receiving regulatory approval because it prioritises reproducibility and dramatically reduces the hallucinations that LLMs are infamous for.
The clinical evaluation masterclass
Now, this is where it gets really interesting. With no playbook to follow, Prof.Valmed® had to invent their own approach to clinical evaluation. They came up with a three-phase process::
Phase 1 focused on technical robustness. They stress-tested the system by asking it hundreds of questions multiple times to check for consistency.
Phase 2 tested the system against board-certified specialists across multiple medical specialties. The goal was to prove that their AI performs at the level of qualified doctors.
Phase 3 brought in real-world testing with medical professionals from different backgrounds, making sure the system worked for everyone from GPs to specialists.
The numbers demonstrate the system's effectiveness: around 900 questions tested in total, with a safety index of 0.26. This means the risk of potential harm if someone followed the AI's advice directly was very low.
One thing that really stood out was their pragmatic approach to the underlying technology. As Vera mentioned, they're completely transparent about using GPT 4.0 as "software of unknown provenance." They don't pretend to control every aspect of it, but they've built robust processes around it.
Key learnings from the Q&A session
During the Q&A, we got some brilliant insights that any company thinking about AI medical device approval should take note of:
Get talking to your notified body early – don't wait until you think your product is perfect
Be completely transparent about what you're doing and why. Regulators appreciate honesty, especially when they're navigating new territory alongside you.
Make sure your quality management system can handle AI's evolving nature. You need processes that allow for improvements without starting from scratch
Forget traditional metrics – AI systems need new approaches to clinical evaluation
Safety first, always. If in doubt, the system should say "I don't know" rather than guess
What does this mean for everyone else?
Prof.Valmed® is already being used in clinics, hospitals, and pharmaceutical companies, particularly for medical education and information support.
But the bigger picture is what this achievement means for the rest of the healthcare AI community. Prof.Valmed® has proven that regulatory approval for LLM-based medical devices is possible. The process requires careful planning, innovative thinking, and a great deal of patience, but it can be achieved.
The regulatory landscape is evolving, and Prof.Valmed®'s success has created a pathway that others can follow. As more companies take this route, we'll likely see the development of more standardised approaches to AI medical device evaluation. Overall, the experience shows that regulators support innovation when companies demonstrate careful consideration of safety and effectiveness.
With the right approach, regulatory approval for AI medical devices is achievable.
And if you're thinking about taking the plunge yourself? Well, you know where to find us.
Hardian Health is a clinical digital consultancy focused on leveraging technology into healthcare markets through clinical strategy, scientific validation, regulation, health economics and intellectual property.
