Critical success factors for a Software as a Medical Device regulatory submission
In this article I’m going to focus on the top five success factors for medical device software (MDSW) submission, whether that’s for Software as a Medical Device (SaMD), including Artificial Intelligence/ Machine Learning software (AIaMD), or for Software in a Medical Device (SiMD)
Manufacturers need to perform Design Control – that is how their chosen Software Development Lifecycle (SDLC) methodology fits into the medical device quality management concept of Design Control.
Manufacturers need to perform Requirements Management – that is the use of formal requirements and requirements traceability.
Manufacturers need to perform formal Risk Management, going hand in hand with Requirements Management.
Manufacturers need to be prepared for the level of Verification and Validation evidence that they need to be able to submit.
Manufacturers need to understand that the SDLC does not end at first submission and certification – Software Maintenance under Change Control is an ongoing commitment for the entire lifetime of the MDSW.
How have I come to this top five? Well, at Hardian Health we are in the fortunate position of seeing a wide variety of approaches to regulatory submission from our various clients, whether for UKCA marking, CE marking or US FDA approval or clearance; we are also in the fortunate position of being party to feedback to manufacturers from the UK MHRA and other regulatory agencies such as the TGA in Australia and the FDA in the USA, as well as from CE Notified Bodies. We put each piece of feedback into our internal knowledge-base so that we can see the direction the various authorities are taking as they all develop and improve their feedback processes; from this knowledge-base, we are continually improving the support we can give to our clients.
This article expands on these five critical success factors, and is complementary to the article we published in 2023 about the standard for the development of medical device software, IEC 62034.
Design Control
Software development needs to contain elements of design input, design output, design verification, design validation, design review and design transfer. These are requirements of the medical device quality management system standard ISO 13485 (section 7.3) and of the current US regulation 21 CFR 820 (section 820.30) – all medical devices must be developed and maintained under design control, including SaMD and SiMD, which is a principle that goes beyond what the MDSW SDLC standard IEC 62304 covers. Whatever agile or other methodology is applied; these elements of design control still need to be apparent. At Hardian Health we have built a reference model Quality Management System including a Design Control procedure and a Software Development Plan template that supports our clients in mapping their chosen software development methodology into design control, to train their developers and represent to the regulatory authorities that design control is being applied.
Requirements Management
Software requirements need to be documented, and they need to be documented in a maintainable way; they also need to be traced down from overall product requirements and traced across to verification test protocols and the results of running these protocols. We find many requirements stacks made in spreadsheets, or other tools, where the various elements are manually traced together, without any reliable, automated, means for checking that the trace is complete; the result of this is that maintenance is difficult, including the change impact analysis i.e. if a requirement is changed, then what is the impact on the requirements surrounding it, the software and documentation already implemented, and the verification and validation tests that prove the correct implementation of the requirements.
IEC 62304 and IEC 82304-1 are international standards that list the types of requirements that apply to SaMD; additionally, there are SiMD requirements based on section 14 of IEC 60601-1. Hardian Health has created a reference model set of product and software requirements, as a baseline for customisation into a variety of Computer Aided Triage, Detection and Diagnosis (CADt, CADe, CADx) as well as Digital Therapeutics (DTx) and other applications of SaMD and SiMD. Our requirements tool, which is built as a Microsoft Excel spreadsheet, is portable to Google Sheets, and contains tools to ensure that the trace is complete and to facilitate change impact analysis.
Very briefly, from our experience of regulatory authorities’ feedback, we suggest in our reference model requirements stack numerous formal requirements covering several headings:
Functional requirements – we tend to group these into the lifecycle stages of software development and operation, namely:
Development (including the tools and datasets used to train AIaMD models)
Deployment, installation & configuration
Registration of users i.e. granting them access to executable software and data
Authentication of users (login, of human users via UI, or machines via API)
Input e.g. uploading radiology images
Processing e.g. what the AIaMD or other algorithm does to the input
Output e.g. the reporting that the clinical users see and use
Termination of user sessions (including inactivity logout, for security)
Revocation of registered users’ access
Postmarket Surveillance (PMS), data protection and technical support
Retirement of software
Retirement is a factor to consider from the start, to understand with SaMD or AIaMD in particular what will then be done to disposition stored patient data once the software reaches the end of its life – see below also about software lifetime.
Finally Labelling, as there are regulatory requirements that
humans can see specific information in the user interface (UI) and that
server software exposes similar information to client software across any Application Programming Interface (API)
Nonfunctional requirements or system attributes – we find the following often missed by manufacturers:
Software lifetime
All software has a lifetime – it is not indefinite – before retirement in favour of something new. The trick is to specify the lifetime correctly, taking account of regulatory requirements such as for the retention of data for specific periods; particularly if the lifetime of the software is less than the regulatory retention period, what then to do to keep any stored data readable once the software is retired.
Availability
Depending on the criticality of the use of the software, there may be particular requirements to maintain availability – think for instance what happens if a cloud-based software system used in a critical care setting becomes inaccessible due to internet problems.
Performance
What is the throughput of the software, in terms of how many input/ process/ output cycles per day/ week/ month/ year over the lifetime of the software? This then leads to considerations about how much storage capacity is needed for data…
What is the required response time of the software, including when a cloud or on-premises service is receding multiple concurrent requests for input/ process/ output?
Quality
What is the accuracy required of any algorithm, expressed as sensitivity & specificity, and or PPV/ NPV, that is non-inferior to the existing clinical state of the art?
Regulatory
Further to the above, what specifics do the regulations in various jurisdictions require, that need to be verified and validated?
Market Access
Not strictly for certification as a medical device, but what further requirements do specific customers or markets need to be satisfied? For example:
DTAC (Digital Technology Assessment Toolkit) for the UK NHS
HIPAA (Health Insurance Portability and Accountability Act) compliance in the USA
DiGA requirements for digital health apps in Germany, and similar in other countries
We also must consider intended use and product requirements in addition to software requirements. At Hardian Health, recognising that tiers of requirements can be called different things, we have made the following definitions:
Tier 0 requirements – meaning intended use and regulatory strategy
Tier 1 requirements – meaning product requirements, which we often render in user story language, e.g. “As a user, I need to be able to upload a DICOM package for analysis”
Tier 2 requirements – software requirements including those for accompanying documentation – so for the DICOM example above
Requirements for checking the format of the package, and rejecting it if in the wrong format
Requirements for what to write in the Instructions for Use about
What are the right and wrong formats of the DICOM package
What is the way that the user performs the upload
Requirements on how to ensure the cybersecurity of the DICOM package, whilst it is in transit (across a network) or at rest (in a database)
Also, how to de-identify and re-identify patients, in case there is a de-identification boundary to avoid patient identification information being transmitted
Tier 3 requirements – specifications e.g. for cloud platforms (virtual servers and database packages) and for the input devices (x-ray machines, MRI scanners, etc. that are compatible with the SaMD)
We suggest that user stories and tasks in an issue management system such as Atlassian Jira are not sufficient as software requirements, though we do tend to apply user story language to Tier 1 requirements.
Risk Management
Closely aligned with requirements are risks – as all risk mitigations or risk control measures (RCMs) are themselves requirements that need to be added to the requirements stack as described above. We need to analyse the risks that failures in the operation of the SaMD or SiMD will cause issues with safety (of patients and users), effectiveness and cybersecurity; we need to reduce as far as possible the likelihood of or impact to the patient, the user, property or the environment of any adverse event caused by the use of the SaMD or SiMD.
New for 2024, we have refactored our ontologies of harms, hazardous situations, hazards, and foreseeable sequences of events – all the elements called for in the international standard for medical device risk management ISO 14971 – using the following state-of-the-art sources:
ISO 14971 itself
ISO/TR 24971, guidance on the use of ISO 14971, which expands on cybersecurity
BS/AAMI 34971, guidance on the application of ISO 14971 to AI/ML systems
The IMDRF (International Medical Device Regulators Forum) Adverse Event Terminology which provides a standardised way of capturing hazards, hazardous situations and harms
During 2023 we have seen an increased emphasis from the US FDA and from EU Notified Bodies on cybersecurity threat modelling as part of SaMD and SiMD risk management. Therefore we have created and utilised tools for threat analysis and documentation based on the requirements of the USA FDA that were published in September 2023, as well as the CE marking guidance published in December 2019.
Verification and Validation
IEC 62034 requires that “The MANUFACTURER shall establish and perform a set of tests, expressed as input stimuli, expected outcomes, pass/fail criteria and procedures, for conducting SOFTWARE SYSTEM testing such that all software requirements are covered” (section 5.7.1); it goes on to state that “The MANUFACTURER shall verify that: a) all software requirements have been tested or otherwise VERIFIED; b) the TRACEABILITY between software requirements and tests or other VERIFICATION is recorded; and c) the test results meet the required pass/ fail criteria.” Our tools for requirements and risk tracing support our clients in showing the regulatory authorities that the trace is complete, from requirement through test protocol to test results for each successive software release.
Maintenance under Change Control
Once you have ‘version 1.00.00’ released for clinical use in a specific regulatory jurisdiction, you need to keep up with the concept of planning for successive releases. As part of planning each release, we advocate creating what we call a Change Control Record (CCR) which is a written memento of answers to questions that include the following:
From the overall product backlog of Problem Reports (PRs) and Feature Requests (FRs), which of these are going to be included in the release backlog – and whose decision is it to include these?
Given the included PRs and FRs, what is the extent of the release?
Is there a change in intended use or basic design (what we call a Tier 0 change) requiring the release to be re-certified?
Is it a change in the Tier 1 requirements, as above, requiring the release to be re-validated for safety (including usability), effectiveness and/ or security, but not re-certified? (What the US FDA call a ‘Letter to File’ change.)
Is it a change in the Tier 2 requirements that do not change the software’s behaviour as far as the users can see, or Tier 3 minor bug fixes or security patches, requiring these changes to be re-verified as being effective, but not re-validated or re-certified?
What version number will the release take?
We promote the use of Semantic Versioning, with a particular meaning to each field x.yy.zz:
x for Tier 0 changes
yy for Tier 1 changes
zz for Tier 2 and Tier 3 changes
Consequently, what technical and clinical documentation will need to be revised so the Medical Device File (MDF)/ Design History File (DHF) will be kept up to date with the content of the release? For example:
Requirements Specifications – certainly, except maybe for Tier 3 patches
Risks Analysis and Risk Management Report – almost certainly for Tier 0, 1 and 2 changes
Clinical Evaluation Report – certainly for Tier 0, maybe for Tier 1
Instructions for Use and other accompanying documentation – almost certainly for Tier 0 and Tier 1
Verification and Validation protocols and reports – verification and validation for Tier 0, 1 or 2 changes, but verification only for Tier 2 or 3 changes
Design Review Reports – certainly for all tiers of change, to finally sign-off the changes made
We find this CCR concept so useful that we even apply it to ‘version 1.00.00’: even the first release of MDSW is in effect a Tier 0 release.
At Hardian Health we can support you to assure regulatory compliance through the entire software development lifecycle, from fundamental planning, through development, initial certification and deployment, and ongoing maintenance all the way to retirement and withdrawal from market of your medical device software.
Hardian Health is a clinical digital consultancy focused on leveraging technology into healthcare markets through clinical strategy, scientific validation, regulation, health economics and intellectual property.
