Why Intended Clinical Benefits Fall Short in the Real World
Medical device software can perform well in testing and still fail in the clinic. We look at why intended benefits go unrealised and what developers and clinical teams can do about it.
Fundamental to all medical devices is the intended benefit they claim to offer to the end users, which manufacturers define at the very start of the regulatory pathway. More medical device software (MDSW) tools are being produced with the goal of reducing physician burnout, streamlining administrative tasks and speeding up diagnosis. But these claims often go unrealised in practice. This insight piece discusses the common reasons for this and potential solutions, grounding the discussion in the regulatory frameworks and annotating it with an example.
The Regulatory Landscape
The Medical Device Coordination Group (MDCG) 2020-1 guidance outlines the evidence requirements for the EU Medical Device Regulation (MDR). There is a clear distinction in this guidance between technical performance and clinical performance. Technical performance is the ability of an MDSW to accurately and reliably process input data into its target output. Clinical performance, on the other hand, is more pragmatic and relates to the ability of an MDSW to yield clinically relevant outputs that support the clinical benefits defined in the Intended Use. While a product can have high technical performance statistics, it's only valuable in the real world if the clinical performance is also present and stands up to the variability of healthcare.
To illustrate the problem, let’s look at AI scribes. Scribes have seen a large uptake since their introduction. A recent study found that 40% of UK GPs surveyed currently use AI scribes and a further 23% have used them in the past. However, another recent paper looking at scribe usage across multiple sites found only a modest decrease in documentation time associated with scribe usage and no significant change in electronic health record (EHR) time outside working hours. While this is not the whole story and scribes may demonstrate other benefits, such as increased engagement with patients, the productivity gains may not be what we predicted. This is a problem that may affect many software and AI medical devices, including more established radiology AI devices.
The MDCG guidance (2020-6) also notes that clinical benefit can be direct or indirect. Medical devices usually deliver direct clinical benefits, but MDSW may instead have an indirect clinical benefit where they do not claim measurable patient outcomes. MDCG 2020-1 instead judges clinical performance on demonstrated reliable use and usability:
"…for MDSW not claiming CLINICAL BENEFITS that can be specified through measurable, patient-relevant clinical outcome(s), clinically relevant outputs are achieved through demonstrated predictable and reliable use and USABILITY" (MDCG 2020-1, section 4.1, p. 11).
The Disconnect
High technical performance doesn't guarantee clinical benefit, and that gap can generally show up in one of three main places: (a) poor human factors engineering and usability; (b) a disconnect between product design and clinical workflow; (c) over-reliance on performance data.
The first way in which medical devices fail in the real world is through poor usability. The administrative burden on healthcare professionals is increasing as new software tools are introduced into practice. Frequently, this creates convoluted workflows with multiple login steps, clumsy user interfaces and unclear warnings. Products that are meant to simplify workflows can end up creating extra work, increasing cognitive burden and making the job harder, which leads to them not being used long-term.
Second, when developers don't have a clear enough understanding of the clinical workflows in which the end product will exist, that gap constrains the product's value. Too often, the product comes first and is then forced into a clinical workflow. One example, in the context of AI scribes, is poor integration into EHRs, which extends the normal workflow for clinicians by adding an extra step of copying over text from one system to another.
The third failure point is that some developers rely solely on standard performance data to demonstrate a product's effectiveness. High sensitivity or area under the curve (AUC) metrics are necessary but not sufficient. Equally important is measuring outcomes that apply to a device within its workflow and how that affects patients and the clinical benefits that are defined in the Intended Use. A strong AUC on retrospective data does not show the device improves patient outcomes once live. For example, Epic reported an acceptable AUC for its Sepsis Model in its own testing, but an external validation study found inferior performance in the real world. The authors note this creates an increased risk of alert fatigue, which may stop clinicians from realising the intended clinical benefit.
The Solutions
There are several ways to tackle this problem, but fundamentally, manufacturers must consider workflow design from the very start of product development. They should also prioritise early interaction with key stakeholders, such as clinicians and IT teams, during the design phase to ensure eventual product-market fit. Testing throughout the design process should then confirm that the product continues to meet user needs. Manufacturers can do this through early expert input, regular focus groups and formative testing. Usability testing should also be baked into the clinical evaluation plan and report. In addition, developers must follow human factors engineering standards such as IEC 62366-1.
The Intended Use is the backbone of medical device regulation. It should accurately reflect the true context of use, the target users and the use environment, providing a central frame of reference throughout design and testing. This Intended Use should then feed into study design that demonstrates a clinical benefit, and aims to replicate the environment in which clinicians will use the MDSW.
Ensuring usability goes beyond the point at which the device is certified. Manufacturers should generate post-market real-world evidence through post-market surveillance and prospective studies that can later feed into new versions of the device and help resolve workflow issues where they exist. This is especially important given how much clinical workflows vary and change over time.
Finally, staff training and change management, while they largely sit outside of medical device regulation, are important to consider as they substantially affect the success of a device within a local deployment.
Conclusion
So, ensuring an MDSW product meets its potential starts with a clearly defined intended purpose and clinical benefit(s), which should anchor design, testing and evidence throughout development. Early involvement of key stakeholders is critical to achieving good product-market fit. Likewise, product design and medical device regulation require a holistic approach, taking into account the technology, its usability and its integration into clinical workflows at all times. Meeting regulatory requirements is the minimum; designing for real-world use is what makes a device worth buying.
Contact us to find out more about how Hardian can help with mapping relevant clinical intended benefits to strengthen your clinical validation and claims.
