When relying on user action to mitigate risk, be extra vigilant
It is best to mitigate risk through safe design and manufacture. Relying on user action is less effective.
Imagine you have recently noticed that you frequently feel shortness of breath and rapid heartbeats. Sometimes you feel light headed and dizzy, as if you are about to faint. You feel anxious and often very tired1. It may be just the new stress in your life; after all you have been promoted and now have a lot more responsibility for driving the business.
But you don’t want to ignore it too much. You are worried it may strike you again at any time, without warning. You have read about it enough on the internet to know this could be serious next time. A heart attack, a stroke you may not have any time to recover from.
You decide to get it checked out, but your doctor cannot find anything wrong with your heart. Everything seems normal at the time of the test. You are in good shape and don’t have any prior history of heart problems.
Your doctor suggests you consider a new wearable sensor that will continuously monitor your heart for up to 14 days and record every single heartbeat. You can push a button if you feel any symptoms. It will transmit the signal to an independent diagnostic facility when you push the button, or if it automatically detects an abnormal pattern. They will send alerts and daily reports to your doctor’s office. At the end of the prescribed wear period, they will process the entire data set and send a full report to your doctor. It will help your doctor in diagnosing the problem and figuring out a treatment plan just for you.
In the meantime, you can maintain your active lifestyle, keep doing what you love, and even take showers or a bath, with the sensor still attached to your chest. It is like having a silent, always-on, sentry by your side!
“Wow, this is great! Sign me up doc!”, you respond. You can’t wait to get started. Good thing the clinic has the sensor readily available, and you walk out with one discreetly attached to your chest, doing its magic, silently recording every single heartbeat.
What you and your doctor don’t know is that there is a limit on the number of “abnormal events” that can be transmitted to the diagnostic facility. If this limit is reached, there is no more data transmission and the system practically shuts down. Your silent, always-on, security guard is still watching, but it can no longer send a distress signal for help when needed. This could be a serious problem in an emergency situation.
Designing and developing a medical device involves many trade-offs. Sometimes, there are unintended consequences.
The engineers who designed the system thought this would be an extremely rare situation because the transmission limit was set at a very high level. They made a trade-off between battery life and data transmission.
Well, as we know from Murphy’s law that anything that can go wrong, will go wrong - and it did go wrong much more frequently for this wearable heart monitor2 according to a recent FDA warning letter3.
As an example, the limit on 500 asymptomatic cardiac events was reached in just 4 days of wear in one case according to the adverse event report4 posted on FDA’s MAUDE database5:
…the device was activated on (b)(6) 2022 (and worn for approximately 5. 5 days of the 7-day prescribed wear-period.) for four days, the device transmitted 500 transmissions. During these four days, the following md notifications occurred by the idtf. Several mdns and daily reports were provided to the account until the max trigger limit was reached on (b)(6) 2022. Each device will transmit up to 500 asymptomatic cardiac events. This constraint is necessary to preserve battery life for the full intended wear-period. Patient-triggered symptomatic transmissions are still able to be transmitted beyond this limit by pressing the large central button located on the outer device housing….
In this case, the doctor’s office was notified that the device was approaching its asymptomatic transmission limit and a replacement device was urgently shipped to the patient. However, the patient did not wear the replacement device. They experienced a serious condition while wearing the original device, and were take to the hospital. Fortunately, they received timely treatment to stabilize their life-threatening condition.
In another similar situation, the patient unfortunately did not survive6.
When compiling the final report, an arrhythmia which met notification criteria (mdn) for slow atrial fibrillation (slow af) was found. The mdn was delayed as it was not transmitted to irhythm servers during the wear-period. When account was contacted to provide notification of the missed event, irhythm was informed that the patient had expired. The account stated the cause of death was unknown. Each device is intended to be worn for a 14-day period and is capable of transmitting 500 asymptomatic episodes. The first device was registered (b)(6) 2022 in clinic but was not activated until (b)(6) 2022 despite multiple attempts to reach the patient and account. On the (b)(6) 2022, irhythm dispatched a replacement device, as the first device was approaching its transmission limit of 500 asymptomatic notifications, and the first device was removed the next day on (b)(6) 2022 (worn for 25 days). The second device (delivered on (b)(6) 2022) was returned unused. Due to the limited information available, unintended use error cannot be ruled as a potential contributing factor to the reported death. A device evaluation of the worn device was completed, and no functional issues were observed.
These two cases illustrate how relying on user action to prevent the occurrence of harm can fall through the cracks. When the transmission limit is reached, there is a cascade of events that requires prompt action from multiple users, including the physician and the patient:
The diagnostic facility needs to alert the physician when the system is about to reach the transmission limit.
The physician is expected to act on this information, but precise action is not clear. Perhaps they need to request a replacement device, alert the patient and make sure the patient wears the replacement device before the transmission limit is reached.
A replacement device needs to be sent to the patient, but it is not clear who is primarily responsible. Perhaps the diagnostic facility escalates the issue to the manufacturer when they don’t hear back from the physician after multiple attempts to contact them. Perhaps someone at the manufacturer then urgently ships the replacement device to the patient.
The patient is supposed to replace the device, but it is not clear if they understand why, especially if they haven’t heard from their physician.
The manufacturer needs to know whether the replacement device is now active, but perhaps it is not clear if the patient is supposed to call them. Perhaps they will receive an alert when the replacement device has been activated, but it is not clear what action needs to be taken, and by whom, when the replacement device is not activated.
These are only a few examples of missteps and complications that can occur when the transmission limit is reached and the system fails to send an alert. There are many opportunities for mistakes, which may trigger different sequences of events eventually leading to a hazardous situation where the patient does not receive timely care.
Making quick judgments is not useful. Take a step back and understand the big picture first.
It may be tempting to conclude that engineers made an incorrect assumption when setting up the transmission limit. But we must avoid making such quick judgments because we don’t understand the full context and chronology of key decision points where trade-offs were made during design and development.
Designing and developing a complex medical device involves many trade-offs. Considering the fact that this device is intended for non-critical care patients where real-time monitoring is not needed7, their assumption and design choices seem reasonable, especially when they were trying to also optimize the battery life.
However, several proactive steps can be considered from the risk management perspective. A key point here is to understand that risk management is a total lifecycle process, and not a one-and-done exercise.
In this way, a good risk management system is the real silent, always-on, sentry protecting your patients and users!
Here are a few ideas to consider8:
Document assumptions. Continually review and evaluate during development and the post-market phase. As noted above, design and development of a medical device involves many trade-offs to achieve design objectives in the most optimal way. Many assumptions are made along the way, but they are not always documented. The rationale for key decisions and trade-offs are not clearly articulated and understood by everyone on the team. It may seem obvious to the design team that the transmission limit in this case is high enough so it will be extremely rare to occur when used as expected. But if this assumption is not clearly documented and understood by everyone, there may be surprises and unintended consequences in the post-market phase.
This is not to say that you should now create a procedural requirement to “document everything”. You need to exercise judgment and make it easy for everyone to document key assumptions and decisions when appropriate.
This documentation will provide appropriate context for evaluating early signals during the post-market phase.
Develop strong signal detection capabilities and a business process to facilitate quick decisions. A trend is not a signal. By the time you see a trend, it is probably too late. Looking for a signal can be like looking for a needle in the haystack, especially when you have a large volume of complaints.
There are different statistical techniques that can be used for signal detection. Time series analysis and disproportionality methods9 can be used. However, you must also create a business process to facilitate confirmation and escalation of signals for timely decisions. Signal detection is only a starting point; often you will find that statistical analysis will generate too many potential signals. Not all of them require immediate attention. The SNIP10 method of signal analysis can help.
Create a process that continually improves your knowledge of reasonably foreseeable misuses. It is important to realize that our foresight of potential misuse(s) is limited at launch. In this case for example, it is probably difficult to imagine how a non-critical care patient would reasonably experience more than 500 asymptomatic cardiac events during the maximum wear period of 14 days11. The most logical conclusion is that other factors in patient selection or use are causing the unexpectedly high rate of occurrence of the transmission limit.
If our signal detection capabilities are strong, we would notice it before it becomes a trend. We would assemble a cross-functional team to understand some of the contributing factors. We may find that doctors are prescribing our device to high-risk patients to avoid keeping them under constant observation at a healthcare facility. This is a type of an intentional misuse12 we may have overlooked in our initial risk analysis. What was not reasonably foreseeable at launch, should now be reasonable foreseeable.
We have to be extra vigilant when we notice an increase in the frequency of adverse events that do not seem to be associated with a device-related issue. We cannot simply file these events away as use-errors.
Create a dedicated post-market surveillance process to augment the complaints handling process. Complaints handling is a labor-intensive, complex process that also includes mandatory reporting of adverse events to regulatory authorities. In a situation, where complaints volume is high13, the complaints handling staff has no time to see the forest from the trees! They cannot notice a signal when they are busy addressing the regulatory requirements for complaints handling.
That is why it is very important to establish a separate post-market surveillance process to review and analyze all relevant data sources. It is best to assemble a cross-functional team to review and analyze data for identifying potential signals and confirm them using the SNIP process. These signals should be promptly escalated to key decision makers so they can assign appropriate resources for timely resolution.
Continually improve the robustness of your design against foreseeable misuse. As you gain new knowledge of foreseeable misuses, whether intentional or not, you should consider improving the robustness of your device through design changes and/or protective measures. Relying only on user action to prevent the occurrence of harm is not very effective as illustrated by this case. It may be that you consider the feedback from your post-market surveillance process to build a better, more robust product next time. Alternately, you can consider appropriate design changes in the existing product or other means of risk mitigation.
Key takeaway
Design and development of a medical device is a complex, iterative process which involves many assumptions and trade-offs. It is important to document key assumptions and design decisions to provide appropriate context for troubleshooting and improvements in the post-market phase. When you rely on user action to mitigate the risk of harm, you have to be extra vigilant and continually improve your foresight of use-errors and misuse(s) you did not anticipate at launch. That is why, you have to build a dedicated post-market surveillance process, separate from your customer handling process. This will allow you to quickly detect potential safety signals and analyze them to confirm and escalate for timely action. In addition, it is a good practice to create a feedback loop back to the design and development process to facilitate better, more robust products in the future.
These are common symptoms of cardiac arrhythmias when the heart is not beating in the proper rhythm. Serious, life-threatening conditions may occur if not diagnosed and treated promptly.
ZIO XT and ZIO AT monitors by iRhythm Technologies
FDA Warning Letter: iRhythm Technologies, MARCS-CMS-643474, May 25, 2023
Report Number 3007208829-2022-00048. Note, the redacted portion of the manufacturer’s narrative in the report is indicated by (b) (6) to protect confidential information.
FDA MAUDE: Manufacturer and user facility device experience searchable database.
Report Number: 3007208829-2022-00048.
Indications for use, K16352 Summary: The Zio QX ECG Monitoring System is intended to capture, analyze and report symptomatic and asymptomatic cardiac events and continuous electrocardiogram (ECG) information for long-term monitoring. While continuously recording patient ECG, both patient-triggered and automatically detected arrhythmia events are transmitted to a monitoring center for reporting. After wear, a final report is generated based on beat-to-beat information from the entire ECG recording. It is indicated for use on patients 18 years or older who may be asymptomatic or who may suffer from transient symptoms such as palpitations, shortness of breath, dizziness, light-headedness, pre-syncope, syncope, fatigue, or anxiety. The reports are provided for review by the intended user to render a diagnosis based on clinical judgment and experience. It is not intended for use on critical care patients.
For educational purposes only. These ideas should not be treated as providing any assurance, whether explicit or implicit, for regulatory compliance. They should also not be interpreted as potential gaps in the manufacturer’s current risk management and/or quality management system.
Data mining methods at FDA - white paper
SNIP: SNIP stands for Strength (S), Newness (N), Importance (I) and Preventability (P) of a detected signal. Each signal can be assessed for using the SNIP framework using a combination of qualitative and quantitative analysis criteria.
The transmission limit as reported in the FDA warning letter (see note 3 above) is 100 patient-reported (symptomatic) and 500 automatically detected (asymptomatic) arrhythmia events. Over a 14-day wear period, this translates to a daily average of approximately 40 such events.
ISO/TR 24971:2020 - examples of reasonably foreseeable misuse include 1) unintentional mistakes; 2) intentional acts of misuse, and 3) intentionally using a medical device for a purpose other than that intended by the manufacturer. Note that intentional acts of misuse are not necessarily by malicious intent.
According to the FDA waring letter (see note 3 above), the manufacturer reviewed a total of 999,328 during the period of March 1, 2019 through August , 2022. This translates to an average daily volume of approximately 800 complaints.
Great writeup highlighting limitations of the Zio monitor. The human heart beats, on average, 100,000 times a day. A 14-day zio monitor looks at about 1.4 million heart beats. From this perspective, a limit of 500 asymptomatic transmissions seems like a drop in the bucket. Another challenge is the "signal to noise" problem. How many of those asymptomatic transmissions are noise? For example, patient movement may create artifact that may trigger a transmission. The heart does not function as a steady state organ. It is very dynamic and asymptomatic "extra" or "missed" heart beats are fairly common. Many of these events may not be clinically relevant. Also to consider is the limited time a physician has to scan these reports. Perhaps AI can help separate "wheat from the chaff" and save providers time, provide more accurate and clinically relevant data, and improve the overall quality. Increasing the limit beyond 500 may overwhelm clinicians with more data, with the risk that important events may be missed.
I appreciate you addressing assumptions and documentation in this article. In either a small business or large, there will be employee turnover that results in lost knowledge and history. Small businesses grow and staff are promoted/move to better opportunities. Large businesses shift priorities and persons change departments. The undocumented rationale for design decisions gets lost during employee turnover in these cases. When new persons picking up the project they are unsure and unwilling to go against what was previously established (regardless of if the rationale is documented). We tend to assume that the last person did the work correctly and that it is important, we just don’t know why.
Suggested methods for documenting assumptions are through memos, meeting minutes, power points, or design tools stored. All of which should be stored in the DHF. It provides a glimpse at what design trade-offs were considered and why a choice was made. It also provides an opportunity for the rest of the team to weigh in and either validate the assumption or propose alternatives. In this manner the assumption will live on, and the team gets awareness to the assumption.