Case study: Using post-market data to evaluate changes in risk level

How you can estimate P1 and P2 from reported adverse events to estimate the probability of harm and monitor changes in risk levels.

What if we could develop a method to directly estimate a probability of occurrence of harm (POH) associated with device malfunctions using actual post-market data?

This could not only help us monitor changes in the risk level for an existing device, but it could also provide a set of baseline data to use in the development of the next generation of devices.

A common challenge during the post-market phase of a medical device is to determine if there is a change in previously estimated risk levels based on complaints and adverse events.

It is a requirement of ISO 14971, the international standard for application of risk management to medical devices, to review information collected during the post-market phase to determine if the risk levels have changed and if any individual risk is no longer acceptable¹. If risk levels have changed to a level no longer acceptable, then timely action is required to ensure that a medical device continues to remain safe and effective throughout its life cycle.

However, this proves to be a challenging exercise in the industry. One reason is the common practice of using an FMEA² as the only method of risk analysis, which fails to correctly and completely identify the risk of harm arising from each hazardous situation. A hazardous situation may arise even when a device is operating in the normal mode and there is no failure. Further, not all failure modes will necessarily result in a hazardous situation. Therefore a direct one-to-one link between failure modes and different hazardous situations is not feasible in an FMEA.

Another industry practice is to use the P1, P2 method³ of estimating the probability of occurrence of harm (POH) in an FMEA to comply with ISO 14971 requirements for risk analysis. FMEA is a technique for analyzing individual failure modes. Therefore, the estimated risk level arising due to a failure mode is applicable only to that specific failure mode/cause combination. In practice, a device malfunction may occur due to a combination of different failure modes through a sequence of events. Therefore, it is not feasible to isolate a specific failure mode/cause combination when a particular device malfunction is reported as a complaint.

As a result, it is generally not feasible to determine if previously estimated risk levels have changed based on actual complaints and adverse events. The best we can do is to monitor the frequency of adverse events and look for unusual trends or outliers.

It is like flying blind, with no way to course-correct in time, because we don’t know how far we have deviated from expected levels!

In this case study, publicly available data is used to develop a method of estimating POH using the P1, P2 approach. Although there are several limitations to this approach, it is one way of potentially creating a more direct link between risk estimation during product development and post-market surveillance.

Case Study: HeartWare ventricular assist device (HVAD) system

The HVAD system is indicated for patients with end-stage heart failure either as a bridge therapy to heart transplant (BTT), or as destination therapy (DT)⁴. As shown in the following figure, it includes an implantable mechanical pump to drive blood from the left ventricle to the body, a driveline that connects to an external controller, dual power sources and a data monitor.

Figure 1: HVAD system components. Image source: fda.gov

Medtronic acquired the HVAD system from HeartWare, International Inc. in 2016⁵ to expand its $5 billion portfolio of Cardiac Rhythm and Heart Failure products. But, in a short span of 5 years, Medtronic announced discontinuation of the sale and distribution of the HeartWare Ventricular Assist Device (HVAD) from the market. This announcement came after multiple recalls associated with a pump malfunction and a higher risk of death or other neurological adverse events compared to another commercially available device. In short, the benefit-risk evaluation of the HVAD system was no longer favorable to support a claim of continued safety and effectiveness.

In December 2020, Medtronic shared details of the pump malfunction involving delay to restart, or failure to restart after a pump stop event⁶.

An internal pump component from three (3) specific lots puts a subset of the finished pumps at higher risk of delay to restart or failure to restart. The risk exists only when the pump is stopped, for example, in a controller exchange when an attempt is made to restart the pump. A delay to restart or failure to restart could occur at any time after a pump stop, even if the pump initially started at the time of implant. If a pump has successfully restarted after a pump stop event, a delay to restart or failure to restart could be experienced in the future.

The implanted pump is expected to provide continuous circulatory support to the patient, and a delay or failure to restart after a pump stop event could lead to a life-threatening situation, including death. This issue is so critical that the system is designed to have multiple redundancies such as dual sources of power (AC or DC adapter plus a battery, or two batteries) to be connected to the controller at all times. The pump is also designed to run with dual stators, but it can continue to operate momentarily with a single stator when the electrical contact is interrupted.

Indeed, Medtronic reported 26 complaints between March 1, 2017 and November 16, 2020, which were directly attributed to this specific malfunction:

As of November 16, 2020, Medtronic has identified two (2) deaths, nine (9) cases of critical harm (such as cardiac arrest or reoperation for pump exchange), seven (7) cases of major harm (such as hospitalization or prolonged implant procedure due to interoperative pump exchange), and eight (8) cases of negligible harm (such as a potentially life-threatening event in which the patient recovered without long term effects, or a patient experienced a delay in implant).

A total of 506 HVAD systems were manufactured and distributed in this timeframe with the impacted components from the 3 specific lots. Medtronic estimated a failure rate of 5.7% for pumps in this subset, compared to a failure rate of 0.087% for general population pumps when operating normally (dual stator) and a failure rate of 0.4% when operating in single stator.

We will use this information to develop a method for applying the P1, P2 approach to estimate the POH. Let us dive in.

Let us first review a few basic concepts.

Concept 1: Harm occurs due to exposure to hazard(s) in a hazardous situation

According to ISO 14971, a hazard is a potential source of harm, and a hazardous situation is circumstance in which people, property or the environment is/are exposed to one or more hazards.

However, the link between hazard(s) and hazardous situation(s) is not immediately obvious. As a general concept, as illustrated in Figure 2 below, an initial event may trigger a sequence or combination of additional events that eventually lead to a hazardous situation here one or more hazards may become activated. It is the direct exposure to these hazards that leads to harm.

Figure 2: Linking hazards to hazardous situations. Image source: ACHIEVE

A key insight is that the link from a trigger event to hazardous situation is not always linear. After a foreseeable sequence of events lead to one hazardous situation, further intervention may itself lead to additional sequence of events and a new hazardous situation. As a result, identifying and outlining a sequence or combination of events becomes quite challenging in a linear analysis tool such as an FMEA. Clearly, there is more to hazard analysis than simply doing an FMEA!

A device malfunction, such as delay to restart or failure to restart following a pump stop event described above, is only an initial event. It may be a consequence of another sequence or combination of events triggered by one or more failure modes, including use-errors. In a fault tree analysis (FTA), this type of a device malfunction would be treated as a top event, related to a set of underlying potential contributing factors. The probability of occurrence of the top event is the combined probability of all events along one or more failure trajectories mapped in an FTA. It is not the same as the probability of occurrence of a single failure mode/cause combination in an FMEA.

Concept 2: P1 is the probability of occurrence of a hazardous situation not that of a failure mode

As illustrated in Figure 3 below, P1 is the combined probability of the initial/trigger event and the sequence or combination of all other events leading to a hazardous situation. It is not the probability of occurrence of a specific failure mode/cause combination in an FMEA.

Figure 3: P1, P2 and their relationship to the probability of occurrence of harm (POH). Image source: ACHIEVE

If we know the probability of occurrence of an initial event (P_initial) and the combined probability of occurrence of the subsequent sequence or combination of events (P_sequence), we can estimate the P1 by the following equation:

Concept 3: P2 is the probability of a hazardous situation leading to harm

Once a hazardous situation has occurred, the probability that harm may occur is now given by P2. Note that P2 is a conditional probability applicable only once a hazardous situation has occurred.

Figure 4: Concept of P2 as a conditional probability. Image source: ACHIEVE

As illustrated in Figure 4, P(B|A) is the conditional probability, P2, of a hazardous situation leading to harm. Like any other probability value, P2 value must be between 0 and 1.

In practice, a single hazardous situation may lead to harms of different severity levels. If a 5 point severity scale is used, a different value for P2 may be assigned to each of the 5 levels. However, the total of all these 5 P2 values must be equal to 1.

Concept 4: Probability of occurrence of harm (POH) is calculated as the product of P1 and P2

Using the rule of conditional probability, and the relationship illustrated in Figure 4, we can calculate the POH using the following equation:

Let us now revisit the HVAD case to develop a method for calculating POH using the P1, P2 approach

Figure 5 summarizes the information from Medtronic’s urgent medical device safety communication released in December 2020.

Figure 5: Summary of relevant information for P1, P2 method calculation method

Step 1: Calculate P1

If we consider the pump failure to restart or delay in restart after a pump stop event as the initial event, then we can use the pump failure rates provided above as P_initial. Another way to estimate P_initial is to use the following equation:

Here, the number of total opportunities is equal to the number of pumps in operation multiplied by an average number of pump stop events. Recall that a pump stop event can occur during a controller exchange or when both sources of power are disconnected from the controller.

In this example, we can directly use the failure rate information provided by Medtronic.

We don’t have a lot of information about different events in the sequence of events. If we make an assumption that when a pump failure occurs, a hazardous situation also occurs, then we could consider P_sequence as 1. Now, P1 is the same as P_initial according to the following equations.

We can now calculate the P1 values for the 3 scenarios based on reported failure rates:

Step 2: Calculate P2

As reported by Medtronic, there were a total of 26 events where a pump failure led to harm. Since P2 is a conditional probability, we can estimate P2 for different types of harms shown in Figure 5 using the count of reported events for each⁷:

Step 3: Calculate POH

Now that we have both P1 and P2 available, we can calculate the probability of occurrence of harm (POH) for each scenario using the following equation:

As an example, POH for death in the general pump population in the normal mode is equal to 0.000067, or about 7 in 10,0000, as shown in the following calculation:

We can now complete the POH calculation for the 3 cases as shown in the table below:

As shown in the table above, the probability of harm increases by nearly 2 orders of magnitude for Case 3 (subset of pumps with defective part) compared to the general population pumps operating in the normal mode with dual stator.

Step 4: Evaluate change in the risk level

Now, we can evaluate the change in the risk level against pre-defined criteria of risk acceptability to determine if the risk pump failure is no longer acceptable.

Here is an example of what a risk matrix for this analysis might look like⁸:

Note that the probability ranges corresponding to each of the 4 levels for the POH rank span a range of 3 orders of magnitude on a log scale.

Let us consider the calculated POH levels for Case 1 corresponding to the general population of pumps operating in normal mode with dual stators to be in the acceptable (green) zone. The change in POH level from Case 1 to Case 3 clearly shows that the risk level is now in the unacceptable (red) zone. As a result, additional actions need to be taken for both Case 2 and Case 3.

Limitations of using complaints data for estimating POH

There are several limitations of using complaints data for estimating POH for the purpose of evaluating changes in the risk level during the post-market phase:

• Generally, there is a lot of variability in complaints reporting. As a result, it is challenging to establish baseline failure rates for estimating P1 and P2.

• Estimating P1 and P2 values using complaints data is highly dependent on the time interval used for data analysis. As an example, the above data set corresponds to a time interval of more than 3 years. Estimated values for P1 and P2 may vary widely across shorter time intervals.

• Often, there is no direct link between a device failure and reported patient harm. As a result, it is challenging to estimate baseline P2 values. A common practice is to utilize subject matter expertise from clinical/medical experts to create a table of baseline P2 values corresponding to each of the harm severity levels. However, these baseline values need to be updated on a periodic basis using complaints data.

• Often, detailed information about the sequence of events leading to the occurrence of a hazardous situation is not available from complaints data. As a result, P1 values may be overestimated if the failure rate is used as a direct measure of P1.

• P2 values may be under or over-estimated, or unevenly distributed among harms of different severity levels due to variability in complaints reporting, and severity level assignments. This issue become more serious if standardized harm terms are not used to classify incoming complaints.

Key points

• If a direct link between device failure and patient harm is available from complaints data, we can estimate P1 and P2 values for different types of harms.

• We can estimate P1 using device failure rates, assuming the combined probability of the sequence of events as 1.

• We can estimate P2 values for each harm severity level, if we have a full distribution of harm events across all severity levels. P2 is the conditional probability applicable to a hazardous situation. As a practical matter, P2 values can be distributed across different harm severity levels, but they must all add up to 1 for a given hazardous situation.

• The probability of occurrence (POH) of harm at each severity level can be calculated as the product of P1 and P2 corresponding to that severity level.

• Changes in the POH level can be mapped on a risk matrix to compare them against a baseline to determine the risk of harm has changed to an unacceptable level.

• Risk of harm should be evaluated on a periodic basis using post-market surveillance information. As much as possible, a quantitative approach should be used to estimate changes in the risk level.

1. See Clause 10.3 in ISO 14971:2019

2. FMEA: Failure Modes and Effects Analysis, is only one of the several risk analysis techniques recommended by ISO/TR 24971:2010, the guidance document for implementation of ISO 14971 requirements. It is most suited to analyze potential failure modes, their causes and effects for the purpose of identifying and implementing control measures to prevent their occurrence. When used alone, an FMEA is not sufficient to identify and estimate the risk of harm in the context of ISO 14971.

3. See ISO/TR 24971:2020

4. FDA: HVAD Summary of Safety and Effectiveness Data (SSED)

5. See Medtronic Press Release, June 27, 2016.

6. See Medtronic Urgent Medical Device Safety Communication, December 2020.

7. Note that there are only 4 levels of severity in this example. Generally, a 5 level severity scale is used in the medical device industry. However, there is no specific requirement on the number of severity levels to be used for risk analysis. In this case, 4 levels are sufficient as reported by Medtronic in the urgent safety communication of Dec 2020.

8. Note: Risk matrix is shown only for the purpose of illustration. It is not intended to reflect the actual risk matrix used by Medtronic in their risk management system.