Beware the severity bias when analyzing risks

A recent poll indicates the dominant view on medical device risk as primarily driven by severity of harm.

What you can’t measure, you can’t manage.

There is some truth to this maxim often attributed to Peter Drucker, the legendary management guru¹. We need to be able to tell where we are in relation to where we need to be, so that we can course-correct in time and achieve our objective.

When it comes to risks, especially those that are related to safety of people, our objective is to reduce them to a sufficiently low level that is as far as possible without negatively impacting the anticipated benefits² of our medical devices.

But how low is low enough? How far is far enough in “as far as possible”?

Practitioners of risk management in the medical device industry struggle with this question. An individual risk associated with the use of a medical device is defined as a combination of the probability of harm and the severity of that harm³. But how do you combine the two? Unlike the risk of financial loss/gain⁴, for example, we cannot use a mathematical equation that connects the probability of harm (POH) to the severity of harm (S) to provide a value for the risk level.

Current industry practice is to assign 3-5 qualitative levels for the probability of harm (POH) and the severity of harm (S)⁵. Generally speaking, it is not possible to assign a numerical “loss” value to severity of a specific risk, especially when the final outcome could be death. On the other hand, it is possible to assign numerical values to probability ranges corresponding to each of these qualitative levels.

That is why risk management in the medical device industry is challenging. We don’t have a good way to measure risk that can be used to compare and evaluate different types of risks. Further, we are not able to quantify, with sufficient precision, the effect of our risk control measures.

Our current approach is to use a two dimensional matrix where different risks can be graphically represented based on the combinations of POH and S levels assigned through a qualitative or semi-quantitative analysis. Figure 1 provides an example of a 5x5 risk matrix with 4 individual risks⁶.

Figure 1: Example of 5x5 risk matrix

Each of these 4 individual risks illustrated in Figure 1 can be represented as an ordered pair of S and POH levels: R1(5,1), R2(1,5), R3(3,2), R4(2,3).

A natural question to ask is whether we can consider any of these individual risks to be equivalent to each other for the purpose of making judgments about risk mitigation and risk acceptability.

As an example, can we consider risks R1 and R2 to be equivalent to each other? Or risks R3 and R4 to be equivalent to each other?

The logic behind this question is that these risk pairs represent a unique combination of S and POH levels - risks R1 and R2 reflect two combinations of levels 1 and 5 for S and POH; risks R3 and R4 reflect two combinations of levels 2 and 3 for S and POH.

In practical terms, should we treat both S and POH to have the same impact on risk, or is one more impactful than the other?

Recently, I asked this question to my colleagues in a LinkedIn poll. Figure 2 below shows the results of this poll⁷.

Figure 2: LinkedIn poll result on the question of risk equivalence

A majority of colleagues indicated that neither of these risk pairs represented equivalent risks. However, about 40% of respondents indicated that there was some level of equivalence; with more leaning towards the risk pair R3 and R4.

The majority opinion seems to be that each of the 25 combinations of S and POH levels in this risk matrix is unique. Therefore, judgments about risk mitigation and risk acceptability should be made on a case by case basis. Here is a comment from one of the respondents:

I would be hesitant on comparing if risks are equivalent. Each individual risk deserves thought and analysis. It must be judged if it is acceptable on its own merits. Comparing two risks because they are within the same risk level may introduce bias to the team. Just because it falls within the "yellow" risk level doesn't mean that all "yellow" risks should be accepted. The more sophisticated S and P techniques may be able to tease out that distinction.

As an example, we may judge R1(5,1) to be at a higher level than R2(1,5) because it corresponds to a much higher severity. Consequently, we may decide that R1 can never be low enough to be acceptable regardless of the likelihood of its occurrence. On the other hand, we may decide that R2 (1,5) is acceptable regardless of the likelihood of occurrence because it corresponds to the lowest severity level where harm is considered negligible.

Similarly, we may judge that R3 and R4 are not equivalent; with R3 reflecting a higher level or risk because it corresponds to a higher severity level.

This approach reflects a dominant mindset that severity of harm (S) has a higher weight in the risk equation. That is, the combination of S and POH is such that S affects the risk level much more than POH.

Of the 40% of those who thought one or both of the risk pairs were equivalent seem to judge S and POH to have the same impact on the risk level. However, it is interesting to note that a much higher proportion of respondents felt that risks R3 and R4 were equivalent compared to those who felt that risks R1 and R2 were equivalent.

As reflected by some of the other comments, it appears that practitioners in the medical device industry consider the severity level to have a bigger impact on the risk level:

None of them can be considered equivalent. Myocardial infraction could be fatal (S5) even though the probability of its occurrence might be low (P1). On the contrary, a headache could be considered as less severe considering its treatability, predictability, and reversibility despite the probability of occurrence could be high (P5).

My introduction to Risk Management and ISO 14971 was through Human Factors (HF) and especially how the US FDA (CDRH especially) expects manufacturers to scope HF validation studies - focusing on severity of harm rather than probability of harm or risk priority number (multiplying S and P). This has shaped my thinking a lot and I would never consider severity level '2' and severity level '3' risks equivalent regardless of their probability estimates.

There are two problems with a severity-only mindset:

First, we need to be highly accurate in assigning severity levels to potential harms and use them consistently in our risk analysis. This is easier said than done!

Second, a severity-only focus undermines the potential for risk reduction through robust design and reliability improvements. Reliability is the twin sibling of safety!

When we focus more on severity, to the extent that we undermine or ignore the probability of occurrence, we are managing risks with one hand tied behind our back!

Photo by Nicolas Gras on Unsplash

Even if we don’t subscribe to the idea of risk equivalency, we should pay adequate attention to the probability of occurrence. As pointed out by one of the respondents, seemingly low severity risks with a high probability of occurrence may have an inadvertent cascading effect leading to much higher severity outcomes:

Great question! I think severity is severity at the end of the day, and regardless of how likely something is, it outweighs the likelihood. However, there is a cascading effect where if a lower severity risk continues to occur unchecked it may lead to other worse hazardous situations and/or higher severity harms.

Risk is not either-or, it is both.

When we don’t have a good measurement system, we must be extra vigilant and constantly check our assumptions. Otherwise we end up overvaluing risks with higher severity and undervaluing those with lower severity but higher probability.

Key takeaway

Current industry practice, when making judgments and decisions about risks related to medical device safety, tends to overemphasize the severity of potential harm in comparison to the likelihood of its occurrence. As a result, there is a severity bias in risk assessments. To some, risks with high severities can never be adequately reduced to an acceptable level through risk mitigation measures that aim to reduce the likelihood of their occurrence. On the other hand, there are many potential risks of low severity that are often left unmitigated because a very high rate of occurrence is still considered acceptable.

We must acknowledge that medical devices can never be completely risk free. Safety is freedom from unacceptable risk⁸, which is not an absolute or static concept. Safety needs to be evaluated in the context of the anticipated benefits of the intended use of a medical device, and in relation to the current state of the art. Our job, therefore, is not to eliminate risk; rather it is to manage it such that our medical device is safe for its intended use.

A severity bias holds us back in considering breakthrough innovative technologies to solve some of the most difficult medical problems. At the same time, it makes us tolerant of low reliability products.

There is no perfect answer. Risk analysis is not perfect, and managing risks is both an art and science. The best we can do is to be aware of our biases and to keep striving for continual improvement.

1. According to the Drucker Institute, Drucker is said to have a more nuanced view of the role of measurement in the context of business improvement.

2. EU-MDR: Medical device regulation of the European Union, for example, requires that risks are reduced to as far as possible (AFAP) without adversely affecting the benefit-risk ratio.

3. ISO 14971:2019 - see definition of risk in clause 3.18.

4. Risk of financial loss/gain can be calculated as the “expected loss/gain” by multiplying the total financial impact of an outcome by the likelihood of its occurrence. As an example, if there is a 1% chance that we will lose $1M in revenue because of a new regulation, then the risk of the new regulation can be estimated as an expected loss of $10,000 (0.01 X $1,000,000).

5. ISO/TR 24971:2020 - see guidance in section 5.5.

6. Note that there may be many more individual risks associated with the use of a medical device. We are using these 4 risks only for the purpose of illustration.

7. Note that this is not a scientific poll and no generalized conclusions should be drawn about industry practice.

8. ISO 14971:2019 - see definition of safety in clause 3.26.