One reason you might be systematically underestimating risk

Risk on a per-procedure basis can be significantly lower than risk on a per-patient basis.

Imagine you come across a New York Times story about your company which suggests that the risk of a serious side effect in a medical procedure involving your device is much higher than previously reported.

Not only is it bad publicity for the medical procedure, it also casts a shadow of doubt over the safety and effectiveness of your medical device. Even when your medical device is not directly causing the serious side effect, it is implicated by association in they eyes of patients and doctors.

“A Beauty Treatment Promised to Zap Fat. For Some, It Brought Disfigurement¹” is the headline of a recent story in the New York Times that caught my eye. This story highlights several high profile cases of a serious side effect after Coolsculpting(R)² treatments, where instead of the fat cells disappearing by cryolipolysis³, they actually grow back again and harden in the treatment area, often taking the shape of the applicator. This condition is called paradoxical adipose hyperplasia (PAH), which generally requires a surgical procedure to correct.

Imagine going for a “beauty” treatment, but instead ending up with an unsightly blob of fat, clearly visible because it is in the shape of the applicator! Not only does it cause psychological stress and trauma to patients, but they also have to pay out of pocket for additional treatments including surgery.

According to the New York Times article, there is a significant gap between the occurrence rates of PAH reported by the manufacturer and clinical literature. A recent article in the Aesthetic Surgery Journal estimated the occurrence rate at 1 in 2000 cycles for new models (3/6061 cycles) of the Coolsculpting(R) system based on a multi-center retrospective chart review of over 2000 patients at 8 Canadian medical centers⁴. This is nearly double the manufacturer-reported rate of 1 in 4000 to 1 in 3000 treatment cycles!

A bigger question is whether this method of estimating occurrence of PAH on a per-cycle basis is appropriate to estimate the risk at the patient level. If you consider the data presented in the Aesthetic Surgery Journal article, the occurrence rate on a per-patient basis is 1 in 882 patients (2/1764 patients) for the new models. The per-cycle occurrence rate of 1 in 2000 appears much lower than the per-patient occurrence rate.

It is important to note that an individual patient may go through multiple cycles of the same treatment to experience the intended benefit. The patients who developed PAH in the above study, for example, went through an average of 3.3 cycles per anatomical area treated.

When we estimate the risk on a per-cycle basis, we do not account for the repeated exposure to the same hazards in multiple treatment cycles. That is why the overall per-patient occurrence rate is more relevant to estimate the risk of harm.

This is one reason why the risk at the patient level may be systematically underestimated during risk analysis.

Let us illustrate this point further with a hypothetical example.

Let us say that we generally expect patients to experience the intended benefits of a treatment after an average of 2-3 cycles but an individual patient may receive up to a maximum of 5 cycles. Treatment is discontinued if signs or symptoms of harm “x” are observed or if satisfactory results are not achieved after 5 cycles.

Let us now consider the following hypothetical data set from a clinic providing this treatment.

Figure 1: Treatment outcomes for 3 patient from a hypothetical clinic

In this example, 3 patients received a total of 10 treatment cycles with 1 of the 3 patients experiencing signs or symptoms of harm “x”. 1 patient did not experience satisfactory results after 5 cycles. Treatment was stopped for these 2 patients according to the manufacturer’s instructions.

On a per-cycle basis, the risk of harm “x” in this case would be 1 in 10 cycles. However, the risk of the adverse event on a per-patient basis is 1 in 3 patients.

This hypothetical example illustrates that the per-cycle method of calculating occurrence underestimates the risk at the patient-level by approximately 3 times.

This is a blind spot for manufacturers of multi-use medical devices such as Coolsculpting(R). Generally speaking, manufactures do not have accurate data on total number of treatment cycles performed and/or number of patients treated by providers. Only complaints related to malfunctions, user experience and adverse events are reported back to the manufacturer. This lack of data is a significant challenge in accurately estimating the risk of harm to patients.

A common industry practice is to estimate the number of treatment cycles (or procedures) based on sales data for consumable items used during the treatment. As an example, we can estimate the number of treatment cycles performed during a given timeframe (say a month) from the average unit sales of a consumable item such as a pretreatment skin wipe. In some cases, a provider may need to use a unique access code to initiate treatment, which is logged in the system each time. If available, this data can also be used to estimate the number of treatment cycles performed.

However, the quality of quantity of data from different providers can vary. Seasonality in sales can also lead to inaccurate estimates. As a result, there is a lot of uncertainty in risk estimates, even on the per-cycle basis.

Underestimation of risk matters more for clinical risks not directly linked to a medical device

PAH is an example of a risk of harm to the patient inherent in the medical procedure (cryolipolysis) that is not directly attributable to the device itself. It is considered to be a serious “side effect” of the procedure, generally observed many months after the last treatment cycle. This type of inherent risk negatively impacts the overall benefit-risk balance of a medical device - Coolsculpting (R) in this case - and raises doubts about its safety and effectiveness during clinical use.

Accurately estimating and evaluating these types of procedure-related inherent risks is a serious challenge for medical device manufacturers. This is because they don’t have the ability to mitigate such risks through design or other protective measures. They can rely only on information for safety such as warning, precautions, contraindications and user-training to alert providers and patients about the potential for harm. Although ISO 14971 recognizes information for safety as a one of the three risk control measure, it is generally considered to be the least effective⁵.

If these inherent clinical risks are underestimated, the overall residual risk of a medical device may appear acceptable when compared to its intended use. This may be sufficient to convince the regulators to allow marketing of the medical device and to convince the healthcare providers to use it to treat their patients. However, it may prove to be an illusion that makes you feel better in the short term, only to be shattered later by reports of patient harms, multiple lawsuits and regulatory action.

When your business relies on selling millions of cycles of treatment, as in the case of Coolsculpting(R), you cannot afford to be confronted by this reality in future. That is why it is very important to have a more accurate estimate of the overall risk and not be blindsided.

Probability comes to the rescue

Fortunately, there is a very simple solution to this problem.

It is quite easy to estimate the overall probability of occurrence of harm “x” after “n” cycles of the same treatment using the probability of occurrence in a single cycle.

As noted above, there may be uncertainties in your estimate of the per-cycle probability (or occurrence rate). However, it is much easier to estimate the per-cycle probability especially if you are tracking the total number of treatment cycles sold, or using the sales data of consumables.

In the hypothetical scenario illustrated below, a patient may undergo a maximum of 3 treatment cycles. Treatment stops if harm occurs in any cycle. If the per-cycle probability of occurrence of harm “x” is known to be 1 in 1000, we can estimate the overall probability in 3 cycles using the following approach.

Figure 2: Cumulative probability of occurrence of harm in 3 treatment cycles

In this example, we consider each treatment cycle to be an independent event with two potential outcomes. Either the patient experiences harm, or there is no harm. If the patient experiences harm, then treatment stops. If there is no harm, then treatment continues until the maximum number of treatment cycles is reached.

We know that the probability of occurrence of harm in any treatment cycle is 1 in 1000, or 0.001. Therefore, the probability of occurrence of “no harm” is (1-0.001) or 0.999. This rule applies individually to each of the 3 treatment cycles.

Now, if we want to estimate the probability of occurrence of “no harm” after 3 cycles of treatment, we simply multiply (1-0.001) by itself 3 times, which results in the cumulative probability of “no harm” equal to 0.997.

Now, we can estimate the probability that harm will occur in any of these 3 cycles as (1-0.997), which results in the probability value of 0.003 or approximately 1 in 333. This can be used as the per-patient probability of occurrence of harm, assuming that each patient can receive a maximum of 3 treatment cycles.

Note that the per-patient probability of occurrence of harm is approximately 3 times more than the per-cycle probability.

This rule can be generalized for the overall probability of harm over “n” cycles by the expression P=1-(1-0.001)^n.

Let us see if this approach can be used to verify the results presented in the Aesthetic Surgery Journal article. A summary of key data with the probability calculation approach is illustrated in the Figure below.

Figure 3: Verification of per-patient probability of harm calculation

As shown in the Figure above, our probability calculation method using the mean of approximately 3 cycles per anatomical area results in a per-patient probability of PAH of 0.0045. This is quite close to the total incidence of PAH by patient rate of 0.43% (9/2114) indicated in Table 2 of the source data above.

Note that when the mean of approximately 5 cycles based on the total body is used in the probability calculation, the resulting probability value is much higher at 0.0075. Therefore, the mean cycles per anatomic area is a better representation of the exposure in this case, because a patient may not develop PAH in all exposed anatomical areas.

An interesting observation yields a simple rule of thumb

As shown in the Figure below, per-patient probability increases linearly with number of treatment cycles, and proportionately with the per-cycle probability.

Figure 4: Per-patient probability vs. number of treatment cycles for 3 different per-cycle probability levels (P=0.001, P=0.005 and P=0.01)

This relationship was not intuitive to me because I was expecting the per-patient probability to increase exponentially with number of treatment cycles. However, it works when the per-cycle probability is low and the number of cycles is small.

Therefore, you can use the following approximation to estimate the per-patient probability from the per-cycle probability for a given number of cycles.

Per-patient probability = per-cycle probability*number of cycles

Key points to remember

The New York Times story about Coolsculpting(R) provides a valuable lesson to risk practitioners in the medical device industry.

It is important to correctly estimate the risk of harm at the patient level, especially for clinical risks that may not be directly attributable to the medical device. This issue become more important when a patient is exposed to multiple cycles of the same procedure. The risk level may be low in a single exposure, but the cumulative risk to the patient increases as they undergo additional cycles of the treatment.

This makes intuitive sense, especially if there are patient-related factors such as gender, age, comorbidities or other demographic factors that may be relevant for harm to occur.

Risk estimation methods that use a per-cycle basis to estimate the probability of occurrence of harm underestimate the risk of patient harm when they are exposed to multiple cycles of the same treatment.

Remember that the overall residual risk of your medical device also includes risks that may be inherent to the medical procedure itself. Safety and effectiveness of your medical device is evaluated by comparing the benefits of the intended use with the overall residual risk. The question is not whether your device is low or high risk; rather the question is if patients would be worse off if they did not have to go through the medical procedure that uses your device.

That is why it is important to fully understand the clinical context and create a strong interface between clinical evaluation and risk management early in the design and development phase. By identifying the potential clinical risks inherent in the medical procedure, you can consider how best to estimate and evaluate them for their contribution to the overall residual risk.

Otherwise you might be surprised to read about how you have been systematically underestimating the risk in a high profile news story. It may not be your fault, but that really doesn’t matter!

1. Source: New York Times article, April 16, 2023

2. Coolsculpting(R) is an FDA-cleared treatment to freeze and eliminate treated fat in 9 different anatomical areas of the body. Source: Allergan, Inc. now an AbbVie company.

3. Cryolipolysis, also known as “fat freezing” is a medical procedure that involves applying cold temperatures to the treatment area to freeze and break down fat cells, which are then removed by the body’s natural process.

4. A multicenter evaluation of paradoxical adipose hyperplasia following cryolipolysis for fat reduction and body contouring: a review of 8658 cycles in 2114 patients. Aesthetic Surgery Journal 2021, Vol 41(8) 932-941.

5. ISO 14971:2019, the international standard for application of risk management to medical devices, outlines three types of risk control options to be considered in the following order of priority - (1) inherently safe design and manufacturing, (2) protective measures and (3) information for safety.