AI in MedTech #1: IDx-DR retinal software for early screening of diabetic retinopathy

A case study to illustrate benefit-risk assessment for an innovative AI-based screening device for early detection of diabetic retinopathy.

Living with diabetes is not easy. Not only do you have have to watch your blood sugar levels on a daily basis to avoid a life-threatening situation, you also have to worry about long term effects such as heart disease, kidney failure, or vision loss.

As an example, consider the long-term risk of losing your vision due to diabetic retinopathy. Diabetes causes damage to the blood vessels all over the body, including the retina. Retinal damage may lead to other complications such as diabetic macular edema (DME), retinal detachment and glaucoma which can cause blindness if not treated. There were an estimated 9.6 million people in the United States living with diabetic retinopathy, of which an estimated 1.84 million had a vision threatening condition¹.

The good news is that diabetic retinopathy progresses slowly and it can be treated if detected early. Bad news is that you need to visit an eye care specialist for a confirmed diagnosis.

What if a software-based solutions could help screen for early signs of diabetic retinopathy as part of the routine checkup without the need for a specialized diagnostic equipment? It could provide early warning to patients most at risk, and refer them to a specialist for diagnosis and treatment.

The IDX-DR retinal diagnostic software² does exactly that. It uses artificial intelligence (AI) to analyze retinal images of sufficient quality obtained from a commonly used retinal camera called the Topcon NW400³. This software device was granted the Breakthrough Device designation⁴ by the FDA, which fast-tracked its development and review through the De Novo reclassification process⁵, eventually leading to marketing authorization in 2018.

Health benefits of this device as part of diabetes management are clear. In the United States alone, an estimated 38.4 million people of all ages, or about 12% of the total population had diabetes in 2021⁶. Diabetes is the eighth leading cause of death in the United States, and number one cause of kidney failure, lower limb amputations and adult blindness⁷. Therefore early detection and management of long-term adverse health effects is important for the overall health and well-being of millions of people affected by diabetes.

But what about the risk(s)? Do the benefit(s) outweigh these risk(s)?

Since the FDA has already cleared this device, the answer is a definite yes! But how did the FDA come to this decision?

FDA has outlined some of the factors used to assess the benefit-risk in a 2019 Guidance Document⁸. This guidance document provides a worksheet to summarize the benefit-risk assessment using 9 questions:

Figure 1: Benefit-Risk Assessment Summary Worksheet. Source: FDA guidance

In this case study article, we will use publicly available information to reverse engineer the benefit-risk assessment worksheet for the IDX-DR device. This exercise can be useful in evaluating benefit-risk of an AI-enabled medical device for the purpose of meeting FDA expectations when preparing a regulatory submission⁹.

Let us dive in.

Benefit-risk determination is directly linked to the regulatory questions of safety and effectiveness

Figure 2: Link between benefit-risk assessment and regulatory question of safety and effectiveness

Article 860.7 in 21 CFR Part 860¹⁰ - Medical Device Classification Procedures - outlines rules for the determination of safety and effectiveness. Section (d)(1) provides the following clarification about determination of safety:

There is reasonable assurance that a device is safe when it can be determined, based upon valid scientific evidence, that the probable benefits to health from use of the device for its intended uses and conditions of use, when accompanied by adequate directions and warnings against unsafe use, outweigh any probable risks.

Therefore, for a device to be judged safe, the probable benefits to health must outweigh any probable risks.

Section (d)(2) describes how a determination of effectiveness is made:

There is reasonable assurance that a device is effective when it can be determined, based upon valid scientific evidence, that in a significant portion of the target population, the use of the device for its intended uses and conditions of use, when accompanied by adequate directions for use and warnings against unsafe use, will provide clinically significant results.

Therefore, for a device to be judged effective, a significant proportion of the target population should experience clinically significant results.

That is why the most important question in FDA’s benefit-risk assessment worksheet is Question 5 which categorically asks if the benefits outweigh the risks.

IDX-DR is indicated as a screening tool for diabetic retinopathy

According to DEN 180001¹¹ issued in January 2018, IDX-DR is indicated for the following use:

IDx-DR is indicated for use by health care providers to automatically detect more than mild diabetic retinopathy (mtmDR) in adults diagnosed with diabetes who have not been previously diagnosed with diabetic retinopathy. IDx-DR is indicated for use with the Topcon NW400.

Watch this brief video to learn about how it works.

AI is used to analyze retinal images obtained using a fundus camera in a primary care setting. There are two potential outcomes:

1. No diabetic retinopathy detected (mtmDR-); retest in 12 months

2. Diabetic retinopathy detected (mtmDR+); refer to an eye care professional

Note that the device does not provide a confirmed diagnosis. Rather it is used as a screening tool to provide a tailored recommendation to the patient based on the exam result.

Assessment of benefits vs. risks

As shown in Figure 1, the benefit-risk assessment worksheet has 3 main sections:

1. Assessment of benefit

   1. Are there any clinical benefits?

   2. What is the level of uncertainty?

2. Assessment of risk

   1. Are known/probable risks minimal?

   2. What is the level of uncertainty?

3. Assessment of benefit-risk

   1. Do benefits outweigh the risks?

   2. Are there any additional considerations?

There are individual checklists in each section to summarize the benefit-risk information. In the following sections, we will complete these checklists for the IDX-DR device using publicly available information.

Part 1: Assessment of benefits and extent of uncertainty

There are two questions in this section of the benefit-risk assessment worksheet:

1. Is there any evidence of clinical benefit?

2. What is the extent of uncertainty for the Benefits?

Notice the keyword clinical used to qualify the type of benefit(s) considered for the benefit-risk assessment within the scope of the intended use of a device. The benefit(s) may be actually experienced by the patients or they may be reflected by an acceptable surrogate.

In this case, the main benefit is avoiding a potentially severe health condition (i.e. vision loss) through early detection and appropriate medical intervention. The benefit is demonstrated by a favorable clinical performance characteristics such as sensitivity, specificity, positive percent agreement and negative percent agreement.

Clinical performance was evaluated in a pivotal study with 900 adult subjects, 22 years or older, with a confirmed diagnosis of diabetes but no prior diagnosis of diabetic retinopathy. AI results were compared with a reference standard obtained based on reading of acquired images by experienced and validated professionals at a Fundus Photography Reading Center (FPRC). A reference standard was established based on the severity of retinopathy and presence of diabetic macular edema (DME):

1. mtmDR+: Level 35 or higher on the Early Treatment for Diabetic Retinopathy Study Severity scale (ETDRS) and/or DME present

2. mtmDR-: ETDRS level 10-20 and DME absent

Performance threshold for sensitivity was defined at 85% and for specificity at 82.5% based on anticipated enrollment numbers and pre-specified regulatory requirements¹². The observed clinical performance based on 819 fully evaluable datasets is shown in the figure below:

Figure 3: Summary of IDX-DR Clinical Performance. Source: DEN 18001 De Novo Decision Summary report.

Based on the favorable clinical performance summarized in Figure 3, we can complete the checklist to answer Q1, as shown in Figure 4.

Figure 4: Completed checklist for Q1 in benefit-risk assessment summary worksheet.

The extent of uncertainty for benefits may be assessed in the context of a false negative result. A sensitivity value of 87.4% indicates that nearly 13% of patients with mtmDR+ (more than mild diabetic retinopathy) would be incorrectly reported as mtmDR- (no mtmDR). This may delay their diagnosis and treatment for up to 12 months when they would be retested.

A second factor to consider in assessing the extent of uncertainty is the confidence interval in sensitivity and specificity. The 95% confidence intervals for observed sensitivity and specificity were reported as following:

1. Sensitivity: 95%CI, 81.9% - 92.9%

2. Specificity: 95% CI, 86.9%-93.1%

The lower bound of the 95% CI for sensitivity is below the performance threshold of 85% noted above. Therefore, there is a significant level of uncertainty for benefits to patients who may be incorrectly screened out using this device.

Additionally, there is no evidence has been provided to directly correlate the AI output to a clinical diagnosis.

Figure 5 shows the completed checklist to answer Q2.

Figure 5: Completed checklist for Q2 in benefit-risk assessment summary worksheet.

The Assessment of Benefit is summarized as (see reference 10 below):

The pivotal clinical study, which enrolled a total of 900 participants, demonstrated observed sensitivity for mtmDR 87.4%, with observed specificity of 89.5%. The device performed well across the range of study participant characteristics enrolled in the study. The vast majority of study participants had a screening diagnosis result. Among study participants who also had fundus photo reading center results (needed for study analysis), 96.1% (819/852) produced an IDx-DR output of mtmDR or negative for mtmDR. Thus, the clinical study successfully and robustly demonstrated safe and effective clinical performance of IDx-DR when used to automatically (without physician assistance) detect mtmDR.

Part 2: Assessment of risk and extent of uncertainty

There are two questions in this section of the benefit-risk assessment worksheet:

1. Are known/probable risks more than minimal?

2. What is the extent of uncertainty for the risks?

The main risk is failing to correctly detect mtmDR for a patient who is highly likely to receive a clinical diagnosis of diabetic retinopathy if examined by an eye care specialist. The rate of occurrence (or prevalence) of this event can be estimated by using the formula [1-NPV], which results in 4.3%.

There are three mitigating factors - first, the patient is advised to retest in 12 months. Second, diabetic retinopathy is a slowly progressing condition, therefore the potential severity is low. Third, a general public health recommendation for everyone is to get an annual eye exam at an eye care provider where specialist care is available. As a result, a large proportion of patients with potential diabetic retinopathy should be able to receive appropriate care within a period of 12 months.

The risk of false positive is not particularly severe because it only results in a visit to the eye care specialist for further checkup and diagnosis.

These risks are similar to all diagnostic devices, however, the level of uncertainty is high due to a wide 95% CI in sensitivity. This variation will affect the overall rate of occurrence of false negatives. As a result, we should treat risks to be more than minimal with a medium level of uncertainty.

Figures 6 and 7 show the completed checklists to answer Q3 and Q4.

Figure 6: Completed checklist for Q3 in benefit-risk assessment summary worksheet

Figure 7: Completed checklist for Q4 in benefit-risk assessment summary worksheet.

The Assessment of Risk is summarized as (see reference 10 below):

The IDx-DR system, which makes use of a standard fundus camera, presents minimal physical harm risks to patients. As with most diagnostic tools, the principal risks are those of false negative results. A false negative result for mtmDR may result in delayed diagnosis and treatment if the patient is not referred to an eye care professional by the health care provider. This risk is mitigated by the health care provider’s recommendation for follow-up screenings. DR is understood to progress slowly and, thus, repeat screenings would provide additional opportunities to correctly identify the existence of mtmDR. In addition, all patients are recommended to see their eye care professional as usual to evaluate for other ophthalmic conditions not addressed by IDx-DR. Thus, some percentage of false negative subjects will be additionally screened for DR during their annual eye care exam. Other risks associated with the device are expected to be rare. A false positive result, for example, would mean an indication of disease in a patient who does not have clinical signs of diabetic retinopathy. This would result in a referral for further examination by an eye care professional which might lead to alarm fatigue. Given that examination by an eye care professional once per year is the recommended standard of care, this would not introduce any significant risk for the patient.

Part 3: Assessment of benefit-risk

There is only one question in this section of the worksheet with additional details:

1. Do the Benefits outweigh the Risks, considering the assessment of Benefit and Risk and the extent of uncertainty identified above?

   1. Yes - the benefits outweigh the risk such that, for this device, additional consideration of relevant factors would not change that determination

   2. Unable to conclude that benefits outweigh the risks - further discussion and consideration of relevant factors is appropriate - Continue to Q6

Although the overall risk is low due to several mitigating factors, there is uncertainty in benefits due to a level of uncertainty in sensitivity (i.e. lower bound of 95% CI is less than the performance threshold). Therefore, additional factors should be considered.

Figure 8 shows the completed checklists to answer Q5.

Figure 8: Completed checklist for Q5 in benefit-risk assessment summary worksheet.

Consideration of additional factors

Questions 6 through 9 in the benefit-risk assessment summary worksheet offer an opportunity to consider additional factors:

• Q6: Do the Benefits outweigh the Risks, when taking into account additional relevant considerations?

• Q7: Can the risks be mitigated, so that Benefits outweigh the Risks?

• Q8: Do the Benefits outweigh the Risks considering the use of post-market actions?

• Q9: Is there any evidence of clinical benefit for a modified Indications for Use?

In this case, could consider additional risk control measures under Q7 to reduce the likelihood of false negatives. These measures can be useful in reducing the level of uncertainty in the benefit-risk assessment to result in a more favorable outcome. There is no need for additional post-market actions to further reduce the risks.

The following table presents as summary of potential health risks and special controls that have been established under 21 CFR 886.1100¹³ for this type of device.

Figure 9: Summary of identified health risks and mitigation measures. Source: DEN 180001 De Novo Classification Decision, FDA.

Figure 10 shows the completed checklists to answer Q7.

Figure 10: Completed checklist for Q7 in benefit-risk assessment summary worksheet.

Benefit-risk assessment summary

Based on the completed checklists above, we can summarize the benefit-risk assessment in the final worksheet as shown in Figure 11 below:

Figure 11: Completed Benefit-Risk Assessment summary worksheet for IDX-DR. Adapted from FDA guidance.

The overall benefit-risk conclusion is summarized as (see reference 10 below):

In conclusion, given the available information above, the data support that for the screening of diabetic retinopathy in patients with diabetes, the probable benefits outweigh the probable risks for the IDx-DR. The device provides substantial benefits and the risks can be mitigated by the use of general controls and the identified special controls.

In conclusion

In this case study article, we used the benefit-risk assessment worksheet and related checklists provided in the FDA guidance for reviewing the benefit-risk for IDX-DR. This is an AI enabled device indicated for screening of diabetic retinopathy in patients who have been diagnosed to have diabetes.

Using publicly available data, we can complete the benefit-risk assessment summary worksheet with a reasonably level of accuracy. This exercise is useful in evaluating benefit-risk of AI-enabled devices for the purpose of meeting FDA expectations when preparing a regulatory submission.

1. CDC: Prevalence of diabetic retinopathy, last updated in 2023.

2. Digital Diagnositcs: IDR-DX has been re-branded as LumineticsCore(TM)

3. Topcon NW400 from Topcon healthcare is a specific model of a fundus camera commonly used in routine eye exams.

4. FDA: Breakthrough Devices Program

5. FDA: De-Novo Classification Request

6. CDC: National diabetes statistics report, updated as of 2021.

7. CDC: Diabetes basic facts

8. FDA: Factors to Consider When Making Benefit-Risk Determinations in Medical Device Premarket Approval and De Novo Classifications, Issued in 2019.

9. Note that this exercise is intended only for educational purpose; it does not attempt to implicitly represent FDA’s actual analysis during the review process.

10. FDA: 21 CFR 860

11. FDA: IDX-DR De Novo database entry

12. FDA: DEN 180001 De Novo Decision Summary

13. FDA: A new regulation 21 CFR 886.1100 for product code PIB has been established as a result of the De Novo classification decision (DEN 180001)