Tips for a risk-based approach to auditing Quality Systems

Insights from a Let's Talk Risk! conversation with Rick Rios

Note: this article highlights key insights gained from a conversation with Rick Rios as part of the Let’s Talk Risk! with Dr. Naveen Agarwal series on LinkedIn, a weekly live audio event where we discuss risk related topics with industry practitioners in a casual, informal way.

Summary

Here are a few key points that emerged from our discussion Rick Rios:

1. Setting the right tone helps establish a collaborative environment for the audit

2. Risk-based approach to auditing means you focus on the high-risk processes of a Quality System

3. Auditing risk management process is more than a review of the risk management file

4. CAPA continues to be a challenge for medical device manufacturers

5. Aging CAPAs are “red flags” representing potential vulnerabilities in your Quality Management System

6. CAPA averse cultures reflect an excessive focus on compliance and not on continuous improvement

Medical devices are heavily regulated across the world

In general, manufacturers are required to establish an effective Quality System to comply with regulatory requirements for continued market access. In the United States, for example, the FDA requires manufacturers of finished devices to comply with the requirements of the Quality System Regulation. In many other parts of the world, a Quality Management System according to ISO 13485 is required for regulatory compliance.

Manufacturers face many audits and regulatory inspections during a year just to maintain compliance. As an example, a leading global medical device manufacturer shares in their recent annual report that they had over 500 inspections by notified bodies and worldwide regulatory agencies in the last 3 years! This is a heavy burden on manufacturers, both small and large, which requires a lot of resources that could otherwise be invested in new product development and continuous improvement initiatives. A culture of compliance prevails in the industry, rather than that of quality, because manufacturers are afraid that they would be found non-compliant during regulatory inspections.

Regulatory authorities across the world are recognizing this issue. Recently, a voluntary single audit program (MDSAP) has been recognized by 5 leading global regulatory authorities to accept a single audit of a medical device manufacturer to satisfy their individual requirements. This program helps manufacturers to consolidate their audit-related work processes, reduced costs and improve productivity.

There is a movement in the regulatory compliance world to adopt a least-burdensome approach for regulatory decision making throughout the device lifecycle. In this context, the auditing practices in the industry are moving from an element-based approach to a risk-based approach.

What is a risk-based approach to auditing? The old model of auditing used a checklist approach that involved evaluating a quality system item-by-item against individual clauses of the applicable regulation or standard. The risk-based approach is a new method, especially in light of MDSAP, which applies a process-based approach to evaluate different processes of a quality system based on their potential impact on product safety and effectiveness. The question is not whether you meet a specific requirement or not, the question is whether your most critical processes are effective in achieving their intended goals. It is more about inputs, outputs, feedback loops and inter-relationships.

Risk-based auditing methods are still evolving. Many auditors (and auditees) are still not familiar with these methods. That is why it is exciting to have a conversation with an experienced auditor like Rick Rios so we can share these emerging best practices with industry colleagues!

Listen to the full recording of the discussion below.

Setting the right tone helps establish a collaborative environment for the audit

An audit is a stressful experience for the auditee. Team members when facing an auditor are generally guarded in their responses because they don’t want to say anything that may be interpreted as non-compliant.

The auditor, on the other hand, is trying to gather as much information as possible to evaluate the effectiveness of the Quality System in meeting regulatory requirements. If they are not able to lower the tension and set the right tone for the conversation, they are unlikely to accomplish their objective for the audit.

The most friendly word to start the conversation is “how”. An auditor needs to listen more and talk less.

A how question is the best way to start the conversation. It allows people to describe what they do without feeling defensive. It is an opportunity for the auditor to understand how things are done in practice rather than reading about them in a procedure. If they are done well, the answer will be simple and easy to understand. You will see a high level of confidence. If not, there will be a lot of hesitation, awkward pauses and rambling explanation!

As an auditor, your tone of voice and body language matters. If you behave as a cop, they will shut down and you will get only yes/no or vague answers. That is why creating a more collaborative environment must be a top priority. It is not an interrogation, it is a conversation!

Risk-based approach to auditing means you focus on the high-risk processes of a Quality System

According to ISO 13485, the term risk is applicable to a Quality Management System (QMS) in 3 categories - product safety, product performance and regulatory requirements.

Generally, the main focus of an audit is to evaluate the effectiveness of the QMS in meeting customer and applicable regulatory requirements. One of the requirements¹ in the context of risk for an organization is to apply a risk-based approach to the control of appropriate processes needed for the QMS. An auditor is primarily going to review those processes of the QMS that the organization has identified as posing the maximum risk to meeting objectives. Not all processes in an QMS are of equal importance. As an example, processes that most directly impact product safety and performance may be considered as high risk, and therefore, prioritized for a review during an audit. Generally speaking, most medical device companies would have 6-10 high-risk processes such as risk management, design control, production process, clinical evaluation and post-market surveillance.

Keep in mind that auditors are auditing each process to the word “establish”. It has 5 components: define, document, implement, maintain and effectiveness.

While the risk-based approach is used to prioritize the QMS processes for a review in an audit, the audit itself is focused on the keyword “establish”. The most important aspect of this review is effectiveness because it is directly linked to the risk of these QMS processes. An ineffective process in the QMS presents an unacceptable risk to meeting objectives.

The goal of a risk-based approach to auditing is not simply to determine compliance or non-compliance. Rather, it is to evaluate process effectiveness in meeting objectives. It involves trying to figure out if a certain process is not effective, what some of the contributing factors might be and the level of risk to QMS.

That is why, risk-based auditing is not a linear, check-the-box activity. It requires a systematic process-based approach. It is like examining a weave with a complex pattern, looking at various strands to figure out their connectivity, and to see if there might be a crack that could compromise the whole pattern!

Auditing risk management process is more than a review of the risk management file

A risk management file is a requirement of ISO 14971, the international standard for application of risk management to medical devices. Although not required by ISO 14971, a common industry practice is to integrate the risk management process within the QMS, especially because it spans across many different processes of the QMS. Generally speaking, a standalone audit of the risk management system is not a common practice. Rather, the risk management process may be reviewed as part of a broader QMS audit.

A risk management file is a key element of the risk management system according to ISO 14971². It is intended to provide documentation of various risk management activities related to a medical device throughout its lifecycle. The term risk in this context primarily refers to product safety. The main purpose of reviewing a risk management file is to ensure that a medical device continues to remain safe and effective during its entire lifecycle, that is, the clinical benefits of its intended use outweigh the overall residual risk (of harm).

Risk-based auditing of the risk management system is not just a simple review of the documents in the risk management file. It involves talking to people, especially those involved in the overall benefit-risk evaluation. The reason is that benefit-risk analysis is not an easy, formulaic task; it involves a considerable amount of clinical judgment. Therefore, a potential concern, for example, would be if this evaluation is performed without involving any clinical experts. Secondly, benefit-risk evaluation is not a one-and-done activity. It has to be a continual exercise based on information gathered through post-market surveillance. There should be documented evidence of continual updates to the risk management file, including to the benefit-risk evaluation to ensure that the medical device continues to remain safe and effective and that appropriate actions are taken to mitigate new and emerging risks.

Risks to the QMS from the perspective of emerging risks (to safety) is linked to the process of analyzing complaints data and other relevant sources.

The term “complaint data” is a misnomer. Instead, we should consider this type of information as “clinical experience” with our medical device.

It is a common industry practice to look at complaints data mainly from a quantitative perspective for trend analysis and to classify them in different categories. There is a lot of qualitative information, more representative of the “voice of the customer”, which is generally reviewed only on a case by case basis as part of adverse event reporting. In light of recent advances in artificial intelligence using natural language processing, it may be worthwhile to extract additional insights about customer experience from this type of qualitative data. We can also design our customer surveys in a way that they questions can facilitate a more quantitative analysis to help identify improvement opportunities.

Here are a few other ideas and insights that emerged from our discussion

CAPA process challenges

The corrective and preventive actions (CAPA) process is an important part of a QMS and a regulatory requirement. The industry continues to struggle with the CAPA process, which also happens to be among the top categories of observations cited in FDA warning letters. Some of the common challenges are - not using the right tools, not conducting good analysis and investigations, not able to identify underlying cause(s), not involving clinical experts, and not using creative problem solving skills. These are signs of potential weaknesses in the QMS which may pose a high risk especially if a CAPA is associated with a safety-related issue.

Aging CAPAs

When a CAPA remains open for a long time it is a potential red flag for the QMS. A CAPA is initiated when there is “something wrong” with the product or the process. While the CAPA is in progress, there is risk to related processes because they are likely operating under interim controls such as additional inspections. These interim control not only add to the operating cost, they may not be sustainable over the long run. As a result, a process operating these interim controls poses a high risk to the QMS. Sometimes there are too many CAPAs open and they cannot be processed in a timely way. This could be due to a culture of compliance which tends to create a CAPA for every issue that may not actually be suitable for a CAPA.

CAPA-averse cultures

On the other end of the spectrum, there are organizational cultures which are highly CAPA-averse. They avoid opening a CAPA as much as possible. Teams are disincentivized and in some cases penalized if they open a CAPA. They try to circumvent the process and try to apply quick-fixes to issues without doing the difficult work of identifying and addressing fundamental systemic issues. The problem with this mindset is that it also does not promote improvement. CAPA is not just about “fixing issues”, it is also about improvement and preventive actions. The result is that there are recurring quality issues in these organizations, which sometimes also compromise patient safety.

About Rick Rios

Rick Rios started his career in the defense industry working as a systems engineer in nuclear programs. Inspired by the Total Quality Management movement, he moved into a Quality and Regulatory role, and later worked in automotive and aerospace industries as an independent auditor. He has been active in the medical device industry as an independent auditor over nearly 15 years. He has successfully developed, integrated, implemented and audited quality systems based on ISO 13485, 21CFR820, MDSAP, EUMDR, DOE QC-1, NQA-1, ISO 9001, ISO 17025, NCSL Z540, and Baldrige Performance Excellence Program.

About Let’s Talk Risk with Dr. Naveen Agarwal

Let’s Talk Risk with Dr. Naveen Agarwal is a weekly live audio event on LinkedIn, where we talk about risk management related topics in a casual, informal way. Join us at 11:00 am EST every Friday on LinkedIn.

Disclaimer

Information and insights presented in this article are for educational purposes only. Views expressed by all speakers are their own and do not reflect those of their respective organizations.

1. See Clause 4.1.2 (b) of ISO 13485:2016.

2. See Clause 4.5 of ISO 14971:2019.