Regulatory Open Forum

 View Only
Expand all | Collapse all

Probability of Occurrence

  • 1.  Probability of Occurrence

    This message was posted by a user wishing to remain anonymous
    Posted 04-Oct-2023 15:13
    This message was posted by a user wishing to remain anonymous

    What is the consensus on the use of separate probabilities (P1, P2) for expressing the probability of occurrence of harm (ISO 14971:2019 Annex C) or just a single probability (P)?

    I believe that the electronics industry has already moved in this direction but conventionally the medical device industry has used one (P) in the use and design FMEAs, but is there a move to separate probabilities, (P1) for the probability of a hazardous situation occurring and (P2) probability of a hazardous situation leading to harm?

    Has anyone in the medical device industry moved to the use of separate probabilities and if so what has been your experience with the regulators – how have they perceived it?



  • 2.  RE: Probability of Occurrence

    Posted 04-Oct-2023 16:24

    There are no circumstances in which you should use P1 and P2. The concept is pedagogical and not practical.

    First, ISO 1497:2019 Annex C is informative. It doesn't create any requirements. Figure C1 shows a conceptual diagram, not a practical application.

    In practice there are a few difficulties.

    The process is to identify a hazard, a sequence of events leading to a hazardous situation, a hazardous situation leading to a harm.

    P1 is the probability of a hazardous situation associated with the hazard mediated by the sequence of events. The steps in the sequence of events should, collectively, be a small probability – say one occurrence in 10,000 times. This means that all items in the sequence must occur at the same time. (Think of Reason's Swiss Cheese Model). In practice, it is a hard number to estimate. In addition, it is usually characterized as a point estimate.  However, it should really be a statistical distribution.

    P2 is the probability that the hazardous situation leads to a harm. This results when the patient or user is exposed to the hazardous situation. Again, it should be a small probability – one occurrence in 10,000 times. It is a hard number to estimate. In addition, a point estimate is the wrong model it should be a distribution.

    P is the product P1×P2. If you have point estimate then multiply them together. However since they are really distributions, their product is the convolution of the distributions. That integral can be very hard to compute. Consider how you would multiply two normal distributions with different means and different standard deviations to get the resultant distribution.

    The problem, in the point estimate case, is that you multiply a very small number by another very small number to get a very very small number.

    In my class I do a thought experiment. There is a problem with a wall outlet in the meeting room. An electrician comes in, troubleshoots the problem and leaves. Unfortunately, she forgot to put the cover plate back. There is now exposed wire with AC mains. This is the hazardous situation. How do you estimate the probability of failure to put the cover plate back – P1?

    Next, a person plugs something into the outlet without the cover plate, touches an exposed wire, and gets a shock (the harm). How do you estimate the  probability of touching the exposed wire – P2?

    In practice, just estimate P. The convention is that P is not a point estimate, but a range of values typically covering an interval that is an order of magnitude. The underlying distribution in uniform across the interval.

    So, in practice, estimate the interval for P or do a lot of contortions that end up with an estimate of the interval for P.

    You mention use and design FMEAs. Typically, an FMEA uses P only and usually defines it in an interval.

    However, an FMEA is not the right tool for medical device risk management. It only considers single point failures (fault condition). In medical device risk management the hazard could be in either a normal or a fault condition. In addition. The sequence of events means it is not a single point event. Multiple things need to happen at the same time.



    ------------------------------
    Dan O'Leary CQA, CQE
    Swanzey NH
    United States
    ------------------------------



  • 3.  RE: Probability of Occurrence

    Posted 05-Oct-2023 09:43

    Dan has done a great job with his explanation, but much more can be said. First of all P1 and P2 was a concept developed as an educational tool in the Informative Annexes of ISO 14971 to explain why the probability of a failure occurrence was not the same as the probability of harm occurring. It was not intended to be a requirement. 

    Second, ISO TR 24971:2020 5.5.2 explains that Quantitative Probability should only be used when there is sufficient confidence in the data. Otherwise, Qualitative estimates should be used. Remember in product development we are making best guesses on the expected performance of the device and until we have Real World Evidence the Probabilities are only Estimates based on the information we have available. 

    For products that have Predicates we may have data with sufficient confidence. But brand new devices will not have that level of confidence until exposed to human use, Design Validation at the earliest. 

    Also as Dan indicates FMEA is not risk analysis. It may be used to identify hazards as input to a risk analysis. It also is late in the process  as it requires Design Output to perform and ISO 13485:2016 7.3.3 c) requires the "outputs of risk management" are Design Inputs. FMEA is a good check tool to make sure no single-fault hazards are missed. Which brings up   another issue as ISO 14971 does not restrict the process to single-fault hazards, but requires identification of "known and foreseeable hazards…in both normal and fault conditions".  So the usefulness of FMEA in Risk Management is limited, as are all the tools. So use the risk tools within their limitations.



    ------------------------------
    Edwin Bills
    Edwin Bills Consultant
    ASQ Fellow CQE, CQA, CQM/OE, RAPS RAC
    elb@edwinbillsconsultant.com
    ------------------------------



  • 4.  RE: Probability of Occurrence

    Posted 05-Oct-2023 09:57

    Edwin, when you said that FMEA is not risk analysis, I'm wondering if you meant to say that it is not risk management?  ISO 14971 and ISO/TR 24971 maintain that FMEA is a valid risk analysis technique, yet not risk management.



    ------------------------------
    Kevin Randall, ASQ CQA, RAC (Europe, U.S., Canada)
    Principal Consultant
    Ridgway, CO
    United States
    © Copyright 2023 by ComplianceAcuity, Inc. All rights reserved.
    ------------------------------



  • 5.  RE: Probability of Occurrence

    Posted 05-Oct-2023 15:52

    Yes, Kevin that is what I meant. ISO TR 24971:2020 refers to IEC 60812 as the FMEA standard. FMEA uses definitions for risk, probability of occurrence and severity that are different from ISO 14971. They talk about "probability of failure" and "severity of effect". The input to Risk Analysis is "hazard" and FMEA is useful in identifying failure effects which may be hazards, but only in single fault conditions. ISO 14971 requires identification of "normal condition" hazards and all hazards, not just st single fault. So there are limitations to its usefulness. Another issue is that FMEA requires Design Outputs in order to perform its analysis, yet ISO 13485:2016 7.3.3 c) requires "outputs of risk management" be Design Inputs. So FMEA is late in the game, and relying on it, as so many companies do, adds cost and time to projects when they wait until after the design is firmed up to perform risk analysis. This causes costly redesigns and may impact already selected suppliers and tooling for example. 

    BUT, FMEA is a good tool to identify any overlooked single-fault hazards that may appear in the Design Output. 

    FMEA is also a great tool for its intended purpose, improving design reliability. Each of the tools identified in ISO TR 24971:2020 Annex B has its own Benefits and Limitations. It is important that we recognize those and use the best tools for the job. Early in Design-Development the best tool is Preliminary Hazard Analysis. For Human Factors/ Usability a new tool not shown in Annex B (because it's new) is Usability Related Risk Analysis (URRA), and while pFMEA has use in manufacturing another tool that intersects with Process Validation and Process Control and Monitoring is Hazard Analysis and Critical Control Points (HACCP).

    While not widely used in medical devices, HACCP is growing in Pharma and originated in Foods (all juices and seafood processed in the US uses HACCP).  HACCP can be used in connection with DOE to determine inputs to a process that have greatest impact and should be controlled. Then these are controlled in Process Validation, and finally the process outputs are monitored as part of HACCP to retain Process Control. 

    Anyway those are my comments on your question. Use FMEA within its limitations just like all tools, use them to your benefit. 



    ------------------------------
    Edwin Bills
    Edwin Bills Consultant
    ASQ Fellow CQE, CQA, CQM/OE, RAPS RAC
    elb@edwinbillsconsultant.com
    ------------------------------



  • 6.  RE: Probability of Occurrence

    Posted 06-Oct-2023 10:39

    Hi Ed, many thanks for your clarifications, have a great weekend!



    ------------------------------
    Kevin Randall, ASQ CQA, RAC (Europe, U.S., Canada)
    Principal Consultant
    Ridgway, CO
    United States
    © Copyright 2023 by ComplianceAcuity, Inc. All rights reserved.
    ------------------------------



  • 7.  RE: Probability of Occurrence

    Posted 05-Oct-2023 09:53

    ISO 14971 / EN ISO 14971 (as amended) Annex C.1 and ISO/TR 24971 sec. 5.5.2 emphasize that the probability of occurrence of harm can be expressed as a combination of separate probabilities (P1, P2) (which can themselves be a combination of sub-probabilities), or can instead be expressed just as a single probability (P).  They go on to remind us that a decomposition into P1 and P2 is not mandatory.  Therefore, doing so or not doing so depends on which approach makes the most sense for the subject case.

    Due to unknown contributing sub-probabilities, I typically focus on the single probability of occurrence of harm (P).  And while I acknowledge the scientific rigor required to calculate or estimate true and accurate P1 (i.e., Pa * Pb * Pc * Pd...), I'm also not prepared to declare that this is impossible, nor to declare that 14971 Annex C and 24971 clauses 5.4.7, H.2.7, etc., are thus worthy of dismissal due to these impracticalities.

    If the contributing sub-probabilities are accurately known and can be legitimately combined, then by all means they should be factored in, even when there may be incomplete information. Seasoned risk management practitioners understand that a risk profile derived using 14971 risk management is oftentimes no more than an elaborate educated best guess or an elaborated brainstorming output. But oftentimes, it is a practical reality that our best educated guess is the best we can do; and that our best is better than quitting or throwing mud on 14971, and that it is generally good enough for the needs of public health.  For example, 24971 says it could be the case that either P1 or P2 may be unknown, in which case a conservative approach can be used by setting the unknown probability equal to 1. I submit that, although we might be able to devise examples where calculating P1 and P2 are impossible, I nonetheless don't believe this means that we must declare it impossible to calculate P1 and P2 for all scenarios.

    Regarding that popularly discredited four-letter word FMEA: FMEA and reliability techniques can be used as part of assessing the sequence of events and figuring out P1.  Indeed, 14971 says that many sequences of events will only be identified by the systematic use of risk analysis techniques (like FMEA).  On that note, it seems to have become quite hip to frown on the use of FMEA as part of risk management.  But 14971 and 24971 are clear that, while FMEA alone is certainly not risk management, it remains true that FMEA absolutely, positively, is a valid risk analysis technique, and that risk analysis is a mandatory part of ISO 14971 risk management.  Accordingly, I continue to push back on the questionable notion that FMEA isn't a valid part of the risk management process.  If we disqualify FMEA or other risk analysis techniques from being allowed as part of 14971 risk management, then we are no longer performing legitimate 14971 risk management.



    ------------------------------
    Kevin Randall, ASQ CQA, RAC (Europe, U.S., Canada)
    Principal Consultant
    Ridgway, CO
    United States
    © Copyright 2023 by ComplianceAcuity, Inc. All rights reserved.
    ------------------------------



  • 8.  RE: Probability of Occurrence

    Posted 05-Oct-2023 10:20

    The chance of an event occurring and the chance of the event occurring and causing a problem. During Covid wearing masks and 6-foot personal separation was the rule. It was all due to the point of a person having the virus. The entire world has a rather full security search boarding an airline after 9/11. The probability of having a weapon vs the probability of having a weapon leading to a security threat are considered the same. The mitigation is the same regardless of the results.



    ------------------------------
    Edward Panek
    VP, QA/RA
    Med Device
    USN Veteran
    Research into Neural Nets - https://www.twitch.tv/edosani
    ------------------------------



  • 9.  RE: Probability of Occurrence

    Posted 05-Oct-2023 11:05

    Improper use of terminology..."Mitigation" is reduction of severity of harm only. "Risk Control" is reduction of either severity of harm or probability of occurrence or harm or both.  This is a common error in medical device risk management and should be avoided.  Use of proper terms and definitions are the key effective communications, and we have some issues with this in our industry.  I learned about this in working with standards.  For instance, the Cambridge English Dictionary has 17 definitions of the term "risk", none of the are the on in ISO 14971.  We use the Oxford English Dictionary in ISO standards along with a large number of ISO Guides to help us understand what we are trying to convey.



    ------------------------------
    Edwin Bills
    Edwin Bills Consultant
    ASQ Fellow CQE, CQA, CQM/OE, RAPS RAC
    elb@edwinbillsconsultant.com
    ------------------------------



  • 10.  RE: Probability of Occurrence

    Posted 05-Oct-2023 11:35

    Further to Ed's comment, the word "mitigation" doesn't appear in ISO 14971:2019.

    In colloquial terms, "mitigation" reduces the effect of the harm after it happens. The manufacturer doesn't conduct mitigation.

    Risk control, as Ed says, reduces the severity of the harm, the probability of the harm, or both. The manufacturer does conduct risk control through the selection of risk control measures o reduce the risk. See ISO 14971:2019, section 7.1, first paragraph.



    ------------------------------
    Dan O'Leary CQA, CQE
    Swanzey NH
    United States
    ------------------------------



  • 11.  RE: Probability of Occurrence

    Posted 05-Oct-2023 13:10
    Edited by Ed Panek 05-Oct-2023 13:24

    Valid perspectives.

    For each identified hazardous situation, the manufacturer shall estimate the associated risk(s) using available information or data.  A hazardous situation that does not result in harm is not necessary in analysis. Looking both ways to cross the street is valuable whether cars are on the street or not.

    Language evolves and adapts to specific fields and industries. In the realm of medical device risk management, the definitions and usage of terms like "mitigation" and "risk control" have been standardized for practical reasons. This specialization aids in clear communication within the industry.


    ------------------------------
    Edward Panek
    VP, QA/RA
    Med Device
    USN Veteran
    Research into Neural Nets - https://www.twitch.tv/edosani
    ------------------------------