Regulatory Open Forum

When tackling the estimation of probability of occurrence of harm during medical device risk management, I think it is important to remember that there is no one-size-fits all approach.  Instead, each manufacturer's approach needs to be derived and appropriate based on the nature and complexity of the subject device, as well as on the availability (or lack thereof) of supporting data.

The particular approach can either be quantitative or qualitative (see ISO 14971 and ISO/TR 24971).  The probability can be estimated per use, per device, per hour of use, within a population, etc. (ISO/TR 24971).  The particular comparator (i.e., per use, per device, per hour of use, within a population, etc.) needs to be chosen based on the nature of the device.  For example, some devices' intended use (e.g., reusable devices) will intrinsically make the "per hour of use" comparator more appropriate than per device.  In contrast, single-use devices may very well be appropriate for a "per device" comparator.  The "per device" comparator is where sales numbers may be utilized.  As hinted at above, these are longstanding principles that have been part of the ISO 14971 risk management paradigm for many years.

When using a "per device sold" comparator, various factors (e.g., nature and complexity of the device; marketplace tenure, emergent risks or device problems, etc.) will determine whether it is appropriate to use sales per day, per week, per month, per year, cumulative all-time sales, or some other timeframe.  It is up to the manufacturer.  Just be sure that the chosen paradigm makes sense by giving meaningful insight toward the ultimate goal of properly managing the subject device's risk profile with regard to public health.

Consideration of device lifetime when estimating probability of occurrence harm is also a longstanding best-practice in risk management.  Indeed, seeds for this can be found more than two decades back in the year 2000 version of ISO 14971 and even in EN 1441 (1998) that was a gold standard before modern risk management really got traction via the ISO 14971 series.  Moreover, the consideration of device lifetime as part of estimating probability of occurrence harm remains a modern best practice today in the latest versions (see ISO 14971:2019 and ISO/TR 24971:2020), for example where it's stated that device lifetime is an important factor for estimating probability of occurrence of harm.  Accordingly, it is wise to figure out what is meant by device "lifetime" in the context of risk management and for the estimation of probability of occurrence of harm.

On that note, my experience has been that there seems to be harmonized general agreement on the meaning of device "lifetime".  That said, I haven't found many standardized, statutory, or legislative definitions for this.  At the moment, I generally rely on, adapt, and/or expand existing explanations such as:

• FDA's 21 CFR Part 803.3: "…(f) Expected life of a device means the time that a device is expected to remain functional after it is placed into use. Certain implanted devices have specified "end of life" (EOL) dates. Other devices are not labeled as to their respective EOL, but are expected to remain operational through activities such as maintenance, repairs, or upgrades, for an estimated period of time…"
• ISO/TR 24971:2020: "…What determines the lifetime of the medical device?...Factors that should be considered include battery depletion, deterioration of materials and failure of components due to ageing, wear, fatigue or repeated use. The availability of spare parts should be considered as well…" [emphasis added].
• FDA's 2016 3^rd Party Servicing Workshop: "…Information about reliability is needed to…understand the expected life of the device…" [emphasis added].
• ISO TC/210: "…the rationale for the determination [of medical device lifetime] should be recorded and can involve consideration of the following…shelf life of the medical device…expiry date for medical devices or components which are subject to degradation over time…number of cycles or periods of use of the medical device, based on life testing of the medical device…anticipated material degradation…stability of packaging material…for implantable devices, the residual risk that results from the entire period of residence of the medical device inside the patient's body…for sterile medical devices, the ability to maintain sterility… organization's ability/willingness or contractual or regulatory obligation…" [emphasis added].

When tackling the estimation of probability of occurrence of harm during medical device risk management, I think it is important to remember that there is no one-size-fits all approach.  Instead, each manufacturer's approach needs to be derived and appropriate based on the nature and complexity of the subject device, as well as on the availability (or lack thereof) of supporting data.

The particular approach can either be quantitative or qualitative (see ISO 14971 and ISO/TR 24971).  The probability can be estimated per use, per device, per hour of use, within a population, etc. (ISO/TR 24971).  The particular comparator (i.e., per use, per device, per hour of use, within a population, etc.) needs to be chosen based on the nature of the device.  For example, some devices' intended use (e.g., reusable devices) will intrinsically make the "per hour of use" comparator more appropriate than per device.  In contrast, single-use devices may very well be appropriate for a "per device" comparator.  The "per device" comparator is where sales numbers may be utilized.  As hinted at above, these are longstanding principles that have been part of the ISO 14971 risk management paradigm for many years.

When using a "per device sold" comparator, various factors (e.g., nature and complexity of the device; marketplace tenure, emergent risks or device problems, etc.) will determine whether it is appropriate to use sales per day, per week, per month, per year, cumulative all-time sales, or some other timeframe.  It is up to the manufacturer.  Just be sure that the chosen paradigm makes sense by giving meaningful insight toward the ultimate goal of properly managing the subject device's risk profile with regard to public health.

Consideration of device lifetime when estimating probability of occurrence harm is also a longstanding best-practice in risk management.  Indeed, seeds for this can be found more than two decades back in the year 2000 version of ISO 14971 and even in EN 1441 (1998) that was a gold standard before modern risk management really got traction via the ISO 14971 series.  Moreover, the consideration of device lifetime as part of estimating probability of occurrence harm remains a modern best practice today in the latest versions (see ISO 14971:2019 and ISO/TR 24971:2020), for example where it's stated that device lifetime is an important factor for estimating probability of occurrence of harm.  Accordingly, it is wise to figure out what is meant by device "lifetime" in the context of risk management and for the estimation of probability of occurrence of harm.

On that note, my experience has been that there seems to be harmonized general agreement on the meaning of device "lifetime".  That said, I haven't found many standardized, statutory, or legislative definitions for this.  At the moment, I generally rely on, adapt, and/or expand existing explanations such as:

• FDA's 21 CFR Part 803.3: "…(f) Expected life of a device means the time that a device is expected to remain functional after it is placed into use. Certain implanted devices have specified "end of life" (EOL) dates. Other devices are not labeled as to their respective EOL, but are expected to remain operational through activities such as maintenance, repairs, or upgrades, for an estimated period of time…"
• ISO/TR 24971:2020: "…What determines the lifetime of the medical device?...Factors that should be considered include battery depletion, deterioration of materials and failure of components due to ageing, wear, fatigue or repeated use. The availability of spare parts should be considered as well…" [emphasis added].
• FDA's 2016 3^rd Party Servicing Workshop: "…Information about reliability is needed to…understand the expected life of the device…" [emphasis added].
• ISO TC/210: "…the rationale for the determination [of medical device lifetime] should be recorded and can involve consideration of the following…shelf life of the medical device…expiry date for medical devices or components which are subject to degradation over time…number of cycles or periods of use of the medical device, based on life testing of the medical device…anticipated material degradation…stability of packaging material…for implantable devices, the residual risk that results from the entire period of residence of the medical device inside the patient's body…for sterile medical devices, the ability to maintain sterility… organization's ability/willingness or contractual or regulatory obligation…" [emphasis added].

The probability discussion has many misconceptions.

The probability in ISO 14971:2019 relates to the harm with the stated severity. In other words, how often the harm occurs.

This is not a true probability, in the mathematical sense, but a rate. My understanding is that "probability" is a compromise word because ISO publishes the standards in multiple languages.

It is actually a rate, which requires a numerator and a denominator. The numerator is the number of occurrences of the harm with the stated severity and the denominator is the number of uses. Uses can have multiple measures. For a single use syringe, it is the use. For a home use blood pressure machine, it is taking one reading. For a CPAP machine is could be one sleep cycle or one hour of use. For a knee implant it could be one day. The team needs to agree on and understand the denominator. Include it in the risk management file.

The rate of harm is not the failure rate. A device could harm in normal operation and not harm in a fault condition. (A good design will include fail safe provisions.) In ISO 15971:2019 the analysis is for the hazard not the failure. This is why an FMECA is not the appropriate tool for medical device risk management.

The rate doesn't change based on sales or the size of the installed base. The rate is a characteristic of the device; the number of devices doesn't change the rate. If each harm results in a complaint, then the number of complaints is a function of the installed base size.

The rate doesn't change based on the life time of the device. Because the rate is a characteristic of the device, consider it constant over each year. An exception is harms related to wear out, but servicing addresses that issue. If each harm results in a complaint, then the number of complaints is a function of the life of the device.

When tackling the estimation of probability of occurrence of harm during medical device risk management, I think it is important to remember that there is no one-size-fits all approach.  Instead, each manufacturer's approach needs to be derived and appropriate based on the nature and complexity of the subject device, as well as on the availability (or lack thereof) of supporting data.

The particular approach can either be quantitative or qualitative (see ISO 14971 and ISO/TR 24971).  The probability can be estimated per use, per device, per hour of use, within a population, etc. (ISO/TR 24971).  The particular comparator (i.e., per use, per device, per hour of use, within a population, etc.) needs to be chosen based on the nature of the device.  For example, some devices' intended use (e.g., reusable devices) will intrinsically make the "per hour of use" comparator more appropriate than per device.  In contrast, single-use devices may very well be appropriate for a "per device" comparator.  The "per device" comparator is where sales numbers may be utilized.  As hinted at above, these are longstanding principles that have been part of the ISO 14971 risk management paradigm for many years.

When using a "per device sold" comparator, various factors (e.g., nature and complexity of the device; marketplace tenure, emergent risks or device problems, etc.) will determine whether it is appropriate to use sales per day, per week, per month, per year, cumulative all-time sales, or some other timeframe.  It is up to the manufacturer.  Just be sure that the chosen paradigm makes sense by giving meaningful insight toward the ultimate goal of properly managing the subject device's risk profile with regard to public health.

Consideration of device lifetime when estimating probability of occurrence harm is also a longstanding best-practice in risk management.  Indeed, seeds for this can be found more than two decades back in the year 2000 version of ISO 14971 and even in EN 1441 (1998) that was a gold standard before modern risk management really got traction via the ISO 14971 series.  Moreover, the consideration of device lifetime as part of estimating probability of occurrence harm remains a modern best practice today in the latest versions (see ISO 14971:2019 and ISO/TR 24971:2020), for example where it's stated that device lifetime is an important factor for estimating probability of occurrence of harm.  Accordingly, it is wise to figure out what is meant by device "lifetime" in the context of risk management and for the estimation of probability of occurrence of harm.

On that note, my experience has been that there seems to be harmonized general agreement on the meaning of device "lifetime".  That said, I haven't found many standardized, statutory, or legislative definitions for this.  At the moment, I generally rely on, adapt, and/or expand existing explanations such as:

• FDA's 21 CFR Part 803.3: "…(f) Expected life of a device means the time that a device is expected to remain functional after it is placed into use. Certain implanted devices have specified "end of life" (EOL) dates. Other devices are not labeled as to their respective EOL, but are expected to remain operational through activities such as maintenance, repairs, or upgrades, for an estimated period of time…"
• ISO/TR 24971:2020: "…What determines the lifetime of the medical device?...Factors that should be considered include battery depletion, deterioration of materials and failure of components due to ageing, wear, fatigue or repeated use. The availability of spare parts should be considered as well…" [emphasis added].
• FDA's 2016 3^rd Party Servicing Workshop: "…Information about reliability is needed to…understand the expected life of the device…" [emphasis added].
• ISO TC/210: "…the rationale for the determination [of medical device lifetime] should be recorded and can involve consideration of the following…shelf life of the medical device…expiry date for medical devices or components which are subject to degradation over time…number of cycles or periods of use of the medical device, based on life testing of the medical device…anticipated material degradation…stability of packaging material…for implantable devices, the residual risk that results from the entire period of residence of the medical device inside the patient's body…for sterile medical devices, the ability to maintain sterility… organization's ability/willingness or contractual or regulatory obligation…" [emphasis added].

• If a particular harm is estimated to happen once per day with a vital signs monitor model, then the probability would seem to be different (higher) than if that harm is estimated to happen only once per year.

• If a particular harm is estimated to happen once per device (e.g., per use, per unit sold, etc.) for a single-use device, then the probability would seem to be different (higher) than if that harm is estimated to happen only 0.001 times per device (e.g., per use, per unit sold, etc.).  For example, one occurrence in 100 units sold = 0.01 occurrences/unit sold, whereas one occurrence in 100,000 units sold is only 0.00001 occurrences/unit sold.

• Or as another example, 100 occurrences over a ten-year lifetime = 10 occurrences / year, whereas 1 occurrence over a ten-year lifetime is only 0.1 occurrences per year.

• Or as a qualitative example:  Likely to happen several times during a ten-year device lifetime would be a higher probability than not likely to happen during a ten-year device lifetime (see Table 3 of ISO/TR 24971).

The probability discussion has many misconceptions.

The probability in ISO 14971:2019 relates to the harm with the stated severity. In other words, how often the harm occurs.

This is not a true probability, in the mathematical sense, but a rate. My understanding is that "probability" is a compromise word because ISO publishes the standards in multiple languages.

It is actually a rate, which requires a numerator and a denominator. The numerator is the number of occurrences of the harm with the stated severity and the denominator is the number of uses. Uses can have multiple measures. For a single use syringe, it is the use. For a home use blood pressure machine, it is taking one reading. For a CPAP machine is could be one sleep cycle or one hour of use. For a knee implant it could be one day. The team needs to agree on and understand the denominator. Include it in the risk management file.

The rate of harm is not the failure rate. A device could harm in normal operation and not harm in a fault condition. (A good design will include fail safe provisions.) In ISO 15971:2019 the analysis is for the hazard not the failure. This is why an FMECA is not the appropriate tool for medical device risk management.

The rate doesn't change based on sales or the size of the installed base. The rate is a characteristic of the device; the number of devices doesn't change the rate. If each harm results in a complaint, then the number of complaints is a function of the installed base size.

The rate doesn't change based on the life time of the device. Because the rate is a characteristic of the device, consider it constant over each year. An exception is harms related to wear out, but servicing addresses that issue. If each harm results in a complaint, then the number of complaints is a function of the life of the device.

When tackling the estimation of probability of occurrence of harm during medical device risk management, I think it is important to remember that there is no one-size-fits all approach.  Instead, each manufacturer's approach needs to be derived and appropriate based on the nature and complexity of the subject device, as well as on the availability (or lack thereof) of supporting data.

The particular approach can either be quantitative or qualitative (see ISO 14971 and ISO/TR 24971).  The probability can be estimated per use, per device, per hour of use, within a population, etc. (ISO/TR 24971).  The particular comparator (i.e., per use, per device, per hour of use, within a population, etc.) needs to be chosen based on the nature of the device.  For example, some devices' intended use (e.g., reusable devices) will intrinsically make the "per hour of use" comparator more appropriate than per device.  In contrast, single-use devices may very well be appropriate for a "per device" comparator.  The "per device" comparator is where sales numbers may be utilized.  As hinted at above, these are longstanding principles that have been part of the ISO 14971 risk management paradigm for many years.

When using a "per device sold" comparator, various factors (e.g., nature and complexity of the device; marketplace tenure, emergent risks or device problems, etc.) will determine whether it is appropriate to use sales per day, per week, per month, per year, cumulative all-time sales, or some other timeframe.  It is up to the manufacturer.  Just be sure that the chosen paradigm makes sense by giving meaningful insight toward the ultimate goal of properly managing the subject device's risk profile with regard to public health.

Consideration of device lifetime when estimating probability of occurrence harm is also a longstanding best-practice in risk management.  Indeed, seeds for this can be found more than two decades back in the year 2000 version of ISO 14971 and even in EN 1441 (1998) that was a gold standard before modern risk management really got traction via the ISO 14971 series.  Moreover, the consideration of device lifetime as part of estimating probability of occurrence harm remains a modern best practice today in the latest versions (see ISO 14971:2019 and ISO/TR 24971:2020), for example where it's stated that device lifetime is an important factor for estimating probability of occurrence of harm.  Accordingly, it is wise to figure out what is meant by device "lifetime" in the context of risk management and for the estimation of probability of occurrence of harm.

On that note, my experience has been that there seems to be harmonized general agreement on the meaning of device "lifetime".  That said, I haven't found many standardized, statutory, or legislative definitions for this.  At the moment, I generally rely on, adapt, and/or expand existing explanations such as:

• FDA's 21 CFR Part 803.3: "…(f) Expected life of a device means the time that a device is expected to remain functional after it is placed into use. Certain implanted devices have specified "end of life" (EOL) dates. Other devices are not labeled as to their respective EOL, but are expected to remain operational through activities such as maintenance, repairs, or upgrades, for an estimated period of time…"
• ISO/TR 24971:2020: "…What determines the lifetime of the medical device?...Factors that should be considered include battery depletion, deterioration of materials and failure of components due to ageing, wear, fatigue or repeated use. The availability of spare parts should be considered as well…" [emphasis added].
• FDA's 2016 3^rd Party Servicing Workshop: "…Information about reliability is needed to…understand the expected life of the device…" [emphasis added].
• ISO TC/210: "…the rationale for the determination [of medical device lifetime] should be recorded and can involve consideration of the following…shelf life of the medical device…expiry date for medical devices or components which are subject to degradation over time…number of cycles or periods of use of the medical device, based on life testing of the medical device…anticipated material degradation…stability of packaging material…for implantable devices, the residual risk that results from the entire period of residence of the medical device inside the patient's body…for sterile medical devices, the ability to maintain sterility… organization's ability/willingness or contractual or regulatory obligation…" [emphasis added].

• If a particular harm is estimated to happen once per day with a vital signs monitor model, then the probability would seem to be different (higher) than if that harm is estimated to happen only once per year.

• If a particular harm is estimated to happen once per device (e.g., per use, per unit sold, etc.) for a single-use device, then the probability would seem to be different (higher) than if that harm is estimated to happen only 0.001 times per device (e.g., per use, per unit sold, etc.).  For example, one occurrence in 100 units sold = 0.01 occurrences/unit sold, whereas one occurrence in 100,000 units sold is only 0.00001 occurrences/unit sold.

• Or as another example, 100 occurrences over a ten-year lifetime = 10 occurrences / year, whereas 1 occurrence over a ten-year lifetime is only 0.1 occurrences per year.

• Or as a qualitative example:  Likely to happen several times during a ten-year device lifetime would be a higher probability than not likely to happen during a ten-year device lifetime (see Table 3 of ISO/TR 24971).

The probability discussion has many misconceptions.

The probability in ISO 14971:2019 relates to the harm with the stated severity. In other words, how often the harm occurs.

This is not a true probability, in the mathematical sense, but a rate. My understanding is that "probability" is a compromise word because ISO publishes the standards in multiple languages.

It is actually a rate, which requires a numerator and a denominator. The numerator is the number of occurrences of the harm with the stated severity and the denominator is the number of uses. Uses can have multiple measures. For a single use syringe, it is the use. For a home use blood pressure machine, it is taking one reading. For a CPAP machine is could be one sleep cycle or one hour of use. For a knee implant it could be one day. The team needs to agree on and understand the denominator. Include it in the risk management file.

The rate of harm is not the failure rate. A device could harm in normal operation and not harm in a fault condition. (A good design will include fail safe provisions.) In ISO 15971:2019 the analysis is for the hazard not the failure. This is why an FMECA is not the appropriate tool for medical device risk management.

The rate doesn't change based on sales or the size of the installed base. The rate is a characteristic of the device; the number of devices doesn't change the rate. If each harm results in a complaint, then the number of complaints is a function of the installed base size.

The rate doesn't change based on the life time of the device. Because the rate is a characteristic of the device, consider it constant over each year. An exception is harms related to wear out, but servicing addresses that issue. If each harm results in a complaint, then the number of complaints is a function of the life of the device.

When tackling the estimation of probability of occurrence of harm during medical device risk management, I think it is important to remember that there is no one-size-fits all approach.  Instead, each manufacturer's approach needs to be derived and appropriate based on the nature and complexity of the subject device, as well as on the availability (or lack thereof) of supporting data.

The particular approach can either be quantitative or qualitative (see ISO 14971 and ISO/TR 24971).  The probability can be estimated per use, per device, per hour of use, within a population, etc. (ISO/TR 24971).  The particular comparator (i.e., per use, per device, per hour of use, within a population, etc.) needs to be chosen based on the nature of the device.  For example, some devices' intended use (e.g., reusable devices) will intrinsically make the "per hour of use" comparator more appropriate than per device.  In contrast, single-use devices may very well be appropriate for a "per device" comparator.  The "per device" comparator is where sales numbers may be utilized.  As hinted at above, these are longstanding principles that have been part of the ISO 14971 risk management paradigm for many years.

When using a "per device sold" comparator, various factors (e.g., nature and complexity of the device; marketplace tenure, emergent risks or device problems, etc.) will determine whether it is appropriate to use sales per day, per week, per month, per year, cumulative all-time sales, or some other timeframe.  It is up to the manufacturer.  Just be sure that the chosen paradigm makes sense by giving meaningful insight toward the ultimate goal of properly managing the subject device's risk profile with regard to public health.

Consideration of device lifetime when estimating probability of occurrence harm is also a longstanding best-practice in risk management.  Indeed, seeds for this can be found more than two decades back in the year 2000 version of ISO 14971 and even in EN 1441 (1998) that was a gold standard before modern risk management really got traction via the ISO 14971 series.  Moreover, the consideration of device lifetime as part of estimating probability of occurrence harm remains a modern best practice today in the latest versions (see ISO 14971:2019 and ISO/TR 24971:2020), for example where it's stated that device lifetime is an important factor for estimating probability of occurrence of harm.  Accordingly, it is wise to figure out what is meant by device "lifetime" in the context of risk management and for the estimation of probability of occurrence of harm.

On that note, my experience has been that there seems to be harmonized general agreement on the meaning of device "lifetime".  That said, I haven't found many standardized, statutory, or legislative definitions for this.  At the moment, I generally rely on, adapt, and/or expand existing explanations such as:

• FDA's 21 CFR Part 803.3: "…(f) Expected life of a device means the time that a device is expected to remain functional after it is placed into use. Certain implanted devices have specified "end of life" (EOL) dates. Other devices are not labeled as to their respective EOL, but are expected to remain operational through activities such as maintenance, repairs, or upgrades, for an estimated period of time…"
• ISO/TR 24971:2020: "…What determines the lifetime of the medical device?...Factors that should be considered include battery depletion, deterioration of materials and failure of components due to ageing, wear, fatigue or repeated use. The availability of spare parts should be considered as well…" [emphasis added].
• FDA's 2016 3^rd Party Servicing Workshop: "…Information about reliability is needed to…understand the expected life of the device…" [emphasis added].
• ISO TC/210: "…the rationale for the determination [of medical device lifetime] should be recorded and can involve consideration of the following…shelf life of the medical device…expiry date for medical devices or components which are subject to degradation over time…number of cycles or periods of use of the medical device, based on life testing of the medical device…anticipated material degradation…stability of packaging material…for implantable devices, the residual risk that results from the entire period of residence of the medical device inside the patient's body…for sterile medical devices, the ability to maintain sterility… organization's ability/willingness or contractual or regulatory obligation…" [emphasis added].

Hi Kevin,

Thank you for the examples. I'll address each one in turn.

First however, it is import to clarify the information sought. The question is "Given a harm with a specific severity, how often does that harm occur?" Harm doesn't occur unless the device is being used, so the response must include the concept of use. The typical answer is qualitative (often, hardly ever, seldom, etc.) or qualitative. The qualitative response typically takes one of two forms: a point estimate (once in every 1234 uses) or a range (between once in 100 uses and once in every 1,000 uses). In my response I'll use the point estimate and save issues with the range for another day.

The definition of use depends on the device. Some uses are discrete (the home use blood pressure machine) and some are continuous (the ventilator). The risk management team must define and document the use.

For the vital signs monitor the usage is continuous so time is a method to define use. To be consistent, let's use days. In the first case the harm occurs 365 times in every 365 usages. In the second case the harm occurs 1 time in every 365 days. I concur that the frequency of occurrence (probability) is higher in the first case than in the second.

For the single use device, the usage is discrete; the number of usages is the number sold. I concur that a harm of once per device used (sold) is a higher frequency of occurrence than 0.001 times per device used (sold).

For the lifetime example, things start to fall apart. The lifetime is not the number of uses. With out knowing the number of uses per year, or ten years, one cannot estimate the frequency of occurrence needed for medical device risk management.

For the Table 3 example, it is incorrect because it falls into the lifetime trap. Without understanding usage, the estimate for medical device risk management doesn't apply. This is more like a reliability or availability estimate.

Other than the lifetime examples I concur.

However, these examples don't address the service life or sales volumes questions. Consider a home use blood pressure machine used once per day. The usage is discrete. The usage is not equal to the sales. Let's say that a particular harm/severity combination happens once every 250 uses.

If there is 1 such machines in use, the harm occurs once in every 250 uses.
If there are 1,000 such machines in use, the harm occurs once in every 250 uses.
If there are 5,000 such machines in use, the harm occurs once in every 250 uses.

Notice that the occurrence rate stays the same. This is the estimate needed for medical device risk management.

Also notice that the number of occurrences changes with the number of machines in use. The rate times the number is use is the estimate needed to staff the complaint handling unit.

A similar situation applies to changes in lifetime. The rate stays the same, but the number of occurrences depends on the device life.

• If a particular harm is estimated to happen once per day with a vital signs monitor model, then the probability would seem to be different (higher) than if that harm is estimated to happen only once per year.

• If a particular harm is estimated to happen once per device (e.g., per use, per unit sold, etc.) for a single-use device, then the probability would seem to be different (higher) than if that harm is estimated to happen only 0.001 times per device (e.g., per use, per unit sold, etc.).  For example, one occurrence in 100 units sold = 0.01 occurrences/unit sold, whereas one occurrence in 100,000 units sold is only 0.00001 occurrences/unit sold.

• Or as another example, 100 occurrences over a ten-year lifetime = 10 occurrences / year, whereas 1 occurrence over a ten-year lifetime is only 0.1 occurrences per year.

• Or as a qualitative example:  Likely to happen several times during a ten-year device lifetime would be a higher probability than not likely to happen during a ten-year device lifetime (see Table 3 of ISO/TR 24971).

The probability discussion has many misconceptions.

The probability in ISO 14971:2019 relates to the harm with the stated severity. In other words, how often the harm occurs.

This is not a true probability, in the mathematical sense, but a rate. My understanding is that "probability" is a compromise word because ISO publishes the standards in multiple languages.

It is actually a rate, which requires a numerator and a denominator. The numerator is the number of occurrences of the harm with the stated severity and the denominator is the number of uses. Uses can have multiple measures. For a single use syringe, it is the use. For a home use blood pressure machine, it is taking one reading. For a CPAP machine is could be one sleep cycle or one hour of use. For a knee implant it could be one day. The team needs to agree on and understand the denominator. Include it in the risk management file.

The rate of harm is not the failure rate. A device could harm in normal operation and not harm in a fault condition. (A good design will include fail safe provisions.) In ISO 15971:2019 the analysis is for the hazard not the failure. This is why an FMECA is not the appropriate tool for medical device risk management.

The rate doesn't change based on sales or the size of the installed base. The rate is a characteristic of the device; the number of devices doesn't change the rate. If each harm results in a complaint, then the number of complaints is a function of the installed base size.

The rate doesn't change based on the life time of the device. Because the rate is a characteristic of the device, consider it constant over each year. An exception is harms related to wear out, but servicing addresses that issue. If each harm results in a complaint, then the number of complaints is a function of the life of the device.

When tackling the estimation of probability of occurrence of harm during medical device risk management, I think it is important to remember that there is no one-size-fits all approach.  Instead, each manufacturer's approach needs to be derived and appropriate based on the nature and complexity of the subject device, as well as on the availability (or lack thereof) of supporting data.

The particular approach can either be quantitative or qualitative (see ISO 14971 and ISO/TR 24971).  The probability can be estimated per use, per device, per hour of use, within a population, etc. (ISO/TR 24971).  The particular comparator (i.e., per use, per device, per hour of use, within a population, etc.) needs to be chosen based on the nature of the device.  For example, some devices' intended use (e.g., reusable devices) will intrinsically make the "per hour of use" comparator more appropriate than per device.  In contrast, single-use devices may very well be appropriate for a "per device" comparator.  The "per device" comparator is where sales numbers may be utilized.  As hinted at above, these are longstanding principles that have been part of the ISO 14971 risk management paradigm for many years.

When using a "per device sold" comparator, various factors (e.g., nature and complexity of the device; marketplace tenure, emergent risks or device problems, etc.) will determine whether it is appropriate to use sales per day, per week, per month, per year, cumulative all-time sales, or some other timeframe.  It is up to the manufacturer.  Just be sure that the chosen paradigm makes sense by giving meaningful insight toward the ultimate goal of properly managing the subject device's risk profile with regard to public health.

Consideration of device lifetime when estimating probability of occurrence harm is also a longstanding best-practice in risk management.  Indeed, seeds for this can be found more than two decades back in the year 2000 version of ISO 14971 and even in EN 1441 (1998) that was a gold standard before modern risk management really got traction via the ISO 14971 series.  Moreover, the consideration of device lifetime as part of estimating probability of occurrence harm remains a modern best practice today in the latest versions (see ISO 14971:2019 and ISO/TR 24971:2020), for example where it's stated that device lifetime is an important factor for estimating probability of occurrence of harm.  Accordingly, it is wise to figure out what is meant by device "lifetime" in the context of risk management and for the estimation of probability of occurrence of harm.

On that note, my experience has been that there seems to be harmonized general agreement on the meaning of device "lifetime".  That said, I haven't found many standardized, statutory, or legislative definitions for this.  At the moment, I generally rely on, adapt, and/or expand existing explanations such as:

• FDA's 21 CFR Part 803.3: "…(f) Expected life of a device means the time that a device is expected to remain functional after it is placed into use. Certain implanted devices have specified "end of life" (EOL) dates. Other devices are not labeled as to their respective EOL, but are expected to remain operational through activities such as maintenance, repairs, or upgrades, for an estimated period of time…"
• ISO/TR 24971:2020: "…What determines the lifetime of the medical device?...Factors that should be considered include battery depletion, deterioration of materials and failure of components due to ageing, wear, fatigue or repeated use. The availability of spare parts should be considered as well…" [emphasis added].
• FDA's 2016 3^rd Party Servicing Workshop: "…Information about reliability is needed to…understand the expected life of the device…" [emphasis added].
• ISO TC/210: "…the rationale for the determination [of medical device lifetime] should be recorded and can involve consideration of the following…shelf life of the medical device…expiry date for medical devices or components which are subject to degradation over time…number of cycles or periods of use of the medical device, based on life testing of the medical device…anticipated material degradation…stability of packaging material…for implantable devices, the residual risk that results from the entire period of residence of the medical device inside the patient's body…for sterile medical devices, the ability to maintain sterility… organization's ability/willingness or contractual or regulatory obligation…" [emphasis added].

Many thanks for those clarifications; glad to hear your concurrence that there are various scenarios in which the frequency (probability) of occurrence can change rather than being static.  I also appreciate the theoretical discussion (but phew, a lot for a caveman like me to unpack there; my apologies for five paragraphs of thoughts below!).  My experience has been that real-world practice tends to demand adjustments once we leave the risk management classroom or webinar.  I explain further below and would be interested to hear any additional thoughts.

I've found that real-world practice doesn't always accommodate the theory asserting that harm doesn't occur unless the device is being used; nor the theory that the discussion must include the concept of use (presuming that "use" means "use" rather than something else).  For example, a common harm for many types of devices comes from not being able to use the device as planned (e.g., from delayed diagnosis, surgery, or other therapy or medical procedure).  Accordingly, a procedural mandate forcing probability estimation only from the vantage point of actual device use would often result in an incomplete risk analysis.  Instead, a proper risk management program needs to allow for various different probability estimation paradigms [such as ones based on sales volumes, lifetime (see ISO/TR 24971 Table 3), durations, patient populations, etc.] in addition to device usage.  I think that's probably why ISO 14971 so clearly emphasizes such liberty.

Thanks also for the illustration of the 'rate-never-changes' principle.  That is helpful to see.  I'm wondering if you can take it a step further and give us an example or two of a particular risk management task where that principle could be put to use?  I ask because of the possibility that the 'rate-never-changes' principle might be of little or no practical value outside the classroom or beyond a time-zero initial guess about probability of occurrence.

Here's what I mean: My experience is that ongoing risk management and regulatory agencies alike are focused on actual, tangible, observable, measurable occurrences rather than theoretical concepts that can't actually be seen.  In other words, effective ongoing attention to public health and device quality using the risk management process boils down to properly managing what is observed and/or measured.  As I see it, if we can't truly know the actual number of uses, then the 'rate-never-changes principle', while theoretically interesting, seems largely moot for the purposes of practicing meaningful risk management.  Instead, it has been my experience that number of actual occurrences compared against a tangible, measurable denominator seems to be the intent of ISO 14971's objectives for considering probability of occurrence and managing device risk accordingly.

Hi Kevin,

Thank you for the examples. I'll address each one in turn.

First however, it is import to clarify the information sought. The question is "Given a harm with a specific severity, how often does that harm occur?" Harm doesn't occur unless the device is being used, so the response must include the concept of use. The typical answer is qualitative (often, hardly ever, seldom, etc.) or qualitative. The qualitative response typically takes one of two forms: a point estimate (once in every 1234 uses) or a range (between once in 100 uses and once in every 1,000 uses). In my response I'll use the point estimate and save issues with the range for another day.

The definition of use depends on the device. Some uses are discrete (the home use blood pressure machine) and some are continuous (the ventilator). The risk management team must define and document the use.

For the vital signs monitor the usage is continuous so time is a method to define use. To be consistent, let's use days. In the first case the harm occurs 365 times in every 365 usages. In the second case the harm occurs 1 time in every 365 days. I concur that the frequency of occurrence (probability) is higher in the first case than in the second.

For the single use device, the usage is discrete; the number of usages is the number sold. I concur that a harm of once per device used (sold) is a higher frequency of occurrence than 0.001 times per device used (sold).

For the lifetime example, things start to fall apart. The lifetime is not the number of uses. With out knowing the number of uses per year, or ten years, one cannot estimate the frequency of occurrence needed for medical device risk management.

For the Table 3 example, it is incorrect because it falls into the lifetime trap. Without understanding usage, the estimate for medical device risk management doesn't apply. This is more like a reliability or availability estimate.

Other than the lifetime examples I concur.

However, these examples don't address the service life or sales volumes questions. Consider a home use blood pressure machine used once per day. The usage is discrete. The usage is not equal to the sales. Let's say that a particular harm/severity combination happens once every 250 uses.

If there is 1 such machines in use, the harm occurs once in every 250 uses.
If there are 1,000 such machines in use, the harm occurs once in every 250 uses.
If there are 5,000 such machines in use, the harm occurs once in every 250 uses.

Notice that the occurrence rate stays the same. This is the estimate needed for medical device risk management.

Also notice that the number of occurrences changes with the number of machines in use. The rate times the number is use is the estimate needed to staff the complaint handling unit.

A similar situation applies to changes in lifetime. The rate stays the same, but the number of occurrences depends on the device life.

• If a particular harm is estimated to happen once per day with a vital signs monitor model, then the probability would seem to be different (higher) than if that harm is estimated to happen only once per year.

• If a particular harm is estimated to happen once per device (e.g., per use, per unit sold, etc.) for a single-use device, then the probability would seem to be different (higher) than if that harm is estimated to happen only 0.001 times per device (e.g., per use, per unit sold, etc.).  For example, one occurrence in 100 units sold = 0.01 occurrences/unit sold, whereas one occurrence in 100,000 units sold is only 0.00001 occurrences/unit sold.

• Or as another example, 100 occurrences over a ten-year lifetime = 10 occurrences / year, whereas 1 occurrence over a ten-year lifetime is only 0.1 occurrences per year.

• Or as a qualitative example:  Likely to happen several times during a ten-year device lifetime would be a higher probability than not likely to happen during a ten-year device lifetime (see Table 3 of ISO/TR 24971).

The probability discussion has many misconceptions.

The probability in ISO 14971:2019 relates to the harm with the stated severity. In other words, how often the harm occurs.

This is not a true probability, in the mathematical sense, but a rate. My understanding is that "probability" is a compromise word because ISO publishes the standards in multiple languages.

It is actually a rate, which requires a numerator and a denominator. The numerator is the number of occurrences of the harm with the stated severity and the denominator is the number of uses. Uses can have multiple measures. For a single use syringe, it is the use. For a home use blood pressure machine, it is taking one reading. For a CPAP machine is could be one sleep cycle or one hour of use. For a knee implant it could be one day. The team needs to agree on and understand the denominator. Include it in the risk management file.

The rate of harm is not the failure rate. A device could harm in normal operation and not harm in a fault condition. (A good design will include fail safe provisions.) In ISO 15971:2019 the analysis is for the hazard not the failure. This is why an FMECA is not the appropriate tool for medical device risk management.

The rate doesn't change based on sales or the size of the installed base. The rate is a characteristic of the device; the number of devices doesn't change the rate. If each harm results in a complaint, then the number of complaints is a function of the installed base size.

The rate doesn't change based on the life time of the device. Because the rate is a characteristic of the device, consider it constant over each year. An exception is harms related to wear out, but servicing addresses that issue. If each harm results in a complaint, then the number of complaints is a function of the life of the device.

When tackling the estimation of probability of occurrence of harm during medical device risk management, I think it is important to remember that there is no one-size-fits all approach.  Instead, each manufacturer's approach needs to be derived and appropriate based on the nature and complexity of the subject device, as well as on the availability (or lack thereof) of supporting data.

The particular approach can either be quantitative or qualitative (see ISO 14971 and ISO/TR 24971).  The probability can be estimated per use, per device, per hour of use, within a population, etc. (ISO/TR 24971).  The particular comparator (i.e., per use, per device, per hour of use, within a population, etc.) needs to be chosen based on the nature of the device.  For example, some devices' intended use (e.g., reusable devices) will intrinsically make the "per hour of use" comparator more appropriate than per device.  In contrast, single-use devices may very well be appropriate for a "per device" comparator.  The "per device" comparator is where sales numbers may be utilized.  As hinted at above, these are longstanding principles that have been part of the ISO 14971 risk management paradigm for many years.

When using a "per device sold" comparator, various factors (e.g., nature and complexity of the device; marketplace tenure, emergent risks or device problems, etc.) will determine whether it is appropriate to use sales per day, per week, per month, per year, cumulative all-time sales, or some other timeframe.  It is up to the manufacturer.  Just be sure that the chosen paradigm makes sense by giving meaningful insight toward the ultimate goal of properly managing the subject device's risk profile with regard to public health.

Consideration of device lifetime when estimating probability of occurrence harm is also a longstanding best-practice in risk management.  Indeed, seeds for this can be found more than two decades back in the year 2000 version of ISO 14971 and even in EN 1441 (1998) that was a gold standard before modern risk management really got traction via the ISO 14971 series.  Moreover, the consideration of device lifetime as part of estimating probability of occurrence harm remains a modern best practice today in the latest versions (see ISO 14971:2019 and ISO/TR 24971:2020), for example where it's stated that device lifetime is an important factor for estimating probability of occurrence of harm.  Accordingly, it is wise to figure out what is meant by device "lifetime" in the context of risk management and for the estimation of probability of occurrence of harm.

On that note, my experience has been that there seems to be harmonized general agreement on the meaning of device "lifetime".  That said, I haven't found many standardized, statutory, or legislative definitions for this.  At the moment, I generally rely on, adapt, and/or expand existing explanations such as:

• FDA's 21 CFR Part 803.3: "…(f) Expected life of a device means the time that a device is expected to remain functional after it is placed into use. Certain implanted devices have specified "end of life" (EOL) dates. Other devices are not labeled as to their respective EOL, but are expected to remain operational through activities such as maintenance, repairs, or upgrades, for an estimated period of time…"
• ISO/TR 24971:2020: "…What determines the lifetime of the medical device?...Factors that should be considered include battery depletion, deterioration of materials and failure of components due to ageing, wear, fatigue or repeated use. The availability of spare parts should be considered as well…" [emphasis added].
• FDA's 2016 3^rd Party Servicing Workshop: "…Information about reliability is needed to…understand the expected life of the device…" [emphasis added].
• ISO TC/210: "…the rationale for the determination [of medical device lifetime] should be recorded and can involve consideration of the following…shelf life of the medical device…expiry date for medical devices or components which are subject to degradation over time…number of cycles or periods of use of the medical device, based on life testing of the medical device…anticipated material degradation…stability of packaging material…for implantable devices, the residual risk that results from the entire period of residence of the medical device inside the patient's body…for sterile medical devices, the ability to maintain sterility… organization's ability/willingness or contractual or regulatory obligation…" [emphasis added].

Many thanks for those clarifications; glad to hear your concurrence that there are various scenarios in which the frequency (probability) of occurrence can change rather than being static.  I also appreciate the theoretical discussion (but phew, a lot for a caveman like me to unpack there; my apologies for five paragraphs of thoughts below!).  My experience has been that real-world practice tends to demand adjustments once we leave the risk management classroom or webinar.  I explain further below and would be interested to hear any additional thoughts.

I've found that real-world practice doesn't always accommodate the theory asserting that harm doesn't occur unless the device is being used; nor the theory that the discussion must include the concept of use (presuming that "use" means "use" rather than something else).  For example, a common harm for many types of devices comes from not being able to use the device as planned (e.g., from delayed diagnosis, surgery, or other therapy or medical procedure).  Accordingly, a procedural mandate forcing probability estimation only from the vantage point of actual device use would often result in an incomplete risk analysis.  Instead, a proper risk management program needs to allow for various different probability estimation paradigms [such as ones based on sales volumes, lifetime (see ISO/TR 24971 Table 3), durations, patient populations, etc.] in addition to device usage.  I think that's probably why ISO 14971 so clearly emphasizes such liberty.

Thanks also for the illustration of the 'rate-never-changes' principle.  That is helpful to see.  I'm wondering if you can take it a step further and give us an example or two of a particular risk management task where that principle could be put to use?  I ask because of the possibility that the 'rate-never-changes' principle might be of little or no practical value outside the classroom or beyond a time-zero initial guess about probability of occurrence.

Here's what I mean: My experience is that ongoing risk management and regulatory agencies alike are focused on actual, tangible, observable, measurable occurrences rather than theoretical concepts that can't actually be seen.  In other words, effective ongoing attention to public health and device quality using the risk management process boils down to properly managing what is observed and/or measured.  As I see it, if we can't truly know the actual number of uses, then the 'rate-never-changes principle', while theoretically interesting, seems largely moot for the purposes of practicing meaningful risk management.  Instead, it has been my experience that number of actual occurrences compared against a tangible, measurable denominator seems to be the intent of ISO 14971's objectives for considering probability of occurrence and managing device risk accordingly.

Hi Kevin,

Thank you for the examples. I'll address each one in turn.

First however, it is import to clarify the information sought. The question is "Given a harm with a specific severity, how often does that harm occur?" Harm doesn't occur unless the device is being used, so the response must include the concept of use. The typical answer is qualitative (often, hardly ever, seldom, etc.) or qualitative. The qualitative response typically takes one of two forms: a point estimate (once in every 1234 uses) or a range (between once in 100 uses and once in every 1,000 uses). In my response I'll use the point estimate and save issues with the range for another day.

The definition of use depends on the device. Some uses are discrete (the home use blood pressure machine) and some are continuous (the ventilator). The risk management team must define and document the use.

For the vital signs monitor the usage is continuous so time is a method to define use. To be consistent, let's use days. In the first case the harm occurs 365 times in every 365 usages. In the second case the harm occurs 1 time in every 365 days. I concur that the frequency of occurrence (probability) is higher in the first case than in the second.

For the single use device, the usage is discrete; the number of usages is the number sold. I concur that a harm of once per device used (sold) is a higher frequency of occurrence than 0.001 times per device used (sold).

For the lifetime example, things start to fall apart. The lifetime is not the number of uses. With out knowing the number of uses per year, or ten years, one cannot estimate the frequency of occurrence needed for medical device risk management.

For the Table 3 example, it is incorrect because it falls into the lifetime trap. Without understanding usage, the estimate for medical device risk management doesn't apply. This is more like a reliability or availability estimate.

Other than the lifetime examples I concur.

However, these examples don't address the service life or sales volumes questions. Consider a home use blood pressure machine used once per day. The usage is discrete. The usage is not equal to the sales. Let's say that a particular harm/severity combination happens once every 250 uses.

If there is 1 such machines in use, the harm occurs once in every 250 uses.
If there are 1,000 such machines in use, the harm occurs once in every 250 uses.
If there are 5,000 such machines in use, the harm occurs once in every 250 uses.

Notice that the occurrence rate stays the same. This is the estimate needed for medical device risk management.

Also notice that the number of occurrences changes with the number of machines in use. The rate times the number is use is the estimate needed to staff the complaint handling unit.

A similar situation applies to changes in lifetime. The rate stays the same, but the number of occurrences depends on the device life.

• If a particular harm is estimated to happen once per day with a vital signs monitor model, then the probability would seem to be different (higher) than if that harm is estimated to happen only once per year.

• If a particular harm is estimated to happen once per device (e.g., per use, per unit sold, etc.) for a single-use device, then the probability would seem to be different (higher) than if that harm is estimated to happen only 0.001 times per device (e.g., per use, per unit sold, etc.).  For example, one occurrence in 100 units sold = 0.01 occurrences/unit sold, whereas one occurrence in 100,000 units sold is only 0.00001 occurrences/unit sold.

• Or as another example, 100 occurrences over a ten-year lifetime = 10 occurrences / year, whereas 1 occurrence over a ten-year lifetime is only 0.1 occurrences per year.

• Or as a qualitative example:  Likely to happen several times during a ten-year device lifetime would be a higher probability than not likely to happen during a ten-year device lifetime (see Table 3 of ISO/TR 24971).

The probability discussion has many misconceptions.

The probability in ISO 14971:2019 relates to the harm with the stated severity. In other words, how often the harm occurs.

This is not a true probability, in the mathematical sense, but a rate. My understanding is that "probability" is a compromise word because ISO publishes the standards in multiple languages.

It is actually a rate, which requires a numerator and a denominator. The numerator is the number of occurrences of the harm with the stated severity and the denominator is the number of uses. Uses can have multiple measures. For a single use syringe, it is the use. For a home use blood pressure machine, it is taking one reading. For a CPAP machine is could be one sleep cycle or one hour of use. For a knee implant it could be one day. The team needs to agree on and understand the denominator. Include it in the risk management file.

The rate of harm is not the failure rate. A device could harm in normal operation and not harm in a fault condition. (A good design will include fail safe provisions.) In ISO 15971:2019 the analysis is for the hazard not the failure. This is why an FMECA is not the appropriate tool for medical device risk management.

The rate doesn't change based on sales or the size of the installed base. The rate is a characteristic of the device; the number of devices doesn't change the rate. If each harm results in a complaint, then the number of complaints is a function of the installed base size.

The rate doesn't change based on the life time of the device. Because the rate is a characteristic of the device, consider it constant over each year. An exception is harms related to wear out, but servicing addresses that issue. If each harm results in a complaint, then the number of complaints is a function of the life of the device.

When tackling the estimation of probability of occurrence of harm during medical device risk management, I think it is important to remember that there is no one-size-fits all approach.  Instead, each manufacturer's approach needs to be derived and appropriate based on the nature and complexity of the subject device, as well as on the availability (or lack thereof) of supporting data.

The particular approach can either be quantitative or qualitative (see ISO 14971 and ISO/TR 24971).  The probability can be estimated per use, per device, per hour of use, within a population, etc. (ISO/TR 24971).  The particular comparator (i.e., per use, per device, per hour of use, within a population, etc.) needs to be chosen based on the nature of the device.  For example, some devices' intended use (e.g., reusable devices) will intrinsically make the "per hour of use" comparator more appropriate than per device.  In contrast, single-use devices may very well be appropriate for a "per device" comparator.  The "per device" comparator is where sales numbers may be utilized.  As hinted at above, these are longstanding principles that have been part of the ISO 14971 risk management paradigm for many years.

When using a "per device sold" comparator, various factors (e.g., nature and complexity of the device; marketplace tenure, emergent risks or device problems, etc.) will determine whether it is appropriate to use sales per day, per week, per month, per year, cumulative all-time sales, or some other timeframe.  It is up to the manufacturer.  Just be sure that the chosen paradigm makes sense by giving meaningful insight toward the ultimate goal of properly managing the subject device's risk profile with regard to public health.

Consideration of device lifetime when estimating probability of occurrence harm is also a longstanding best-practice in risk management.  Indeed, seeds for this can be found more than two decades back in the year 2000 version of ISO 14971 and even in EN 1441 (1998) that was a gold standard before modern risk management really got traction via the ISO 14971 series.  Moreover, the consideration of device lifetime as part of estimating probability of occurrence harm remains a modern best practice today in the latest versions (see ISO 14971:2019 and ISO/TR 24971:2020), for example where it's stated that device lifetime is an important factor for estimating probability of occurrence of harm.  Accordingly, it is wise to figure out what is meant by device "lifetime" in the context of risk management and for the estimation of probability of occurrence of harm.

On that note, my experience has been that there seems to be harmonized general agreement on the meaning of device "lifetime".  That said, I haven't found many standardized, statutory, or legislative definitions for this.  At the moment, I generally rely on, adapt, and/or expand existing explanations such as:

• FDA's 21 CFR Part 803.3: "…(f) Expected life of a device means the time that a device is expected to remain functional after it is placed into use. Certain implanted devices have specified "end of life" (EOL) dates. Other devices are not labeled as to their respective EOL, but are expected to remain operational through activities such as maintenance, repairs, or upgrades, for an estimated period of time…"
• ISO/TR 24971:2020: "…What determines the lifetime of the medical device?...Factors that should be considered include battery depletion, deterioration of materials and failure of components due to ageing, wear, fatigue or repeated use. The availability of spare parts should be considered as well…" [emphasis added].
• FDA's 2016 3^rd Party Servicing Workshop: "…Information about reliability is needed to…understand the expected life of the device…" [emphasis added].
• ISO TC/210: "…the rationale for the determination [of medical device lifetime] should be recorded and can involve consideration of the following…shelf life of the medical device…expiry date for medical devices or components which are subject to degradation over time…number of cycles or periods of use of the medical device, based on life testing of the medical device…anticipated material degradation…stability of packaging material…for implantable devices, the residual risk that results from the entire period of residence of the medical device inside the patient's body…for sterile medical devices, the ability to maintain sterility… organization's ability/willingness or contractual or regulatory obligation…" [emphasis added].

This chain has brought up lots of good information in the discussion of probability of harm. Kevin has now brought up some other aspects. 

Yes, probability changes over time. Initially our risk estimates are best guesses, based on our limited knowledge of the device and its use. Once the device enters commerce we get actual real world data and our probabilities are no longer estimates. The application of POH changes at this point, for a brand new never before on market,  from qualitative estimates to quantitative values. For devices with a history (devices that are upgrades of previous models, for example) we MAY have sufficient data to release with quantitative values, but not always, these too may be released with qualitative estimates of POH.  But in any case, at release they are estimates, or guesses.

Another aspect of change now enters the picture.  In the Production-postproduction requirements of the standard the manufacturer is required to actively seek information on the performance of the device from a risk perspective and to update the Risk Management File with the new information gained, such as POH.  

With a brand new device especially, until the device is exposed to the general population, similar to drugs, our guesses or estimates are replaced with actual data. This is where we move from qualitative estimates to quantitative values including POH.

ISO 14971 is a lifecycle standard, and the manufacturer is required to update the RMF with information gained over the entire life of the device, so the standard expects updates to the RMF. Auditors that are knowledgeable of the standard will be looking to see that the RMF has been revised over time, based of results of postmarket  monitoring. This activity is also a requirement of ISO 13485, Clause 8, and an expectation of Postmarket Surveillance. 

For a brand new higher risk device the expectation is that postmarket monitoring is more frequent than a device with a long history and lower risk. There is not a "one size fits all" application of this requirement. Once the  data becomes stable and of a higher confidence level we can adjust the frequency of monitoring activity unless a new event occurs that may cause heightened concerns. Aspects like new markets and new uses can impact this, for instance. 

Many thanks for those clarifications; glad to hear your concurrence that there are various scenarios in which the frequency (probability) of occurrence can change rather than being static.  I also appreciate the theoretical discussion (but phew, a lot for a caveman like me to unpack there; my apologies for five paragraphs of thoughts below!).  My experience has been that real-world practice tends to demand adjustments once we leave the risk management classroom or webinar.  I explain further below and would be interested to hear any additional thoughts.

I've found that real-world practice doesn't always accommodate the theory asserting that harm doesn't occur unless the device is being used; nor the theory that the discussion must include the concept of use (presuming that "use" means "use" rather than something else).  For example, a common harm for many types of devices comes from not being able to use the device as planned (e.g., from delayed diagnosis, surgery, or other therapy or medical procedure).  Accordingly, a procedural mandate forcing probability estimation only from the vantage point of actual device use would often result in an incomplete risk analysis.  Instead, a proper risk management program needs to allow for various different probability estimation paradigms [such as ones based on sales volumes, lifetime (see ISO/TR 24971 Table 3), durations, patient populations, etc.] in addition to device usage.  I think that's probably why ISO 14971 so clearly emphasizes such liberty.

Thanks also for the illustration of the 'rate-never-changes' principle.  That is helpful to see.  I'm wondering if you can take it a step further and give us an example or two of a particular risk management task where that principle could be put to use?  I ask because of the possibility that the 'rate-never-changes' principle might be of little or no practical value outside the classroom or beyond a time-zero initial guess about probability of occurrence.

Here's what I mean: My experience is that ongoing risk management and regulatory agencies alike are focused on actual, tangible, observable, measurable occurrences rather than theoretical concepts that can't actually be seen.  In other words, effective ongoing attention to public health and device quality using the risk management process boils down to properly managing what is observed and/or measured.  As I see it, if we can't truly know the actual number of uses, then the 'rate-never-changes principle', while theoretically interesting, seems largely moot for the purposes of practicing meaningful risk management.  Instead, it has been my experience that number of actual occurrences compared against a tangible, measurable denominator seems to be the intent of ISO 14971's objectives for considering probability of occurrence and managing device risk accordingly.

Hi Kevin,

Thank you for the examples. I'll address each one in turn.

First however, it is import to clarify the information sought. The question is "Given a harm with a specific severity, how often does that harm occur?" Harm doesn't occur unless the device is being used, so the response must include the concept of use. The typical answer is qualitative (often, hardly ever, seldom, etc.) or qualitative. The qualitative response typically takes one of two forms: a point estimate (once in every 1234 uses) or a range (between once in 100 uses and once in every 1,000 uses). In my response I'll use the point estimate and save issues with the range for another day.

The definition of use depends on the device. Some uses are discrete (the home use blood pressure machine) and some are continuous (the ventilator). The risk management team must define and document the use.

For the vital signs monitor the usage is continuous so time is a method to define use. To be consistent, let's use days. In the first case the harm occurs 365 times in every 365 usages. In the second case the harm occurs 1 time in every 365 days. I concur that the frequency of occurrence (probability) is higher in the first case than in the second.

For the single use device, the usage is discrete; the number of usages is the number sold. I concur that a harm of once per device used (sold) is a higher frequency of occurrence than 0.001 times per device used (sold).

For the lifetime example, things start to fall apart. The lifetime is not the number of uses. With out knowing the number of uses per year, or ten years, one cannot estimate the frequency of occurrence needed for medical device risk management.

For the Table 3 example, it is incorrect because it falls into the lifetime trap. Without understanding usage, the estimate for medical device risk management doesn't apply. This is more like a reliability or availability estimate.

Other than the lifetime examples I concur.

However, these examples don't address the service life or sales volumes questions. Consider a home use blood pressure machine used once per day. The usage is discrete. The usage is not equal to the sales. Let's say that a particular harm/severity combination happens once every 250 uses.

If there is 1 such machines in use, the harm occurs once in every 250 uses.
If there are 1,000 such machines in use, the harm occurs once in every 250 uses.
If there are 5,000 such machines in use, the harm occurs once in every 250 uses.

Notice that the occurrence rate stays the same. This is the estimate needed for medical device risk management.

Also notice that the number of occurrences changes with the number of machines in use. The rate times the number is use is the estimate needed to staff the complaint handling unit.

A similar situation applies to changes in lifetime. The rate stays the same, but the number of occurrences depends on the device life.

• If a particular harm is estimated to happen once per day with a vital signs monitor model, then the probability would seem to be different (higher) than if that harm is estimated to happen only once per year.

• If a particular harm is estimated to happen once per device (e.g., per use, per unit sold, etc.) for a single-use device, then the probability would seem to be different (higher) than if that harm is estimated to happen only 0.001 times per device (e.g., per use, per unit sold, etc.).  For example, one occurrence in 100 units sold = 0.01 occurrences/unit sold, whereas one occurrence in 100,000 units sold is only 0.00001 occurrences/unit sold.

• Or as another example, 100 occurrences over a ten-year lifetime = 10 occurrences / year, whereas 1 occurrence over a ten-year lifetime is only 0.1 occurrences per year.

• Or as a qualitative example:  Likely to happen several times during a ten-year device lifetime would be a higher probability than not likely to happen during a ten-year device lifetime (see Table 3 of ISO/TR 24971).

The probability discussion has many misconceptions.

The probability in ISO 14971:2019 relates to the harm with the stated severity. In other words, how often the harm occurs.

This is not a true probability, in the mathematical sense, but a rate. My understanding is that "probability" is a compromise word because ISO publishes the standards in multiple languages.

It is actually a rate, which requires a numerator and a denominator. The numerator is the number of occurrences of the harm with the stated severity and the denominator is the number of uses. Uses can have multiple measures. For a single use syringe, it is the use. For a home use blood pressure machine, it is taking one reading. For a CPAP machine is could be one sleep cycle or one hour of use. For a knee implant it could be one day. The team needs to agree on and understand the denominator. Include it in the risk management file.

The rate of harm is not the failure rate. A device could harm in normal operation and not harm in a fault condition. (A good design will include fail safe provisions.) In ISO 15971:2019 the analysis is for the hazard not the failure. This is why an FMECA is not the appropriate tool for medical device risk management.

The rate doesn't change based on sales or the size of the installed base. The rate is a characteristic of the device; the number of devices doesn't change the rate. If each harm results in a complaint, then the number of complaints is a function of the installed base size.

The rate doesn't change based on the life time of the device. Because the rate is a characteristic of the device, consider it constant over each year. An exception is harms related to wear out, but servicing addresses that issue. If each harm results in a complaint, then the number of complaints is a function of the life of the device.

When tackling the estimation of probability of occurrence of harm during medical device risk management, I think it is important to remember that there is no one-size-fits all approach.  Instead, each manufacturer's approach needs to be derived and appropriate based on the nature and complexity of the subject device, as well as on the availability (or lack thereof) of supporting data.

The particular approach can either be quantitative or qualitative (see ISO 14971 and ISO/TR 24971).  The probability can be estimated per use, per device, per hour of use, within a population, etc. (ISO/TR 24971).  The particular comparator (i.e., per use, per device, per hour of use, within a population, etc.) needs to be chosen based on the nature of the device.  For example, some devices' intended use (e.g., reusable devices) will intrinsically make the "per hour of use" comparator more appropriate than per device.  In contrast, single-use devices may very well be appropriate for a "per device" comparator.  The "per device" comparator is where sales numbers may be utilized.  As hinted at above, these are longstanding principles that have been part of the ISO 14971 risk management paradigm for many years.

When using a "per device sold" comparator, various factors (e.g., nature and complexity of the device; marketplace tenure, emergent risks or device problems, etc.) will determine whether it is appropriate to use sales per day, per week, per month, per year, cumulative all-time sales, or some other timeframe.  It is up to the manufacturer.  Just be sure that the chosen paradigm makes sense by giving meaningful insight toward the ultimate goal of properly managing the subject device's risk profile with regard to public health.

Consideration of device lifetime when estimating probability of occurrence harm is also a longstanding best-practice in risk management.  Indeed, seeds for this can be found more than two decades back in the year 2000 version of ISO 14971 and even in EN 1441 (1998) that was a gold standard before modern risk management really got traction via the ISO 14971 series.  Moreover, the consideration of device lifetime as part of estimating probability of occurrence harm remains a modern best practice today in the latest versions (see ISO 14971:2019 and ISO/TR 24971:2020), for example where it's stated that device lifetime is an important factor for estimating probability of occurrence of harm.  Accordingly, it is wise to figure out what is meant by device "lifetime" in the context of risk management and for the estimation of probability of occurrence of harm.

On that note, my experience has been that there seems to be harmonized general agreement on the meaning of device "lifetime".  That said, I haven't found many standardized, statutory, or legislative definitions for this.  At the moment, I generally rely on, adapt, and/or expand existing explanations such as:

• FDA's 21 CFR Part 803.3: "…(f) Expected life of a device means the time that a device is expected to remain functional after it is placed into use. Certain implanted devices have specified "end of life" (EOL) dates. Other devices are not labeled as to their respective EOL, but are expected to remain operational through activities such as maintenance, repairs, or upgrades, for an estimated period of time…"
• ISO/TR 24971:2020: "…What determines the lifetime of the medical device?...Factors that should be considered include battery depletion, deterioration of materials and failure of components due to ageing, wear, fatigue or repeated use. The availability of spare parts should be considered as well…" [emphasis added].
• FDA's 2016 3^rd Party Servicing Workshop: "…Information about reliability is needed to…understand the expected life of the device…" [emphasis added].
• ISO TC/210: "…the rationale for the determination [of medical device lifetime] should be recorded and can involve consideration of the following…shelf life of the medical device…expiry date for medical devices or components which are subject to degradation over time…number of cycles or periods of use of the medical device, based on life testing of the medical device…anticipated material degradation…stability of packaging material…for implantable devices, the residual risk that results from the entire period of residence of the medical device inside the patient's body…for sterile medical devices, the ability to maintain sterility… organization's ability/willingness or contractual or regulatory obligation…" [emphasis added].

This chain has brought up lots of good information in the discussion of probability of harm. Kevin has now brought up some other aspects. 

Yes, probability changes over time. Initially our risk estimates are best guesses, based on our limited knowledge of the device and its use. Once the device enters commerce we get actual real world data and our probabilities are no longer estimates. The application of POH changes at this point, for a brand new never before on market,  from qualitative estimates to quantitative values. For devices with a history (devices that are upgrades of previous models, for example) we MAY have sufficient data to release with quantitative values, but not always, these too may be released with qualitative estimates of POH.  But in any case, at release they are estimates, or guesses.

Another aspect of change now enters the picture.  In the Production-postproduction requirements of the standard the manufacturer is required to actively seek information on the performance of the device from a risk perspective and to update the Risk Management File with the new information gained, such as POH.  

With a brand new device especially, until the device is exposed to the general population, similar to drugs, our guesses or estimates are replaced with actual data. This is where we move from qualitative estimates to quantitative values including POH.

ISO 14971 is a lifecycle standard, and the manufacturer is required to update the RMF with information gained over the entire life of the device, so the standard expects updates to the RMF. Auditors that are knowledgeable of the standard will be looking to see that the RMF has been revised over time, based of results of postmarket  monitoring. This activity is also a requirement of ISO 13485, Clause 8, and an expectation of Postmarket Surveillance. 

For a brand new higher risk device the expectation is that postmarket monitoring is more frequent than a device with a long history and lower risk. There is not a "one size fits all" application of this requirement. Once the  data becomes stable and of a higher confidence level we can adjust the frequency of monitoring activity unless a new event occurs that may cause heightened concerns. Aspects like new markets and new uses can impact this, for instance. 

Many thanks for those clarifications; glad to hear your concurrence that there are various scenarios in which the frequency (probability) of occurrence can change rather than being static.  I also appreciate the theoretical discussion (but phew, a lot for a caveman like me to unpack there; my apologies for five paragraphs of thoughts below!).  My experience has been that real-world practice tends to demand adjustments once we leave the risk management classroom or webinar.  I explain further below and would be interested to hear any additional thoughts.

I've found that real-world practice doesn't always accommodate the theory asserting that harm doesn't occur unless the device is being used; nor the theory that the discussion must include the concept of use (presuming that "use" means "use" rather than something else).  For example, a common harm for many types of devices comes from not being able to use the device as planned (e.g., from delayed diagnosis, surgery, or other therapy or medical procedure).  Accordingly, a procedural mandate forcing probability estimation only from the vantage point of actual device use would often result in an incomplete risk analysis.  Instead, a proper risk management program needs to allow for various different probability estimation paradigms [such as ones based on sales volumes, lifetime (see ISO/TR 24971 Table 3), durations, patient populations, etc.] in addition to device usage.  I think that's probably why ISO 14971 so clearly emphasizes such liberty.

Thanks also for the illustration of the 'rate-never-changes' principle.  That is helpful to see.  I'm wondering if you can take it a step further and give us an example or two of a particular risk management task where that principle could be put to use?  I ask because of the possibility that the 'rate-never-changes' principle might be of little or no practical value outside the classroom or beyond a time-zero initial guess about probability of occurrence.

Here's what I mean: My experience is that ongoing risk management and regulatory agencies alike are focused on actual, tangible, observable, measurable occurrences rather than theoretical concepts that can't actually be seen.  In other words, effective ongoing attention to public health and device quality using the risk management process boils down to properly managing what is observed and/or measured.  As I see it, if we can't truly know the actual number of uses, then the 'rate-never-changes principle', while theoretically interesting, seems largely moot for the purposes of practicing meaningful risk management.  Instead, it has been my experience that number of actual occurrences compared against a tangible, measurable denominator seems to be the intent of ISO 14971's objectives for considering probability of occurrence and managing device risk accordingly.

Hi Kevin,

Thank you for the examples. I'll address each one in turn.

First however, it is import to clarify the information sought. The question is "Given a harm with a specific severity, how often does that harm occur?" Harm doesn't occur unless the device is being used, so the response must include the concept of use. The typical answer is qualitative (often, hardly ever, seldom, etc.) or qualitative. The qualitative response typically takes one of two forms: a point estimate (once in every 1234 uses) or a range (between once in 100 uses and once in every 1,000 uses). In my response I'll use the point estimate and save issues with the range for another day.

The definition of use depends on the device. Some uses are discrete (the home use blood pressure machine) and some are continuous (the ventilator). The risk management team must define and document the use.

For the vital signs monitor the usage is continuous so time is a method to define use. To be consistent, let's use days. In the first case the harm occurs 365 times in every 365 usages. In the second case the harm occurs 1 time in every 365 days. I concur that the frequency of occurrence (probability) is higher in the first case than in the second.

For the single use device, the usage is discrete; the number of usages is the number sold. I concur that a harm of once per device used (sold) is a higher frequency of occurrence than 0.001 times per device used (sold).

For the lifetime example, things start to fall apart. The lifetime is not the number of uses. With out knowing the number of uses per year, or ten years, one cannot estimate the frequency of occurrence needed for medical device risk management.

For the Table 3 example, it is incorrect because it falls into the lifetime trap. Without understanding usage, the estimate for medical device risk management doesn't apply. This is more like a reliability or availability estimate.

Other than the lifetime examples I concur.

However, these examples don't address the service life or sales volumes questions. Consider a home use blood pressure machine used once per day. The usage is discrete. The usage is not equal to the sales. Let's say that a particular harm/severity combination happens once every 250 uses.

If there is 1 such machines in use, the harm occurs once in every 250 uses.
If there are 1,000 such machines in use, the harm occurs once in every 250 uses.
If there are 5,000 such machines in use, the harm occurs once in every 250 uses.

Notice that the occurrence rate stays the same. This is the estimate needed for medical device risk management.

Also notice that the number of occurrences changes with the number of machines in use. The rate times the number is use is the estimate needed to staff the complaint handling unit.

A similar situation applies to changes in lifetime. The rate stays the same, but the number of occurrences depends on the device life.

• If a particular harm is estimated to happen once per day with a vital signs monitor model, then the probability would seem to be different (higher) than if that harm is estimated to happen only once per year.

• If a particular harm is estimated to happen once per device (e.g., per use, per unit sold, etc.) for a single-use device, then the probability would seem to be different (higher) than if that harm is estimated to happen only 0.001 times per device (e.g., per use, per unit sold, etc.).  For example, one occurrence in 100 units sold = 0.01 occurrences/unit sold, whereas one occurrence in 100,000 units sold is only 0.00001 occurrences/unit sold.

• Or as another example, 100 occurrences over a ten-year lifetime = 10 occurrences / year, whereas 1 occurrence over a ten-year lifetime is only 0.1 occurrences per year.

• Or as a qualitative example:  Likely to happen several times during a ten-year device lifetime would be a higher probability than not likely to happen during a ten-year device lifetime (see Table 3 of ISO/TR 24971).

The probability discussion has many misconceptions.

The probability in ISO 14971:2019 relates to the harm with the stated severity. In other words, how often the harm occurs.

This is not a true probability, in the mathematical sense, but a rate. My understanding is that "probability" is a compromise word because ISO publishes the standards in multiple languages.

It is actually a rate, which requires a numerator and a denominator. The numerator is the number of occurrences of the harm with the stated severity and the denominator is the number of uses. Uses can have multiple measures. For a single use syringe, it is the use. For a home use blood pressure machine, it is taking one reading. For a CPAP machine is could be one sleep cycle or one hour of use. For a knee implant it could be one day. The team needs to agree on and understand the denominator. Include it in the risk management file.

The rate of harm is not the failure rate. A device could harm in normal operation and not harm in a fault condition. (A good design will include fail safe provisions.) In ISO 15971:2019 the analysis is for the hazard not the failure. This is why an FMECA is not the appropriate tool for medical device risk management.

The rate doesn't change based on sales or the size of the installed base. The rate is a characteristic of the device; the number of devices doesn't change the rate. If each harm results in a complaint, then the number of complaints is a function of the installed base size.

The rate doesn't change based on the life time of the device. Because the rate is a characteristic of the device, consider it constant over each year. An exception is harms related to wear out, but servicing addresses that issue. If each harm results in a complaint, then the number of complaints is a function of the life of the device.

When tackling the estimation of probability of occurrence of harm during medical device risk management, I think it is important to remember that there is no one-size-fits all approach.  Instead, each manufacturer's approach needs to be derived and appropriate based on the nature and complexity of the subject device, as well as on the availability (or lack thereof) of supporting data.

The particular approach can either be quantitative or qualitative (see ISO 14971 and ISO/TR 24971).  The probability can be estimated per use, per device, per hour of use, within a population, etc. (ISO/TR 24971).  The particular comparator (i.e., per use, per device, per hour of use, within a population, etc.) needs to be chosen based on the nature of the device.  For example, some devices' intended use (e.g., reusable devices) will intrinsically make the "per hour of use" comparator more appropriate than per device.  In contrast, single-use devices may very well be appropriate for a "per device" comparator.  The "per device" comparator is where sales numbers may be utilized.  As hinted at above, these are longstanding principles that have been part of the ISO 14971 risk management paradigm for many years.

When using a "per device sold" comparator, various factors (e.g., nature and complexity of the device; marketplace tenure, emergent risks or device problems, etc.) will determine whether it is appropriate to use sales per day, per week, per month, per year, cumulative all-time sales, or some other timeframe.  It is up to the manufacturer.  Just be sure that the chosen paradigm makes sense by giving meaningful insight toward the ultimate goal of properly managing the subject device's risk profile with regard to public health.

Consideration of device lifetime when estimating probability of occurrence harm is also a longstanding best-practice in risk management.  Indeed, seeds for this can be found more than two decades back in the year 2000 version of ISO 14971 and even in EN 1441 (1998) that was a gold standard before modern risk management really got traction via the ISO 14971 series.  Moreover, the consideration of device lifetime as part of estimating probability of occurrence harm remains a modern best practice today in the latest versions (see ISO 14971:2019 and ISO/TR 24971:2020), for example where it's stated that device lifetime is an important factor for estimating probability of occurrence of harm.  Accordingly, it is wise to figure out what is meant by device "lifetime" in the context of risk management and for the estimation of probability of occurrence of harm.

On that note, my experience has been that there seems to be harmonized general agreement on the meaning of device "lifetime".  That said, I haven't found many standardized, statutory, or legislative definitions for this.  At the moment, I generally rely on, adapt, and/or expand existing explanations such as:

• FDA's 21 CFR Part 803.3: "…(f) Expected life of a device means the time that a device is expected to remain functional after it is placed into use. Certain implanted devices have specified "end of life" (EOL) dates. Other devices are not labeled as to their respective EOL, but are expected to remain operational through activities such as maintenance, repairs, or upgrades, for an estimated period of time…"
• ISO/TR 24971:2020: "…What determines the lifetime of the medical device?...Factors that should be considered include battery depletion, deterioration of materials and failure of components due to ageing, wear, fatigue or repeated use. The availability of spare parts should be considered as well…" [emphasis added].
• FDA's 2016 3^rd Party Servicing Workshop: "…Information about reliability is needed to…understand the expected life of the device…" [emphasis added].
• ISO TC/210: "…the rationale for the determination [of medical device lifetime] should be recorded and can involve consideration of the following…shelf life of the medical device…expiry date for medical devices or components which are subject to degradation over time…number of cycles or periods of use of the medical device, based on life testing of the medical device…anticipated material degradation…stability of packaging material…for implantable devices, the residual risk that results from the entire period of residence of the medical device inside the patient's body…for sterile medical devices, the ability to maintain sterility… organization's ability/willingness or contractual or regulatory obligation…" [emphasis added].

Many thanks for those clarifications; glad to hear your concurrence that there are various scenarios in which the frequency (probability) of occurrence can change rather than being static.  I also appreciate the theoretical discussion (but phew, a lot for a caveman like me to unpack there; my apologies for five paragraphs of thoughts below!).  My experience has been that real-world practice tends to demand adjustments once we leave the risk management classroom or webinar.  I explain further below and would be interested to hear any additional thoughts.

I've found that real-world practice doesn't always accommodate the theory asserting that harm doesn't occur unless the device is being used; nor the theory that the discussion must include the concept of use (presuming that "use" means "use" rather than something else).  For example, a common harm for many types of devices comes from not being able to use the device as planned (e.g., from delayed diagnosis, surgery, or other therapy or medical procedure).  Accordingly, a procedural mandate forcing probability estimation only from the vantage point of actual device use would often result in an incomplete risk analysis.  Instead, a proper risk management program needs to allow for various different probability estimation paradigms [such as ones based on sales volumes, lifetime (see ISO/TR 24971 Table 3), durations, patient populations, etc.] in addition to device usage.  I think that's probably why ISO 14971 so clearly emphasizes such liberty.

Thanks also for the illustration of the 'rate-never-changes' principle.  That is helpful to see.  I'm wondering if you can take it a step further and give us an example or two of a particular risk management task where that principle could be put to use?  I ask because of the possibility that the 'rate-never-changes' principle might be of little or no practical value outside the classroom or beyond a time-zero initial guess about probability of occurrence.

Here's what I mean: My experience is that ongoing risk management and regulatory agencies alike are focused on actual, tangible, observable, measurable occurrences rather than theoretical concepts that can't actually be seen.  In other words, effective ongoing attention to public health and device quality using the risk management process boils down to properly managing what is observed and/or measured.  As I see it, if we can't truly know the actual number of uses, then the 'rate-never-changes principle', while theoretically interesting, seems largely moot for the purposes of practicing meaningful risk management.  Instead, it has been my experience that number of actual occurrences compared against a tangible, measurable denominator seems to be the intent of ISO 14971's objectives for considering probability of occurrence and managing device risk accordingly.

Hi Kevin,

Thank you for the examples. I'll address each one in turn.

First however, it is import to clarify the information sought. The question is "Given a harm with a specific severity, how often does that harm occur?" Harm doesn't occur unless the device is being used, so the response must include the concept of use. The typical answer is qualitative (often, hardly ever, seldom, etc.) or qualitative. The qualitative response typically takes one of two forms: a point estimate (once in every 1234 uses) or a range (between once in 100 uses and once in every 1,000 uses). In my response I'll use the point estimate and save issues with the range for another day.

The definition of use depends on the device. Some uses are discrete (the home use blood pressure machine) and some are continuous (the ventilator). The risk management team must define and document the use.

For the vital signs monitor the usage is continuous so time is a method to define use. To be consistent, let's use days. In the first case the harm occurs 365 times in every 365 usages. In the second case the harm occurs 1 time in every 365 days. I concur that the frequency of occurrence (probability) is higher in the first case than in the second.

For the single use device, the usage is discrete; the number of usages is the number sold. I concur that a harm of once per device used (sold) is a higher frequency of occurrence than 0.001 times per device used (sold).

For the lifetime example, things start to fall apart. The lifetime is not the number of uses. With out knowing the number of uses per year, or ten years, one cannot estimate the frequency of occurrence needed for medical device risk management.

For the Table 3 example, it is incorrect because it falls into the lifetime trap. Without understanding usage, the estimate for medical device risk management doesn't apply. This is more like a reliability or availability estimate.

Other than the lifetime examples I concur.

However, these examples don't address the service life or sales volumes questions. Consider a home use blood pressure machine used once per day. The usage is discrete. The usage is not equal to the sales. Let's say that a particular harm/severity combination happens once every 250 uses.

If there is 1 such machines in use, the harm occurs once in every 250 uses.
If there are 1,000 such machines in use, the harm occurs once in every 250 uses.
If there are 5,000 such machines in use, the harm occurs once in every 250 uses.

Notice that the occurrence rate stays the same. This is the estimate needed for medical device risk management.

Also notice that the number of occurrences changes with the number of machines in use. The rate times the number is use is the estimate needed to staff the complaint handling unit.

A similar situation applies to changes in lifetime. The rate stays the same, but the number of occurrences depends on the device life.

• If a particular harm is estimated to happen once per day with a vital signs monitor model, then the probability would seem to be different (higher) than if that harm is estimated to happen only once per year.

• If a particular harm is estimated to happen once per device (e.g., per use, per unit sold, etc.) for a single-use device, then the probability would seem to be different (higher) than if that harm is estimated to happen only 0.001 times per device (e.g., per use, per unit sold, etc.).  For example, one occurrence in 100 units sold = 0.01 occurrences/unit sold, whereas one occurrence in 100,000 units sold is only 0.00001 occurrences/unit sold.

• Or as another example, 100 occurrences over a ten-year lifetime = 10 occurrences / year, whereas 1 occurrence over a ten-year lifetime is only 0.1 occurrences per year.

• Or as a qualitative example:  Likely to happen several times during a ten-year device lifetime would be a higher probability than not likely to happen during a ten-year device lifetime (see Table 3 of ISO/TR 24971).

The probability discussion has many misconceptions.

The probability in ISO 14971:2019 relates to the harm with the stated severity. In other words, how often the harm occurs.

This is not a true probability, in the mathematical sense, but a rate. My understanding is that "probability" is a compromise word because ISO publishes the standards in multiple languages.

It is actually a rate, which requires a numerator and a denominator. The numerator is the number of occurrences of the harm with the stated severity and the denominator is the number of uses. Uses can have multiple measures. For a single use syringe, it is the use. For a home use blood pressure machine, it is taking one reading. For a CPAP machine is could be one sleep cycle or one hour of use. For a knee implant it could be one day. The team needs to agree on and understand the denominator. Include it in the risk management file.

The rate of harm is not the failure rate. A device could harm in normal operation and not harm in a fault condition. (A good design will include fail safe provisions.) In ISO 15971:2019 the analysis is for the hazard not the failure. This is why an FMECA is not the appropriate tool for medical device risk management.

The rate doesn't change based on sales or the size of the installed base. The rate is a characteristic of the device; the number of devices doesn't change the rate. If each harm results in a complaint, then the number of complaints is a function of the installed base size.

The rate doesn't change based on the life time of the device. Because the rate is a characteristic of the device, consider it constant over each year. An exception is harms related to wear out, but servicing addresses that issue. If each harm results in a complaint, then the number of complaints is a function of the life of the device.

When tackling the estimation of probability of occurrence of harm during medical device risk management, I think it is important to remember that there is no one-size-fits all approach.  Instead, each manufacturer's approach needs to be derived and appropriate based on the nature and complexity of the subject device, as well as on the availability (or lack thereof) of supporting data.

The particular approach can either be quantitative or qualitative (see ISO 14971 and ISO/TR 24971).  The probability can be estimated per use, per device, per hour of use, within a population, etc. (ISO/TR 24971).  The particular comparator (i.e., per use, per device, per hour of use, within a population, etc.) needs to be chosen based on the nature of the device.  For example, some devices' intended use (e.g., reusable devices) will intrinsically make the "per hour of use" comparator more appropriate than per device.  In contrast, single-use devices may very well be appropriate for a "per device" comparator.  The "per device" comparator is where sales numbers may be utilized.  As hinted at above, these are longstanding principles that have been part of the ISO 14971 risk management paradigm for many years.

When using a "per device sold" comparator, various factors (e.g., nature and complexity of the device; marketplace tenure, emergent risks or device problems, etc.) will determine whether it is appropriate to use sales per day, per week, per month, per year, cumulative all-time sales, or some other timeframe.  It is up to the manufacturer.  Just be sure that the chosen paradigm makes sense by giving meaningful insight toward the ultimate goal of properly managing the subject device's risk profile with regard to public health.

Consideration of device lifetime when estimating probability of occurrence harm is also a longstanding best-practice in risk management.  Indeed, seeds for this can be found more than two decades back in the year 2000 version of ISO 14971 and even in EN 1441 (1998) that was a gold standard before modern risk management really got traction via the ISO 14971 series.  Moreover, the consideration of device lifetime as part of estimating probability of occurrence harm remains a modern best practice today in the latest versions (see ISO 14971:2019 and ISO/TR 24971:2020), for example where it's stated that device lifetime is an important factor for estimating probability of occurrence of harm.  Accordingly, it is wise to figure out what is meant by device "lifetime" in the context of risk management and for the estimation of probability of occurrence of harm.

On that note, my experience has been that there seems to be harmonized general agreement on the meaning of device "lifetime".  That said, I haven't found many standardized, statutory, or legislative definitions for this.  At the moment, I generally rely on, adapt, and/or expand existing explanations such as:

• FDA's 21 CFR Part 803.3: "…(f) Expected life of a device means the time that a device is expected to remain functional after it is placed into use. Certain implanted devices have specified "end of life" (EOL) dates. Other devices are not labeled as to their respective EOL, but are expected to remain operational through activities such as maintenance, repairs, or upgrades, for an estimated period of time…"
• ISO/TR 24971:2020: "…What determines the lifetime of the medical device?...Factors that should be considered include battery depletion, deterioration of materials and failure of components due to ageing, wear, fatigue or repeated use. The availability of spare parts should be considered as well…" [emphasis added].
• FDA's 2016 3^rd Party Servicing Workshop: "…Information about reliability is needed to…understand the expected life of the device…" [emphasis added].
• ISO TC/210: "…the rationale for the determination [of medical device lifetime] should be recorded and can involve consideration of the following…shelf life of the medical device…expiry date for medical devices or components which are subject to degradation over time…number of cycles or periods of use of the medical device, based on life testing of the medical device…anticipated material degradation…stability of packaging material…for implantable devices, the residual risk that results from the entire period of residence of the medical device inside the patient's body…for sterile medical devices, the ability to maintain sterility… organization's ability/willingness or contractual or regulatory obligation…" [emphasis added].

Kevin,

I think there is a misunderstanding here. You said, "Many thanks for those clarifications; glad to hear your concurrence that there are various scenarios in which the frequency (probability) of occurrence can change rather than being static."

I didn't concur that the rate changes, only that the rates in your examples are different from each other.

The rate is static. It is, to the best of our knowledge, the rate determined as the residual risk. The rate is an inherent part of the design and changes when the design changes, i.e., different, or additional risk reduction measures. The device does "wake up one morning" and decide to switch from 11 harms in 1,000 uses to 541 harms in 1,000 uses.

During design the rate is the "best guess" of the team. Actual use data may improve the guess, but doesn't change the actual value.

Dan

Many thanks for those clarifications; glad to hear your concurrence that there are various scenarios in which the frequency (probability) of occurrence can change rather than being static.  I also appreciate the theoretical discussion (but phew, a lot for a caveman like me to unpack there; my apologies for five paragraphs of thoughts below!).  My experience has been that real-world practice tends to demand adjustments once we leave the risk management classroom or webinar.  I explain further below and would be interested to hear any additional thoughts.

I've found that real-world practice doesn't always accommodate the theory asserting that harm doesn't occur unless the device is being used; nor the theory that the discussion must include the concept of use (presuming that "use" means "use" rather than something else).  For example, a common harm for many types of devices comes from not being able to use the device as planned (e.g., from delayed diagnosis, surgery, or other therapy or medical procedure).  Accordingly, a procedural mandate forcing probability estimation only from the vantage point of actual device use would often result in an incomplete risk analysis.  Instead, a proper risk management program needs to allow for various different probability estimation paradigms [such as ones based on sales volumes, lifetime (see ISO/TR 24971 Table 3), durations, patient populations, etc.] in addition to device usage.  I think that's probably why ISO 14971 so clearly emphasizes such liberty.

Thanks also for the illustration of the 'rate-never-changes' principle.  That is helpful to see.  I'm wondering if you can take it a step further and give us an example or two of a particular risk management task where that principle could be put to use?  I ask because of the possibility that the 'rate-never-changes' principle might be of little or no practical value outside the classroom or beyond a time-zero initial guess about probability of occurrence.

Here's what I mean: My experience is that ongoing risk management and regulatory agencies alike are focused on actual, tangible, observable, measurable occurrences rather than theoretical concepts that can't actually be seen.  In other words, effective ongoing attention to public health and device quality using the risk management process boils down to properly managing what is observed and/or measured.  As I see it, if we can't truly know the actual number of uses, then the 'rate-never-changes principle', while theoretically interesting, seems largely moot for the purposes of practicing meaningful risk management.  Instead, it has been my experience that number of actual occurrences compared against a tangible, measurable denominator seems to be the intent of ISO 14971's objectives for considering probability of occurrence and managing device risk accordingly.

Hi Kevin,

Thank you for the examples. I'll address each one in turn.

First however, it is import to clarify the information sought. The question is "Given a harm with a specific severity, how often does that harm occur?" Harm doesn't occur unless the device is being used, so the response must include the concept of use. The typical answer is qualitative (often, hardly ever, seldom, etc.) or qualitative. The qualitative response typically takes one of two forms: a point estimate (once in every 1234 uses) or a range (between once in 100 uses and once in every 1,000 uses). In my response I'll use the point estimate and save issues with the range for another day.

The definition of use depends on the device. Some uses are discrete (the home use blood pressure machine) and some are continuous (the ventilator). The risk management team must define and document the use.

For the vital signs monitor the usage is continuous so time is a method to define use. To be consistent, let's use days. In the first case the harm occurs 365 times in every 365 usages. In the second case the harm occurs 1 time in every 365 days. I concur that the frequency of occurrence (probability) is higher in the first case than in the second.

For the single use device, the usage is discrete; the number of usages is the number sold. I concur that a harm of once per device used (sold) is a higher frequency of occurrence than 0.001 times per device used (sold).

For the lifetime example, things start to fall apart. The lifetime is not the number of uses. With out knowing the number of uses per year, or ten years, one cannot estimate the frequency of occurrence needed for medical device risk management.

For the Table 3 example, it is incorrect because it falls into the lifetime trap. Without understanding usage, the estimate for medical device risk management doesn't apply. This is more like a reliability or availability estimate.

Other than the lifetime examples I concur.

However, these examples don't address the service life or sales volumes questions. Consider a home use blood pressure machine used once per day. The usage is discrete. The usage is not equal to the sales. Let's say that a particular harm/severity combination happens once every 250 uses.

If there is 1 such machines in use, the harm occurs once in every 250 uses.
If there are 1,000 such machines in use, the harm occurs once in every 250 uses.
If there are 5,000 such machines in use, the harm occurs once in every 250 uses.

Notice that the occurrence rate stays the same. This is the estimate needed for medical device risk management.

Also notice that the number of occurrences changes with the number of machines in use. The rate times the number is use is the estimate needed to staff the complaint handling unit.

A similar situation applies to changes in lifetime. The rate stays the same, but the number of occurrences depends on the device life.

• If a particular harm is estimated to happen once per day with a vital signs monitor model, then the probability would seem to be different (higher) than if that harm is estimated to happen only once per year.

• If a particular harm is estimated to happen once per device (e.g., per use, per unit sold, etc.) for a single-use device, then the probability would seem to be different (higher) than if that harm is estimated to happen only 0.001 times per device (e.g., per use, per unit sold, etc.).  For example, one occurrence in 100 units sold = 0.01 occurrences/unit sold, whereas one occurrence in 100,000 units sold is only 0.00001 occurrences/unit sold.

• Or as another example, 100 occurrences over a ten-year lifetime = 10 occurrences / year, whereas 1 occurrence over a ten-year lifetime is only 0.1 occurrences per year.

• Or as a qualitative example:  Likely to happen several times during a ten-year device lifetime would be a higher probability than not likely to happen during a ten-year device lifetime (see Table 3 of ISO/TR 24971).

The probability discussion has many misconceptions.

The probability in ISO 14971:2019 relates to the harm with the stated severity. In other words, how often the harm occurs.

This is not a true probability, in the mathematical sense, but a rate. My understanding is that "probability" is a compromise word because ISO publishes the standards in multiple languages.

It is actually a rate, which requires a numerator and a denominator. The numerator is the number of occurrences of the harm with the stated severity and the denominator is the number of uses. Uses can have multiple measures. For a single use syringe, it is the use. For a home use blood pressure machine, it is taking one reading. For a CPAP machine is could be one sleep cycle or one hour of use. For a knee implant it could be one day. The team needs to agree on and understand the denominator. Include it in the risk management file.

The rate of harm is not the failure rate. A device could harm in normal operation and not harm in a fault condition. (A good design will include fail safe provisions.) In ISO 15971:2019 the analysis is for the hazard not the failure. This is why an FMECA is not the appropriate tool for medical device risk management.

The rate doesn't change based on sales or the size of the installed base. The rate is a characteristic of the device; the number of devices doesn't change the rate. If each harm results in a complaint, then the number of complaints is a function of the installed base size.

The rate doesn't change based on the life time of the device. Because the rate is a characteristic of the device, consider it constant over each year. An exception is harms related to wear out, but servicing addresses that issue. If each harm results in a complaint, then the number of complaints is a function of the life of the device.

When tackling the estimation of probability of occurrence of harm during medical device risk management, I think it is important to remember that there is no one-size-fits all approach.  Instead, each manufacturer's approach needs to be derived and appropriate based on the nature and complexity of the subject device, as well as on the availability (or lack thereof) of supporting data.

The particular approach can either be quantitative or qualitative (see ISO 14971 and ISO/TR 24971).  The probability can be estimated per use, per device, per hour of use, within a population, etc. (ISO/TR 24971).  The particular comparator (i.e., per use, per device, per hour of use, within a population, etc.) needs to be chosen based on the nature of the device.  For example, some devices' intended use (e.g., reusable devices) will intrinsically make the "per hour of use" comparator more appropriate than per device.  In contrast, single-use devices may very well be appropriate for a "per device" comparator.  The "per device" comparator is where sales numbers may be utilized.  As hinted at above, these are longstanding principles that have been part of the ISO 14971 risk management paradigm for many years.

When using a "per device sold" comparator, various factors (e.g., nature and complexity of the device; marketplace tenure, emergent risks or device problems, etc.) will determine whether it is appropriate to use sales per day, per week, per month, per year, cumulative all-time sales, or some other timeframe.  It is up to the manufacturer.  Just be sure that the chosen paradigm makes sense by giving meaningful insight toward the ultimate goal of properly managing the subject device's risk profile with regard to public health.

Consideration of device lifetime when estimating probability of occurrence harm is also a longstanding best-practice in risk management.  Indeed, seeds for this can be found more than two decades back in the year 2000 version of ISO 14971 and even in EN 1441 (1998) that was a gold standard before modern risk management really got traction via the ISO 14971 series.  Moreover, the consideration of device lifetime as part of estimating probability of occurrence harm remains a modern best practice today in the latest versions (see ISO 14971:2019 and ISO/TR 24971:2020), for example where it's stated that device lifetime is an important factor for estimating probability of occurrence of harm.  Accordingly, it is wise to figure out what is meant by device "lifetime" in the context of risk management and for the estimation of probability of occurrence of harm.

On that note, my experience has been that there seems to be harmonized general agreement on the meaning of device "lifetime".  That said, I haven't found many standardized, statutory, or legislative definitions for this.  At the moment, I generally rely on, adapt, and/or expand existing explanations such as:

• FDA's 21 CFR Part 803.3: "…(f) Expected life of a device means the time that a device is expected to remain functional after it is placed into use. Certain implanted devices have specified "end of life" (EOL) dates. Other devices are not labeled as to their respective EOL, but are expected to remain operational through activities such as maintenance, repairs, or upgrades, for an estimated period of time…"
• ISO/TR 24971:2020: "…What determines the lifetime of the medical device?...Factors that should be considered include battery depletion, deterioration of materials and failure of components due to ageing, wear, fatigue or repeated use. The availability of spare parts should be considered as well…" [emphasis added].
• FDA's 2016 3^rd Party Servicing Workshop: "…Information about reliability is needed to…understand the expected life of the device…" [emphasis added].
• ISO TC/210: "…the rationale for the determination [of medical device lifetime] should be recorded and can involve consideration of the following…shelf life of the medical device…expiry date for medical devices or components which are subject to degradation over time…number of cycles or periods of use of the medical device, based on life testing of the medical device…anticipated material degradation…stability of packaging material…for implantable devices, the residual risk that results from the entire period of residence of the medical device inside the patient's body…for sterile medical devices, the ability to maintain sterility… organization's ability/willingness or contractual or regulatory obligation…" [emphasis added].

Thanks Dan.  Hmm, obviously I'm still having a hard time finding a place in my day-to-day (after time-zero / early-stage guessing) risk management toolbelt for the universal 'rate-never-changes' principle.  For common practice of risk management, I'm still not seeing how that principle helps me as I operate the risk management system during the meat of a device's (or device-family's) lifetime.

I'm still seeing that if at timepoint 1, a harm happens, say, once per year for a certain install or sales base or model family, but then at timepoint 2 (e.g., due to emerging or new quality problems from manufacturing issues, material aging, new user habits, etc.) a harm starts happening at a higher frequency for that same install base, or sales base or model, then it means to me that the probability (rate) of occurrence has changed for that install base, or sales base or model.  Maybe that view isn't the correct view for purposes of mathematical theory, but it has served me well out in the field and also internally with my risk management clients and teammates.  In the vernacular of your narrative, yes, it seems to me that the device does literally wake up one morning and start behaving in a way that switches to (results in) a higher rate of harm.  In my experience, I've seen that happen time and time again (thus the reason a device/model can be safe from recall or redesign considerations one day, but then subject to recall or redesign the next).

When such changes do happen, then the risk analysis needs to be updated to revise the probability of occurrence of harm to reflect the increased probability (rate) as applicable to the affected units and/or model(s).  My experience has been that public health officials (like the FDA and its recall coordinators and OPEQ staff) also share this view.  Although proper statistics and quantitation is certainly important, I've found that health officials don't care for when I use clever mathematical gymnastics to characterize the essence of a situation.  At the end of the day, risk management is ultimately aimed to benefit people.  And people need tangible, actionable concepts and information in order to make important decisions.

Thanks again for helping a learner find the way.  I'll keep pondering about the benefits of the universal 'rate-never-changes' principle and its use during medical device risk management.

Kevin,

I think there is a misunderstanding here. You said, "Many thanks for those clarifications; glad to hear your concurrence that there are various scenarios in which the frequency (probability) of occurrence can change rather than being static."

I didn't concur that the rate changes, only that the rates in your examples are different from each other.

The rate is static. It is, to the best of our knowledge, the rate determined as the residual risk. The rate is an inherent part of the design and changes when the design changes, i.e., different, or additional risk reduction measures. The device does "wake up one morning" and decide to switch from 11 harms in 1,000 uses to 541 harms in 1,000 uses.

During design the rate is the "best guess" of the team. Actual use data may improve the guess, but doesn't change the actual value.

Dan

Many thanks for those clarifications; glad to hear your concurrence that there are various scenarios in which the frequency (probability) of occurrence can change rather than being static.  I also appreciate the theoretical discussion (but phew, a lot for a caveman like me to unpack there; my apologies for five paragraphs of thoughts below!).  My experience has been that real-world practice tends to demand adjustments once we leave the risk management classroom or webinar.  I explain further below and would be interested to hear any additional thoughts.

I've found that real-world practice doesn't always accommodate the theory asserting that harm doesn't occur unless the device is being used; nor the theory that the discussion must include the concept of use (presuming that "use" means "use" rather than something else).  For example, a common harm for many types of devices comes from not being able to use the device as planned (e.g., from delayed diagnosis, surgery, or other therapy or medical procedure).  Accordingly, a procedural mandate forcing probability estimation only from the vantage point of actual device use would often result in an incomplete risk analysis.  Instead, a proper risk management program needs to allow for various different probability estimation paradigms [such as ones based on sales volumes, lifetime (see ISO/TR 24971 Table 3), durations, patient populations, etc.] in addition to device usage.  I think that's probably why ISO 14971 so clearly emphasizes such liberty.

Thanks also for the illustration of the 'rate-never-changes' principle.  That is helpful to see.  I'm wondering if you can take it a step further and give us an example or two of a particular risk management task where that principle could be put to use?  I ask because of the possibility that the 'rate-never-changes' principle might be of little or no practical value outside the classroom or beyond a time-zero initial guess about probability of occurrence.

Here's what I mean: My experience is that ongoing risk management and regulatory agencies alike are focused on actual, tangible, observable, measurable occurrences rather than theoretical concepts that can't actually be seen.  In other words, effective ongoing attention to public health and device quality using the risk management process boils down to properly managing what is observed and/or measured.  As I see it, if we can't truly know the actual number of uses, then the 'rate-never-changes principle', while theoretically interesting, seems largely moot for the purposes of practicing meaningful risk management.  Instead, it has been my experience that number of actual occurrences compared against a tangible, measurable denominator seems to be the intent of ISO 14971's objectives for considering probability of occurrence and managing device risk accordingly.

Hi Kevin,

Thank you for the examples. I'll address each one in turn.

First however, it is import to clarify the information sought. The question is "Given a harm with a specific severity, how often does that harm occur?" Harm doesn't occur unless the device is being used, so the response must include the concept of use. The typical answer is qualitative (often, hardly ever, seldom, etc.) or qualitative. The qualitative response typically takes one of two forms: a point estimate (once in every 1234 uses) or a range (between once in 100 uses and once in every 1,000 uses). In my response I'll use the point estimate and save issues with the range for another day.

The definition of use depends on the device. Some uses are discrete (the home use blood pressure machine) and some are continuous (the ventilator). The risk management team must define and document the use.

For the vital signs monitor the usage is continuous so time is a method to define use. To be consistent, let's use days. In the first case the harm occurs 365 times in every 365 usages. In the second case the harm occurs 1 time in every 365 days. I concur that the frequency of occurrence (probability) is higher in the first case than in the second.

For the single use device, the usage is discrete; the number of usages is the number sold. I concur that a harm of once per device used (sold) is a higher frequency of occurrence than 0.001 times per device used (sold).

For the lifetime example, things start to fall apart. The lifetime is not the number of uses. With out knowing the number of uses per year, or ten years, one cannot estimate the frequency of occurrence needed for medical device risk management.

For the Table 3 example, it is incorrect because it falls into the lifetime trap. Without understanding usage, the estimate for medical device risk management doesn't apply. This is more like a reliability or availability estimate.

Other than the lifetime examples I concur.

However, these examples don't address the service life or sales volumes questions. Consider a home use blood pressure machine used once per day. The usage is discrete. The usage is not equal to the sales. Let's say that a particular harm/severity combination happens once every 250 uses.

If there is 1 such machines in use, the harm occurs once in every 250 uses.
If there are 1,000 such machines in use, the harm occurs once in every 250 uses.
If there are 5,000 such machines in use, the harm occurs once in every 250 uses.

Notice that the occurrence rate stays the same. This is the estimate needed for medical device risk management.

Also notice that the number of occurrences changes with the number of machines in use. The rate times the number is use is the estimate needed to staff the complaint handling unit.

A similar situation applies to changes in lifetime. The rate stays the same, but the number of occurrences depends on the device life.

• If a particular harm is estimated to happen once per day with a vital signs monitor model, then the probability would seem to be different (higher) than if that harm is estimated to happen only once per year.

• If a particular harm is estimated to happen once per device (e.g., per use, per unit sold, etc.) for a single-use device, then the probability would seem to be different (higher) than if that harm is estimated to happen only 0.001 times per device (e.g., per use, per unit sold, etc.).  For example, one occurrence in 100 units sold = 0.01 occurrences/unit sold, whereas one occurrence in 100,000 units sold is only 0.00001 occurrences/unit sold.

• Or as another example, 100 occurrences over a ten-year lifetime = 10 occurrences / year, whereas 1 occurrence over a ten-year lifetime is only 0.1 occurrences per year.

• Or as a qualitative example:  Likely to happen several times during a ten-year device lifetime would be a higher probability than not likely to happen during a ten-year device lifetime (see Table 3 of ISO/TR 24971).

The probability discussion has many misconceptions.

The probability in ISO 14971:2019 relates to the harm with the stated severity. In other words, how often the harm occurs.

This is not a true probability, in the mathematical sense, but a rate. My understanding is that "probability" is a compromise word because ISO publishes the standards in multiple languages.

It is actually a rate, which requires a numerator and a denominator. The numerator is the number of occurrences of the harm with the stated severity and the denominator is the number of uses. Uses can have multiple measures. For a single use syringe, it is the use. For a home use blood pressure machine, it is taking one reading. For a CPAP machine is could be one sleep cycle or one hour of use. For a knee implant it could be one day. The team needs to agree on and understand the denominator. Include it in the risk management file.

The rate of harm is not the failure rate. A device could harm in normal operation and not harm in a fault condition. (A good design will include fail safe provisions.) In ISO 15971:2019 the analysis is for the hazard not the failure. This is why an FMECA is not the appropriate tool for medical device risk management.

The rate doesn't change based on sales or the size of the installed base. The rate is a characteristic of the device; the number of devices doesn't change the rate. If each harm results in a complaint, then the number of complaints is a function of the installed base size.

The rate doesn't change based on the life time of the device. Because the rate is a characteristic of the device, consider it constant over each year. An exception is harms related to wear out, but servicing addresses that issue. If each harm results in a complaint, then the number of complaints is a function of the life of the device.

When tackling the estimation of probability of occurrence of harm during medical device risk management, I think it is important to remember that there is no one-size-fits all approach.  Instead, each manufacturer's approach needs to be derived and appropriate based on the nature and complexity of the subject device, as well as on the availability (or lack thereof) of supporting data.

The particular approach can either be quantitative or qualitative (see ISO 14971 and ISO/TR 24971).  The probability can be estimated per use, per device, per hour of use, within a population, etc. (ISO/TR 24971).  The particular comparator (i.e., per use, per device, per hour of use, within a population, etc.) needs to be chosen based on the nature of the device.  For example, some devices' intended use (e.g., reusable devices) will intrinsically make the "per hour of use" comparator more appropriate than per device.  In contrast, single-use devices may very well be appropriate for a "per device" comparator.  The "per device" comparator is where sales numbers may be utilized.  As hinted at above, these are longstanding principles that have been part of the ISO 14971 risk management paradigm for many years.

When using a "per device sold" comparator, various factors (e.g., nature and complexity of the device; marketplace tenure, emergent risks or device problems, etc.) will determine whether it is appropriate to use sales per day, per week, per month, per year, cumulative all-time sales, or some other timeframe.  It is up to the manufacturer.  Just be sure that the chosen paradigm makes sense by giving meaningful insight toward the ultimate goal of properly managing the subject device's risk profile with regard to public health.

Consideration of device lifetime when estimating probability of occurrence harm is also a longstanding best-practice in risk management.  Indeed, seeds for this can be found more than two decades back in the year 2000 version of ISO 14971 and even in EN 1441 (1998) that was a gold standard before modern risk management really got traction via the ISO 14971 series.  Moreover, the consideration of device lifetime as part of estimating probability of occurrence harm remains a modern best practice today in the latest versions (see ISO 14971:2019 and ISO/TR 24971:2020), for example where it's stated that device lifetime is an important factor for estimating probability of occurrence of harm.  Accordingly, it is wise to figure out what is meant by device "lifetime" in the context of risk management and for the estimation of probability of occurrence of harm.

On that note, my experience has been that there seems to be harmonized general agreement on the meaning of device "lifetime".  That said, I haven't found many standardized, statutory, or legislative definitions for this.  At the moment, I generally rely on, adapt, and/or expand existing explanations such as:

• FDA's 21 CFR Part 803.3: "…(f) Expected life of a device means the time that a device is expected to remain functional after it is placed into use. Certain implanted devices have specified "end of life" (EOL) dates. Other devices are not labeled as to their respective EOL, but are expected to remain operational through activities such as maintenance, repairs, or upgrades, for an estimated period of time…"
• ISO/TR 24971:2020: "…What determines the lifetime of the medical device?...Factors that should be considered include battery depletion, deterioration of materials and failure of components due to ageing, wear, fatigue or repeated use. The availability of spare parts should be considered as well…" [emphasis added].
• FDA's 2016 3^rd Party Servicing Workshop: "…Information about reliability is needed to…understand the expected life of the device…" [emphasis added].
• ISO TC/210: "…the rationale for the determination [of medical device lifetime] should be recorded and can involve consideration of the following…shelf life of the medical device…expiry date for medical devices or components which are subject to degradation over time…number of cycles or periods of use of the medical device, based on life testing of the medical device…anticipated material degradation…stability of packaging material…for implantable devices, the residual risk that results from the entire period of residence of the medical device inside the patient's body…for sterile medical devices, the ability to maintain sterility… organization's ability/willingness or contractual or regulatory obligation…" [emphasis added].

Kevin,

First distinguish between the number of occurrences and rate of occurrence. The number is what shows up as complaints. The rate could be the number of complaints per use.

My assertion is that, with perfect knowledge, the rate of a certain harm with a specified severity is the rate set at determining the residual risk. It is inherent in the device and, unless something changes it is fixed.

Now consider the case where something changes; the rate could change.

A design improvement applied to the devices in use, could lower the rate to a new fixed level.

An unaddressed wear-out failure could change the rate. Preventive maintenance should prevent it from happening.

A change in use conditions could change the rate.

Post-market surveillance should detect the unintended (not design change) rate changes leading to an investigation and determination of the cause. With this information the manufacturer should work to restore the status quo ante. One method is an ISO 13485:2016 advisory notice explaining the situation.

I have been arguing that the harm rate is implicit in the device which includes intended use, proper maintenance, etc. In other words, the design includes a set of conditions that sets the rate. The conditions don't include, for example, sales volume.

An interesting issue is the EU-MDR Article 88 trend reporting for non-serious incidents. The system requires reporting when there is a statistically significant increase in the rate. In other words, look for changes and, should a change occur, report it to the CA.

The implicit assumption here is that the rate is fixed and were it to change, then the CA requires investigation and reporting.

We can save the trend detection mechanism for another day.

Thanks Dan.  Hmm, obviously I'm still having a hard time finding a place in my day-to-day (after time-zero / early-stage guessing) risk management toolbelt for the universal 'rate-never-changes' principle.  For common practice of risk management, I'm still not seeing how that principle helps me as I operate the risk management system during the meat of a device's (or device-family's) lifetime.

I'm still seeing that if at timepoint 1, a harm happens, say, once per year for a certain install or sales base or model family, but then at timepoint 2 (e.g., due to emerging or new quality problems from manufacturing issues, material aging, new user habits, etc.) a harm starts happening at a higher frequency for that same install base, or sales base or model, then it means to me that the probability (rate) of occurrence has changed for that install base, or sales base or model.  Maybe that view isn't the correct view for purposes of mathematical theory, but it has served me well out in the field and also internally with my risk management clients and teammates.  In the vernacular of your narrative, yes, it seems to me that the device does literally wake up one morning and start behaving in a way that switches to (results in) a higher rate of harm.  In my experience, I've seen that happen time and time again (thus the reason a device/model can be safe from recall or redesign considerations one day, but then subject to recall or redesign the next).

When such changes do happen, then the risk analysis needs to be updated to revise the probability of occurrence of harm to reflect the increased probability (rate) as applicable to the affected units and/or model(s).  My experience has been that public health officials (like the FDA and its recall coordinators and OPEQ staff) also share this view.  Although proper statistics and quantitation is certainly important, I've found that health officials don't care for when I use clever mathematical gymnastics to characterize the essence of a situation.  At the end of the day, risk management is ultimately aimed to benefit people.  And people need tangible, actionable concepts and information in order to make important decisions.

Thanks again for helping a learner find the way.  I'll keep pondering about the benefits of the universal 'rate-never-changes' principle and its use during medical device risk management.

Kevin,

I think there is a misunderstanding here. You said, "Many thanks for those clarifications; glad to hear your concurrence that there are various scenarios in which the frequency (probability) of occurrence can change rather than being static."

I didn't concur that the rate changes, only that the rates in your examples are different from each other.

The rate is static. It is, to the best of our knowledge, the rate determined as the residual risk. The rate is an inherent part of the design and changes when the design changes, i.e., different, or additional risk reduction measures. The device does "wake up one morning" and decide to switch from 11 harms in 1,000 uses to 541 harms in 1,000 uses.

During design the rate is the "best guess" of the team. Actual use data may improve the guess, but doesn't change the actual value.

Dan

Many thanks for those clarifications; glad to hear your concurrence that there are various scenarios in which the frequency (probability) of occurrence can change rather than being static.  I also appreciate the theoretical discussion (but phew, a lot for a caveman like me to unpack there; my apologies for five paragraphs of thoughts below!).  My experience has been that real-world practice tends to demand adjustments once we leave the risk management classroom or webinar.  I explain further below and would be interested to hear any additional thoughts.

I've found that real-world practice doesn't always accommodate the theory asserting that harm doesn't occur unless the device is being used; nor the theory that the discussion must include the concept of use (presuming that "use" means "use" rather than something else).  For example, a common harm for many types of devices comes from not being able to use the device as planned (e.g., from delayed diagnosis, surgery, or other therapy or medical procedure).  Accordingly, a procedural mandate forcing probability estimation only from the vantage point of actual device use would often result in an incomplete risk analysis.  Instead, a proper risk management program needs to allow for various different probability estimation paradigms [such as ones based on sales volumes, lifetime (see ISO/TR 24971 Table 3), durations, patient populations, etc.] in addition to device usage.  I think that's probably why ISO 14971 so clearly emphasizes such liberty.

Thanks also for the illustration of the 'rate-never-changes' principle.  That is helpful to see.  I'm wondering if you can take it a step further and give us an example or two of a particular risk management task where that principle could be put to use?  I ask because of the possibility that the 'rate-never-changes' principle might be of little or no practical value outside the classroom or beyond a time-zero initial guess about probability of occurrence.

Here's what I mean: My experience is that ongoing risk management and regulatory agencies alike are focused on actual, tangible, observable, measurable occurrences rather than theoretical concepts that can't actually be seen.  In other words, effective ongoing attention to public health and device quality using the risk management process boils down to properly managing what is observed and/or measured.  As I see it, if we can't truly know the actual number of uses, then the 'rate-never-changes principle', while theoretically interesting, seems largely moot for the purposes of practicing meaningful risk management.  Instead, it has been my experience that number of actual occurrences compared against a tangible, measurable denominator seems to be the intent of ISO 14971's objectives for considering probability of occurrence and managing device risk accordingly.

Hi Kevin,

Thank you for the examples. I'll address each one in turn.

First however, it is import to clarify the information sought. The question is "Given a harm with a specific severity, how often does that harm occur?" Harm doesn't occur unless the device is being used, so the response must include the concept of use. The typical answer is qualitative (often, hardly ever, seldom, etc.) or qualitative. The qualitative response typically takes one of two forms: a point estimate (once in every 1234 uses) or a range (between once in 100 uses and once in every 1,000 uses). In my response I'll use the point estimate and save issues with the range for another day.

The definition of use depends on the device. Some uses are discrete (the home use blood pressure machine) and some are continuous (the ventilator). The risk management team must define and document the use.

For the vital signs monitor the usage is continuous so time is a method to define use. To be consistent, let's use days. In the first case the harm occurs 365 times in every 365 usages. In the second case the harm occurs 1 time in every 365 days. I concur that the frequency of occurrence (probability) is higher in the first case than in the second.

For the single use device, the usage is discrete; the number of usages is the number sold. I concur that a harm of once per device used (sold) is a higher frequency of occurrence than 0.001 times per device used (sold).

For the lifetime example, things start to fall apart. The lifetime is not the number of uses. With out knowing the number of uses per year, or ten years, one cannot estimate the frequency of occurrence needed for medical device risk management.

For the Table 3 example, it is incorrect because it falls into the lifetime trap. Without understanding usage, the estimate for medical device risk management doesn't apply. This is more like a reliability or availability estimate.

Other than the lifetime examples I concur.

However, these examples don't address the service life or sales volumes questions. Consider a home use blood pressure machine used once per day. The usage is discrete. The usage is not equal to the sales. Let's say that a particular harm/severity combination happens once every 250 uses.

If there is 1 such machines in use, the harm occurs once in every 250 uses.
If there are 1,000 such machines in use, the harm occurs once in every 250 uses.
If there are 5,000 such machines in use, the harm occurs once in every 250 uses.

Notice that the occurrence rate stays the same. This is the estimate needed for medical device risk management.

Also notice that the number of occurrences changes with the number of machines in use. The rate times the number is use is the estimate needed to staff the complaint handling unit.

A similar situation applies to changes in lifetime. The rate stays the same, but the number of occurrences depends on the device life.

• If a particular harm is estimated to happen once per day with a vital signs monitor model, then the probability would seem to be different (higher) than if that harm is estimated to happen only once per year.

• If a particular harm is estimated to happen once per device (e.g., per use, per unit sold, etc.) for a single-use device, then the probability would seem to be different (higher) than if that harm is estimated to happen only 0.001 times per device (e.g., per use, per unit sold, etc.).  For example, one occurrence in 100 units sold = 0.01 occurrences/unit sold, whereas one occurrence in 100,000 units sold is only 0.00001 occurrences/unit sold.

• Or as another example, 100 occurrences over a ten-year lifetime = 10 occurrences / year, whereas 1 occurrence over a ten-year lifetime is only 0.1 occurrences per year.

• Or as a qualitative example:  Likely to happen several times during a ten-year device lifetime would be a higher probability than not likely to happen during a ten-year device lifetime (see Table 3 of ISO/TR 24971).

The probability discussion has many misconceptions.

The probability in ISO 14971:2019 relates to the harm with the stated severity. In other words, how often the harm occurs.

This is not a true probability, in the mathematical sense, but a rate. My understanding is that "probability" is a compromise word because ISO publishes the standards in multiple languages.

It is actually a rate, which requires a numerator and a denominator. The numerator is the number of occurrences of the harm with the stated severity and the denominator is the number of uses. Uses can have multiple measures. For a single use syringe, it is the use. For a home use blood pressure machine, it is taking one reading. For a CPAP machine is could be one sleep cycle or one hour of use. For a knee implant it could be one day. The team needs to agree on and understand the denominator. Include it in the risk management file.

The rate of harm is not the failure rate. A device could harm in normal operation and not harm in a fault condition. (A good design will include fail safe provisions.) In ISO 15971:2019 the analysis is for the hazard not the failure. This is why an FMECA is not the appropriate tool for medical device risk management.

The rate doesn't change based on sales or the size of the installed base. The rate is a characteristic of the device; the number of devices doesn't change the rate. If each harm results in a complaint, then the number of complaints is a function of the installed base size.

The rate doesn't change based on the life time of the device. Because the rate is a characteristic of the device, consider it constant over each year. An exception is harms related to wear out, but servicing addresses that issue. If each harm results in a complaint, then the number of complaints is a function of the life of the device.

When tackling the estimation of probability of occurrence of harm during medical device risk management, I think it is important to remember that there is no one-size-fits all approach.  Instead, each manufacturer's approach needs to be derived and appropriate based on the nature and complexity of the subject device, as well as on the availability (or lack thereof) of supporting data.

The particular approach can either be quantitative or qualitative (see ISO 14971 and ISO/TR 24971).  The probability can be estimated per use, per device, per hour of use, within a population, etc. (ISO/TR 24971).  The particular comparator (i.e., per use, per device, per hour of use, within a population, etc.) needs to be chosen based on the nature of the device.  For example, some devices' intended use (e.g., reusable devices) will intrinsically make the "per hour of use" comparator more appropriate than per device.  In contrast, single-use devices may very well be appropriate for a "per device" comparator.  The "per device" comparator is where sales numbers may be utilized.  As hinted at above, these are longstanding principles that have been part of the ISO 14971 risk management paradigm for many years.

When using a "per device sold" comparator, various factors (e.g., nature and complexity of the device; marketplace tenure, emergent risks or device problems, etc.) will determine whether it is appropriate to use sales per day, per week, per month, per year, cumulative all-time sales, or some other timeframe.  It is up to the manufacturer.  Just be sure that the chosen paradigm makes sense by giving meaningful insight toward the ultimate goal of properly managing the subject device's risk profile with regard to public health.

Consideration of device lifetime when estimating probability of occurrence harm is also a longstanding best-practice in risk management.  Indeed, seeds for this can be found more than two decades back in the year 2000 version of ISO 14971 and even in EN 1441 (1998) that was a gold standard before modern risk management really got traction via the ISO 14971 series.  Moreover, the consideration of device lifetime as part of estimating probability of occurrence harm remains a modern best practice today in the latest versions (see ISO 14971:2019 and ISO/TR 24971:2020), for example where it's stated that device lifetime is an important factor for estimating probability of occurrence of harm.  Accordingly, it is wise to figure out what is meant by device "lifetime" in the context of risk management and for the estimation of probability of occurrence of harm.

On that note, my experience has been that there seems to be harmonized general agreement on the meaning of device "lifetime".  That said, I haven't found many standardized, statutory, or legislative definitions for this.  At the moment, I generally rely on, adapt, and/or expand existing explanations such as:

• FDA's 21 CFR Part 803.3: "…(f) Expected life of a device means the time that a device is expected to remain functional after it is placed into use. Certain implanted devices have specified "end of life" (EOL) dates. Other devices are not labeled as to their respective EOL, but are expected to remain operational through activities such as maintenance, repairs, or upgrades, for an estimated period of time…"
• ISO/TR 24971:2020: "…What determines the lifetime of the medical device?...Factors that should be considered include battery depletion, deterioration of materials and failure of components due to ageing, wear, fatigue or repeated use. The availability of spare parts should be considered as well…" [emphasis added].
• FDA's 2016 3^rd Party Servicing Workshop: "…Information about reliability is needed to…understand the expected life of the device…" [emphasis added].
• ISO TC/210: "…the rationale for the determination [of medical device lifetime] should be recorded and can involve consideration of the following…shelf life of the medical device…expiry date for medical devices or components which are subject to degradation over time…number of cycles or periods of use of the medical device, based on life testing of the medical device…anticipated material degradation…stability of packaging material…for implantable devices, the residual risk that results from the entire period of residence of the medical device inside the patient's body…for sterile medical devices, the ability to maintain sterility… organization's ability/willingness or contractual or regulatory obligation…" [emphasis added].