Regulatory Open Forum

 View Only
Expand all | Collapse all

Usability and State of the Art

  • 1.  Usability and State of the Art

    Posted 16-Mar-2018 16:11
    In an attempt to close a major non-conformity from our technical file review, we submitted an updated report to address an insufficiency in our current Usability report.  We had not updated the report from the requirements in EN 62366:2008 because the newer revision had not been published in the Official Journal.  However, our Notified Body was requiring it to close the non-conformity claiming it was not state of the art, we updated the report based on EN 62366-1:2015 with the User Interface of Unknown Provenance (UOUP) clause and submitted the report.  The following response was given as part of our latest review:

    The new report "does not include evidence of that usability has been handled according to state of the art.  However, since usability is handled in risk management and information regarding user risks are included in the IFU, the non-conformity can be regarded as a minor non-conformity.  Further, evidence of that usability is verified and validated according to state of the art may be followed up at coming surveillance activities."
    I am at a loss as to what could be more state of the art than the latest revision of the usability standard.  I have contemplated that maybe I should have included a reference to EN 60601-1-6:2010 / A1:2015 but all that really does is point you to EN ISO 14971 and EN 62366, so I don't understand the benefit of that for other than documentation purposes.  I cannot see going back and doing the comprehensive usability engineering process on devices that were designed and developed in 2004 and have not had any modifications to the user interface since.  Our risk management file has been been through the review process as well and been cleared as sufficient.
    Any advice on what they could be wanting to see and the direction we should take going forward would be appreciated.

    Thank you in advance,



    ------------------------------
    Olivia Mullen
    Compliance / Regulatory Affairs
    Birmingham AL
    United States
    ------------------------------


  • 2.  RE: Usability and State of the Art

    Posted 17-Mar-2018 06:17
    Dear Olivia,

    I admit, that Ido not fully understand the issue yet.
    So allow me to reply with two questions.

    1.) I understand, that you cited clause 5.10 of IEC 62366-1:2015 and presented a report based on the evaluation process mandated by Annex C?
    2.) When was your user interface developed and commercialized? Prior or after the publication of the standard in February 2015? Was there a change of the user interface after 2015-02?

    Greetings from the South of Germany, Uwe

    ------------------------------
    Uwe Zeller
    Biberach an der Riß
    Germany
    ------------------------------



  • 3.  RE: Usability and State of the Art

    Posted 17-Mar-2018 11:13
    As I noted in Uwe'' reply, I find myself somewhat confused by the NB comments.  My thinking is that since you indicated the User Interface was initially implemented in 2004, and that there have been no changes, then your Post Market information, specifically any reported issues concerning the User Interface would be a valid method to substantiate the viability of the User Interface, assuming there were no or negligible customer allegations of problems.  I would identify the quantity of devices within the installed base and therefore present the related interface issues in comparison.  This should then be presented as retrospective validation.

    Best Regards

    Bob

    ------------------------------
    Robert Schiffman RAC
    Principle Consultant
    Yelm WA
    United States
    ------------------------------



  • 4.  RE: Usability and State of the Art

    Posted 17-Mar-2018 11:21

    I see a few issues here that make me nervous. The shift from IEC 62366:2007 to IEC 62366-1:2015 is a major change. One of my fears has been that when the new standard is harmonized, Notified Bodies will require manufacturers to create whole new usability files based on the new version. In my opinion, this will be a major task that will not improve the product nor make it safer. It will be a non-value-added paperwork exercise that will make the shift to EN ISO 14971:2012 look simple.

    I always recommend that people declare user interfaces developed before the new version of the standard as User Interface of Unknown Provenance, UOUP, and therefore not subject to the new standard until chnaged. Following the definition in 3.15, UOUP means user interfaces developed before the new standard is available. There is no requirement, that I know of, to apply the new standard retroactively to UOUP. I know however, that Annex C, Example 1 covers the case of an unchanged legacy user interface evaluation using Annex C to determine conformance to the new standard. It is not a requirement, but a possible application.

    The next question is the meeting of state-of-the-art. The MDD, Annex I, Section 2 says, "The solutions adopted by the manufacturer for the design and construction of the devices must conform to safety principles, taking account of the generally acknowledged state of the art." Unfortunately, the MDD does not define state-of-the-art. Notice, however, it is the device that is state-of-the-art, not the supporting documentation.

    EN ISO 14971:2012, section D.4 has a discussion of state-of-the-art for a particular medical device. In this case, as in the MDD, the requirement is state-of-the-art for the device, not the application of a particular standard.

    In particular, I infer that the interface of the device did not change. It was equally safe both before and after you updated the usability engineering file. In other words, without a change in the requirement for the content of a user interface, it was already state-of-the-art.

    My first reaction is that the initial nonconformance is wrong, since there is no requirement to apply the new version of the standard to UOUP. In my opinion, the initial issue should have been why the medical device is not state-of-the-art as opposed to not following the most recent standard to create a usability engineering file.

    Having agreed to follow Annex C of the new standard, my next reaction is that, perhaps, there is information missing from the updated usability engineering file (C.2.1 to C.2.6). If this were the case, the nonconformance should have identified any gaps.

    Lastly, this is a poorly written nonconformance since it doesn't clearly state the requirement. The term, state-of-the-art, is ambiguous and the Notified Body should have provided a more explicit statement of what they mean. In particular, they need to explain why the device is not state-of-the-art, rather then why they don't like your usability engineering file.

    My recommendation is that you ask the Notified Body for specific clarification of their expectations of state-of-the-art relative to usability. Beyond getting this additional clarification, I'm at a loss to develop a plan to close the nonconformance. I'm not event convinced there is a non-conformance.



    ------------------------------
    Dan O'Leary
    Swanzey NH
    United States
    ------------------------------



  • 5.  RE: Usability and State of the Art

    Posted 17-Mar-2018 12:31
    ​Dan, my impression of the original intention of "state of the art" has been that it had nothing to do with the latest version of a standard, nor the latest version of the device, but with the state of the clinical art for the device's intended use.

    Thus, your latest device version could meet the latest technical standard, and still have fallen behind the state of the art.  From the perspective of those who would use the device, physicians or patients, there could be "better" devices out there.  This struck me as something of an echo of FDA's concept of clinical utility, but more consistent with the attitude of third-party payers who want to see more than just clinical safety and effectiveness; they want to see cost effectiveness as well.  The MDD may have wanted to see the device that was the best fit for the current clinical state of the art.  Or maybe not.

    I have had interactions with one NB on one Class III device that seemed consistent with this interpretation.  A manufacturer came out with a "new and improved" version of a device, and collected just enough clinical data to support the claim that it was actually improved.  The original version remained as good as it always was, meeting all current standards, benefits clearly outweighing risks, etc.  As it turned out, some customers loved the new version, but many others continued to strongly prefer the old version.  The question became why the NB should continue to certify the old one, when the data clearly showed that it was no longer the "state of the art" for its intended use.

    I was impressed that the NB looked at it from a clinical perspective, because, on the whole, I don't think the NBs have done a particularly good job in applying any aspects of the MDD that are clinically oriented.  I think that's because the NBs on the whole don' have enough clinical expertise to understand the clinical perspective, and therefore tend to re-interpret clinically driven requirements into technical requirements.  This was from a Class III review team, so probably not representative of the NB, much less NBs as a whole.

    What the term was actually supposed to mean, I can't say...and apparently the authors of the MDD couldn't say, either, lol.  I can only say that this is one other way I've heard it interpreted.

    ------------------------------
    Julie Omohundro, ex-RAC (US, GS), still an MBA
    Principal Consultant
    Class Three, LLC
    Durham, North Carolina, USA
    919-544-3366 (T)
    434-964-1614 (C)
    julie@class3devices.com
    ------------------------------



  • 6.  RE: Usability and State of the Art

    Posted 17-Mar-2018 13:23

    Your comments are a reason to go to ISO 14971:2007, Clause D.3 which has the following discussion.

     

     

    "State of the art" is used here to mean what is currently and generally accepted as good practice. Various methods can be used to determine "state of the art" for a particular medical device. Examples are:

    ⎯ standards used for the same or similar devices;

    ⎯ best practices as used in other devices of the same or similar type;

    ⎯ results of accepted scientific research.

    State of the art does not necessarily mean the most technologically advanced solution.

     

     

    The troubling issue for me is that, I infer, the device in questions did not change. The didn't it appears to me, apply the principle of state-of-the art to the device. Instead, it appears they asked for an update to the usability engineering file, which did not change the device.



    ------------------------------
    Dan O'Leary
    Swanzey NH
    United States
    ------------------------------



  • 7.  RE: Usability and State of the Art

    Posted 17-Mar-2018 14:02
    ​Thanks for the additional observations, Dan. 

    I think everyone finds this confusing, including the NBs.  I always find it discouraging when regulators don't define their terms.  Makes me wonder if they are equally confused, probably by the lawmakers, who may not be all that clear on things, either.

    ------------------------------
    Julie Omohundro, ex-RAC (US, GS), still an MBA
    Principal Consultant
    Class Three, LLC
    Durham, North Carolina, USA
    919-544-3366 (T)
    434-964-1614 (C)
    julie@class3devices.com
    ------------------------------



  • 8.  RE: Usability and State of the Art

    Posted 19-Mar-2018 10:56
    The devices were designed, developed and put into production in 2004.  Our clinical evidence was accepted and found sufficient, as well as our risk management file.  The user interface had remained unchanged until this technical file review when we had to update our IFU to remove the "CE Mark" from the glossary of symbols and change our indicator light color (which goes against the particular standard published in the EU Official Journal).  So I suppose based on the definition and requirements in Annex C, they could be expecting us to do a full blown usability study on those changes?

    Unfortunately, throughout the audit process, the most commonly used reason for this major non-conformance has been not meeting "state of the art" when it came to our application of standards.  We had taken the stance that if it had not been published in the Official Journal then it did not presume conformity, therefore we showed conformance to those listed in the Official Journal.  We have discussed and argued this point with them ad nauseam to no avail.  They are requiring conformance to the most current version of every standard that could be applied to us, including a reference to an amendment that was just issued in October of 2017.

    We have spent countless hours over the past year doing nothing more than a futile exercise in documentation that has not made our device any safer.

    Thank you for your feedback.

    ------------------------------
    Olivia Mullen
    Compliance / Regulatory Affairs
    Birmingham AL
    United States
    ------------------------------



  • 9.  RE: Usability and State of the Art

    Posted 19-Mar-2018 12:51
    Olivia:
    We have spent countless hours over the past year doing nothing more than a futile exercise in documentation that has not made our device any safer.
    Here is what you should have done to quickly address such a non-conformance. 

    By making a reference to the latest standard (whether you make a claim of compliance or not), you provide scientifically sound and clinical meaningful justification using public information (e.g., why additional studies won't be necessary (e.g., usability not affected using literature info, if any)).

    In your CER and risk management, any potential usability-related safety signals/risks are to be covered as well.  

    Your NB would have accepted your rationale/justification without giving you a migraine headache or something to that effect! 

    David

    ------------------------------
    Dr. David Lim, Ph.D., RAC, ASQ-CQA
    REGULATORY DOCTOR
    http://www.RegulatoryDoctor.US
    http://www.GlobalComplianceSeminar.com
    Phone (Toll-Free): 1-(800) 321-8567
    E-mail: David@RegulatoryDoctor.US
    ------------------------------



  • 10.  RE: Usability and State of the Art

    Posted 21-Mar-2018 10:29
    Regarding State of the Art:

    Olivia,

    My recent experience aligns with yours. During a first time ISO 13485:2016 (not MDSAP) quality system certification for a company, the registrar insisted that the company have on hand copies of the most recent standards and regulations and that they had evidence showing their devices complied with all of these current standards and regulations as THE way to demonstrate the devices were "state of the art". Again, this audit did not include tech file review, only registration of the quality system to ISO 13485:2016. The lead auditor considered this a critical issue to address.

    I would recommend maintaining a matrix of all applicable regulations and standards. This may be maintained by device or company-wide with a column for applicable devices, or both ways if there are generic standards vs device-specific standards you want to manage separately. You should have an SOP for establishing and maintaining this matrix that defines minimum review cycles for these standards. For example, you may determine that you should review the matrix quarterly to determine whether any of the listed items have been updated or are about to be updated. As part of this SOP and process, establish a form to use to evaluate new and/or updated standards for your matrix. This form would basically be your checklist of considerations for reviewing devices or processes against a new or updated standard, and the completed form would become your evidence that you adequately reviewed the standard against impacted devices, including a list of or reference to action plans taken to implement the new or updated standard. Note that you must also report on this activity in your management review as part of the ISO 13485:2016 requirements for management review per section 5.6.2(l). The ISO auditor looked specifically for this in the management review records.

    I hope this helps.

    Julie,

    While I appreciate the idea of "state of the art" meaning more than just meeting current industry standards (I don't disagree), I think expanding this to mean more than that poses issues with then having to define for the company or for the device what the "state of the art" is. To me, that may lead the conversation with an inspector or regulator down a path of no return or put the client in a situation to provide even more evidence to back up their self-proclaimed definition of "state of the art."

    ------------------------------
    Christie Hughes
    Founder and Principal Consultant
    Integrys Consulting
    Austin,TX
    United States
    ------------------------------



  • 11.  RE: Usability and State of the Art

    Posted 21-Mar-2018 12:24
    I very much like this idea -- Thank you!

    ------------------------------
    Olivia Mullen
    Compliance / Regulatory Affairs
    Birmingham AL
    United States
    ------------------------------



  • 12.  RE: Usability and State of the Art

    Posted 21-Mar-2018 12:28
    Also, you might try to talk to senior management of the NB. At least three times I''ve had similar problems resolved in this manner (the analyst did not know enough, but senior management knew the correct requirements and overturned the analysis decision).

    ------------------------------
    Marcelo Antunes
    Regulatory Strategy Consultant
    São Paulo
    Brazil
    ------------------------------



  • 13.  RE: Usability and State of the Art

    Posted 28-Mar-2018 14:07
    ​​Christie, I fully appreciate the practical perspective,  I mostly meant to illustrate the range of possibilities that open up when legislators and regulators fail to define a key term.  It seems to me in practice this one has become mostly a generic non-conformity that NBs can use without need for substantive justification.  Who knows, maybe that was the whole point.

    ------------------------------
    Julie Omohundro, ex-RAC (US, GS), still an MBA
    Principal Consultant
    Class Three, LLC
    Durham, North Carolina, USA
    919-544-3366 (T)
    434-964-1614 (C)
    julie@class3devices.com
    ------------------------------



  • 14.  RE: Usability and State of the Art

    Posted 20-Mar-2018 07:31
    Curiously, the "legacy device" option in IEC 62366 (which was developed in Amendment 1 for the original version) was created due to the JWG 4 group (I was one of the persons that pushed this, due to standards being mandatory in Brazil) perceiving a problem as NBs were requiring full compliance with the standard for all devices (and I'm not even talking about the fact that standards are voluntary in Europe). 

    Regarding state of the art, we are finally including a definition in the revision of ISO 14971

    "3.28
    state of the art
    developed stage of technical capability at a given time as regards products, processes and services, based on the relevant consolidated findings of science, technology and experience

    Note 1 to entry: The state of the art embodies what is currently and generally accepted as good practice. The state of the art does not necessarily imply the most technologically advanced solution. The state of the art described here is sometimes referred to as the "generally acknowledged state of the art". "

    Also, state of the art is not related only to clinical aspects, but to technology in general. For example, it's very common to say that some solution is a state-of-the-art safety feature (such as using lead clothes to protect against x-rays and things like that).



    ------------------------------
    Marcelo Antunes
    Regulatory Strategy Consultant
    São Paulo
    Brazil
    ------------------------------



  • 15.  RE: Usability and State of the Art

    Posted 21-Mar-2018 12:11
    I requested clarification from our NB on the reviewer's comments and asking how it did not meet state of the art or how it was insufficient.  The reply:
    "All requirements in EN 62366-1:2015, and traceability to references to evidence of compliance, are not included."

    I made references to the MDD requirement and the purpose of the Usability standard and how compliance to the standard would be based on UOUP and why.  For C.2.1, I referenced our Use Specification document (by document number) and gave specifics as to how each requirement in section 5.1 was addressed in this document.  For C.2.2, I referenced our Hazard Traceability Matrix (by document number) and current Post Market surveillance report (by report number) to show Post Production information Reviews and how those reviews affected other documentation (like CAPAs, MDRs, risk review, etc).  For C.2.3, I referenced the Hazard Traceability Matrix again and made a table referencing specific hazards and hazardous situations in our matrix that dealt with Usability.  For the control portion in C.2.4, I took the same table of hazards and added a column for the controls used to mitigate the probability of occurrence of those hazards.  For C.2.5, I referenced the residual risk analysis done on hazards individually after mitigation and referenced the Risk Management Plan and Report (by document number) showing the overall residual risk analysis done for the device as a whole.

    All documents and reports referenced in the Usability report have been reviewed and accepted by the NB.
    Am I just missing something terribly?

    Thank you in advance,

    ------------------------------
    Olivia Mullen
    Compliance / Regulatory Affairs
    Birmingham AL
    United States
    ------------------------------



  • 16.  RE: Usability and State of the Art

    Posted 21-Mar-2018 12:19
    You are missing the fact that, as too too frequently occur, the NB does not know what he's doing, or prefers to take an "easier" instance (ask for compliance with a standard) than what the requirements define (standards are voluntary).

    A lot of NBs still do not know or bypass the fact that compliance must be reviewed against Essential Principles, and one of the ways is to use standards (and if you use Harmonized Standards you gain presumption of conformity). Some know that, but it's more work for them to verify a generic solution, so they keep pushing for compliance with standards. 

    Usually, my suggestion is to either
    - discuss with the NB the fact that standard are not mandatory (youo can use both the text from the directive or the Blue Book, or either) and trey to make him accept your alternative solution (if it's an alternative)
    - change the NB (not good to do now).

    ------------------------------
    Marcelo Antunes
    Regulatory Strategy Consultant
    São Paulo
    Brazil
    ------------------------------



  • 17.  RE: Usability and State of the Art

    This message was posted by a user wishing to remain anonymous
    Posted 29-Mar-2018 13:43
    This message was posted by a user wishing to remain anonymous

    Hi Olivia,

    I just read through up to your most recent update.  I take it the NB is still not ready to close the Minor Non-conformity?  Since they originally said they would follow-up at the next surveillance audit, you could even park it for now and discuss in greater depth at the audit.  You should then at least be able to get clarity on what the finding is.  The risk is that it could become a Major.

    I do just wonder if there is a clue in your statement:
    'For C.2.2, I referenced our Hazard Traceability Matrix (by document number) and current Post Market surveillance report (by report number) to show Post Production information Reviews and how those reviews affected other documentation (like CAPAs, MDRs, risk review, etc).  '

    It's hard to tell from here if what I am reading into your statement could be significant or not.  Did your PMS cover issues that you related to usability?  If it so happened that there were none reported in this PMS, as is very possible as the device has been on the market for so long, it could be argued that there is no direct evidence that you feed back usability information into the other processes.  I guess the ideal evidence would be a potentially usability related issue in PMS, that traces through risk, CAPA etc to some amended deliverable e.g. IFU, interface change, further evidence from users/clinicians.  

    Then again, I may be looking at it in completely the wrong way!  If tackled during surveillance, you could also use evidence from other items in your product range where there IS the linkage of usability through PMS -> Risk -> etc.

    R