Thanks for the additional observations, Dan.
I think everyone finds this confusing, including the NBs. I always find it discouraging when regulators don't define their terms. Makes me wonder if they are equally confused, probably by the lawmakers, who may not be all that clear on things, either.
Original Message:
Sent: 17-Mar-2018 13:22
From: Dan O'Leary
Subject: Usability and State of the Art
Your comments are a reason to go to ISO 14971:2007, Clause D.3 which has the following discussion.
"State of the art" is used here to mean what is currently and generally accepted as good practice. Various methods can be used to determine "state of the art" for a particular medical device. Examples are:
⎯ standards used for the same or similar devices;
⎯ best practices as used in other devices of the same or similar type;
⎯ results of accepted scientific research.
State of the art does not necessarily mean the most technologically advanced solution.
The troubling issue for me is that, I infer, the device in questions did not change. The didn't it appears to me, apply the principle of state-of-the art to the device. Instead, it appears they asked for an update to the usability engineering file, which did not change the device.
------------------------------
Dan O'Leary
Swanzey NH
United States
Original Message:
Sent: 17-Mar-2018 12:31
From: Julie Omohundro
Subject: Usability and State of the Art
Dan, my impression of the original intention of "state of the art" has been that it had nothing to do with the latest version of a standard, nor the latest version of the device, but with the state of the clinical art for the device's intended use.
Thus, your latest device version could meet the latest technical standard, and still have fallen behind the state of the art. From the perspective of those who would use the device, physicians or patients, there could be "better" devices out there. This struck me as something of an echo of FDA's concept of clinical utility, but more consistent with the attitude of third-party payers who want to see more than just clinical safety and effectiveness; they want to see cost effectiveness as well. The MDD may have wanted to see the device that was the best fit for the current clinical state of the art. Or maybe not.
I have had interactions with one NB on one Class III device that seemed consistent with this interpretation. A manufacturer came out with a "new and improved" version of a device, and collected just enough clinical data to support the claim that it was actually improved. The original version remained as good as it always was, meeting all current standards, benefits clearly outweighing risks, etc. As it turned out, some customers loved the new version, but many others continued to strongly prefer the old version. The question became why the NB should continue to certify the old one, when the data clearly showed that it was no longer the "state of the art" for its intended use.
I was impressed that the NB looked at it from a clinical perspective, because, on the whole, I don't think the NBs have done a particularly good job in applying any aspects of the MDD that are clinically oriented. I think that's because the NBs on the whole don' have enough clinical expertise to understand the clinical perspective, and therefore tend to re-interpret clinically driven requirements into technical requirements. This was from a Class III review team, so probably not representative of the NB, much less NBs as a whole.
What the term was actually supposed to mean, I can't say...and apparently the authors of the MDD couldn't say, either, lol. I can only say that this is one other way I've heard it interpreted.
------------------------------
Julie Omohundro, ex-RAC (US, GS), still an MBA
Principal Consultant
Class Three, LLC
Durham, North Carolina, USA
919-544-3366 (T)
434-964-1614 (C)
julie@class3devices.com
Original Message:
Sent: 17-Mar-2018 11:20
From: Dan O'Leary
Subject: Usability and State of the Art
I see a few issues here that make me nervous. The shift from IEC 62366:2007 to IEC 62366-1:2015 is a major change. One of my fears has been that when the new standard is harmonized, Notified Bodies will require manufacturers to create whole new usability files based on the new version. In my opinion, this will be a major task that will not improve the product nor make it safer. It will be a non-value-added paperwork exercise that will make the shift to EN ISO 14971:2012 look simple.
I always recommend that people declare user interfaces developed before the new version of the standard as User Interface of Unknown Provenance, UOUP, and therefore not subject to the new standard until chnaged. Following the definition in 3.15, UOUP means user interfaces developed before the new standard is available. There is no requirement, that I know of, to apply the new standard retroactively to UOUP. I know however, that Annex C, Example 1 covers the case of an unchanged legacy user interface evaluation using Annex C to determine conformance to the new standard. It is not a requirement, but a possible application.
The next question is the meeting of state-of-the-art. The MDD, Annex I, Section 2 says, "The solutions adopted by the manufacturer for the design and construction of the devices must conform to safety principles, taking account of the generally acknowledged state of the art." Unfortunately, the MDD does not define state-of-the-art. Notice, however, it is the device that is state-of-the-art, not the supporting documentation.
EN ISO 14971:2012, section D.4 has a discussion of state-of-the-art for a particular medical device. In this case, as in the MDD, the requirement is state-of-the-art for the device, not the application of a particular standard.
In particular, I infer that the interface of the device did not change. It was equally safe both before and after you updated the usability engineering file. In other words, without a change in the requirement for the content of a user interface, it was already state-of-the-art.
My first reaction is that the initial nonconformance is wrong, since there is no requirement to apply the new version of the standard to UOUP. In my opinion, the initial issue should have been why the medical device is not state-of-the-art as opposed to not following the most recent standard to create a usability engineering file.
Having agreed to follow Annex C of the new standard, my next reaction is that, perhaps, there is information missing from the updated usability engineering file (C.2.1 to C.2.6). If this were the case, the nonconformance should have identified any gaps.
Lastly, this is a poorly written nonconformance since it doesn't clearly state the requirement. The term, state-of-the-art, is ambiguous and the Notified Body should have provided a more explicit statement of what they mean. In particular, they need to explain why the device is not state-of-the-art, rather then why they don't like your usability engineering file.
My recommendation is that you ask the Notified Body for specific clarification of their expectations of state-of-the-art relative to usability. Beyond getting this additional clarification, I'm at a loss to develop a plan to close the nonconformance. I'm not event convinced there is a non-conformance.
------------------------------
Dan O'Leary
Swanzey NH
United States
Original Message:
Sent: 16-Mar-2018 16:10
From: Olivia Mullen
Subject: Usability and State of the Art
In an attempt to close a major non-conformity from our technical file review, we submitted an updated report to address an insufficiency in our current Usability report. We had not updated the report from the requirements in EN 62366:2008 because the newer revision had not been published in the Official Journal. However, our Notified Body was requiring it to close the non-conformity claiming it was not state of the art, we updated the report based on EN 62366-1:2015 with the User Interface of Unknown Provenance (UOUP) clause and submitted the report. The following response was given as part of our latest review:The new report "does not include evidence of that usability has been handled according to state of the art. However, since usability is handled in risk management and information regarding user risks are included in the IFU, the non-conformity can be regarded as a minor non-conformity. Further, evidence of that usability is verified and validated according to state of the art may be followed up at coming surveillance activities."
I am at a loss as to what could be more state of the art than the latest revision of the usability standard. I have contemplated that maybe I should have included a reference to EN 60601-1-6:2010 / A1:2015 but all that really does is point you to EN ISO 14971 and EN 62366, so I don't understand the benefit of that for other than documentation purposes. I cannot see going back and doing the comprehensive usability engineering process on devices that were designed and developed in 2004 and have not had any modifications to the user interface since. Our risk management file has been been through the review process as well and been cleared as sufficient.
Any advice on what they could be wanting to see and the direction we should take going forward would be appreciated.
Thank you in advance,
------------------------------
Olivia Mullen
Compliance / Regulatory Affairs
Birmingham AL
United States
------------------------------