Regulatory Open Forum

 View Only
  • 1.  Sample Size for Design Verification of Capital Equipment

    This message was posted by a user wishing to remain anonymous
    Posted 05-Oct-2018 21:57
    This message was posted by a user wishing to remain anonymous

    Hi All,

    I'm trying to determine an adequate sample size that I could use for the design verification of capital equipment (i.e. MRI scanner), so only a limited number of devices can be produced here for the verification testing.

    Using Binomial distribution utilized to calculate the sample size at 95% confidence to detect no more than zero defect out of 95% of the population would require at-least 59 number of trials or samples. Since the device is reusable capital equipment the number of trials will be divided into three devices where each device will perform 20 trials with total of 60 trials. This will discover any inherit risk present in the device. The device gets 100% functionally tested during manufacturing.

    Is this sample size rational valid for reusable capital equipment?
    Can I divide number of trials or samples into multiple devices?


  • 2.  RE: Sample Size for Design Verification of Capital Equipment

    Posted 07-Oct-2018 20:32
    For this kind of capital equipment you wouldn't normally take a statistical sampling approach - there simply are not enough devices to do that effectively.

    Rather do (as you say you do) a comprehensive functional testing during manufacturer, followed by installation validation for each separate site.  There are often site variations anyway. 

    Effectively you are doing a Type test of one.

    ------------------------------
    Arthur Brandwood PhD FRAPS
    Founder and Principal Consultant
    Brandwood Biomedical
    Sydney, Australia
    ------------------------------



  • 3.  RE: Sample Size for Design Verification of Capital Equipment

    Posted 10-Oct-2018 07:12

    The question assumes that design verification requires making product. This naturally leads to the question of how many to make. The assumption doesn't hold, so the question of how many to make is not an issue.

     

    QSR, in 820.30(f), says, "Design verification shall confirm that the design output meets the design input requirements." Similarly, ISO 13485:2016, 7.3.6 says design verification ensures "that the design and development outputs have met the design and development

    input requirements". Notice that neither requirement includes product testing.

     

    The assumption is that design verification requires product testing, but it does not. Over time, the concepts of design verification and manufacturability have been conflated.

     

    Design verification is a check that all of the design inputs are represented in the design outputs. This can, except in rare cases, be done by comparing documents. For example, if the design input requires a 10 foot long detachable power cord with a North American plug, the design output will be a purchasing specification. Design verification ensures that the item specified includes all of the attributes from the design input. I don't need to purchase 59 power cords for the design verification.

     

    Similarly, for a manufactured part, the question is whether all the design inputs are correctly represented in the design output documents, typically drawings and specifications. Again, design verification compares the two sets of documents; it does not require making the parts represented in the design output. The standard approach is some kind of map from design input to design output to ensure full coverage. ISO 13485:2016, 7.3.2.e now requires documentation of "the methods to ensure traceability of design and development outputs to design and development inputs".

     

    In terms of the design process the place you need a product is design validation. There, one product is sufficient.



    ------------------------------
    Dan O'Leary
    Swanzey NH
    United States
    ------------------------------



  • 4.  RE: Sample Size for Design Verification of Capital Equipment

    Posted 11-Oct-2018 08:58
    Hello Dan

    I respectfully disagree that design verification does not require testing (which is what your post says to me).  Design verification should be  verification that you made what you intended, not just checking that you have an output specification to cover the design requirement. Very few inspectors or authorities would let this be enough. Design Validation, as you know, is confirming you built what the customer wanted. (if you want to broadly define customer here as end-user and regulatory authorities (for regulatory requirements), perhaps you can cover the actual performance testing here but not typical to be this late in the process as verification). 

    Typically I see  representative prototype or final device performance testing as Design Verification, and Customer Use, HF or clinical studies or simulated use testing as Design Validation. 

    Best Regards,

    ------------------------------
    Ginger Cantor, MBA, RAC
    Founder/Principal Consultant
    Centaur Consulting LLC
    River Falls, Wisconsin 54022 USA
    715-307-1850
    centaurconsultingllc@gmail.com
    ------------------------------



  • 5.  RE: Sample Size for Design Verification of Capital Equipment

    Posted 11-Oct-2018 09:28

    Ginger,

     

    You have read the post correctly; I said that design verification does not require testing.

     

    The problem is that many people have adopted the convention that design verification does require testing. My assertion is that this is a confusion between manufacturability and design verification. QSR, for example, 820.30(f), says, "Design verification shall confirm that the design output meets the design input requirements." One can, in nearly all cases (I can't think of an exception), perform the confirmation without product testing or inspection.

     

    Design output is, generally, drawings and specifications, not product. There is no regulatory requirement, that I know of, that requires making product as a design output. Product doesn't "appear" until after design transfer.

     

    You say, "Very few inspectors or authorities would let this be enough." While that may be true it would be, in my opinion, an example of bad auditing, since this imposes expectations that are not regulatory requirements. This is particularly troublesome in an FDA Investigation because a 483 is an assertion of a violation of the law.

     

    You say, "Typically I see representative prototype or final device performance testing as Design Verification". This is a common approach, but not required. The representative prototype or final device isn't required until design validation.

     

    Returning to the initial post, if one follows the sample size convention cited, one representative prototype may not be sufficient; you could need 56!

     

    My point is that I can confirm that the design output meets the design input requirements by comparing documents. There is no regulatory requirement to convert the design output into product to confirm that it meets the design input requirements.

     

    If you know of such a requirement, please point it out to me.



    ------------------------------
    Dan O'Leary
    Swanzey NH
    United States
    ------------------------------



  • 6.  RE: Sample Size for Design Verification of Capital Equipment

    Posted 12-Oct-2018 10:31
    I would commend to your attention the definition at 21 CFR 820.3(g), which says in part that "...The total finished design output consists of the device, its packaging and labeling, and the device master record." [emphasis mine]. If verification means (at 21 CFR 820.3[aa]) "confirmation by objective evidence that specified requirements have been fulfilled," how does one confirm that with the regard to the device itself (not the DMR) without testing the relevant characteristics of the device?

    ------------------------------
    Michael Parmenter
    Director, Reg. Affairs & Quality Assurance
    Boulder CO
    United States
    ------------------------------



  • 7.  RE: Sample Size for Design Verification of Capital Equipment

    Posted 13-Oct-2018 13:50

    The issue here is not the definition of design output, but the tools and techniques to "confirm that the design output meets the design input requirements", i.e., to provide the objective evidence for design verification.

     

    The answer to your question is to build a trace matrix from each design input to each design output. Then confirm that the output meets the input. The record of the characteristic's checks, and their conformance is the objective evidence. This can be done without testing. As in my original example, the design inputs for a power cord leads to a design output as a purchasing specification, then compare the design inputs to the purchasing specification to verify the characteristics. There is no need to buy a power cord and test it.



    ------------------------------
    Dan O'Leary
    Swanzey NH
    United States
    ------------------------------



  • 8.  RE: Sample Size for Design Verification of Capital Equipment

    Posted 12-Oct-2018 11:01

    Hi Colleagues:

    Whilst Dan is technically and literally accurate - within both 21 CFR 820.30(f) as well as ISO 13485, 7.3.6 (Design Verification), there is no language requiring testing, for a number of devices, including the example specified by Anon in his/her original post (the MRI system), there are many design verification requirements which are subject to inspection or investigation by FDA as well as EU Notified Bodies which require objective evidence of conformity and that objective evidence is normally test reports where the sample was a finished device (and some test reports by only an accredited test lab will be accepted - and generally the lab records the finished device serial or lot number for traceability.).  For example, with regard to active medical devices in general, there are regulatory requirements to verify leakage current does not exceed maximums.  If the active device is a surgical laser, there are regulatory requirements to verify the correct wavelength, power output, and time it takes the shutter to close.  For an orthpaedic surgical robot, there are regulatory requirements to verify the robot only cuts the targeted bone and not the nearby artery.  This is what Ginger was getting at - the practical and realistic side of the answer to the question - which is what I believe is important and valuable to Anon so he/she makes the right decision - not what theoretically should be audited or investigated.

    Now, having some familiarity with design verification for the example chosen by Anon - the MRI system, there's not a single manufacturer of any Diagnostic Imaging device I'm aware of who got through QSR inspection or notified body audit of design verification without testing.  And they all can't be guilty of bad auditing.

    Finally, to Anon's specific query about Sample Size - my experience is with both FDA and Notified Bodies, where the device is priced at hundreds of thousands if not millions of dollars/Euros, the reviewers and investigators/auditors generally cut the manufacturer 'some slack' and expect at least a single finished device - for such verifications as leakage current, EMC, energy/radiation testing, etc.  On whether or not in theory testing is required for design verification - I don't have a dog for that fight.

    Respectfully (and hopefully realistically) submitted,

    Glen



    ------------------------------
    Glen Emelock
    CRO Group Inc.
    Melrose, MA USA
    crogroup.com
    gemelock@crogroup.com
    888-881-5637
    ------------------------------



  • 9.  RE: Sample Size for Design Verification of Capital Equipment

    Posted 12-Oct-2018 12:39
    I think that it is more nuanced than "does require" or doesn't require" for DV testing. Yes, design verification is demonstrating that design outputs meet design inputs (or, in actual design jargon, that the design meets its requirements). The things is, there are different types of "design inputs."

    So, if the design input is "must fit in an 11" square" then a simple examination of the drawing or other design output can be "verification" and measuring that it can be done consistently is really part of a process/manufacturing qualification.

    However, if the design input (or requirement) is "must allow 200 J or energy to pass through over 1 hour without cracking" I have a hard time seeing how this would be "verified" without doing some testing. This is even more true if the input requirement has a tolerance or standard deviation embedded in it.

    There are some design input that could arguably do either, though I do tend to agree FDA inspectors would except to see testing on these. An example would be "must consistently deliver 40 N-m of torque to the tool." I can see scenarios where engineers could do enough math calculations from their solid model to demonstrate this, but I can also see that it could be much easier just to test the thing.

    the other Ginger

    ------------------------------
    Ginger Glaser RAC
    Chief Technology Officer
    MN
    ------------------------------