Regulatory Open Forum

FDA QSR in §820.198(b) requires, “Sampling plans, when used, shall be written and based on a valid statistical rationale.”

Recently, I was asked to explain the term “valid statistical rationale”. I lapsed into a discussion of recognized consensus standards, published books, reviewing OC curves, etc. While I may (or not) have satisfied the questioner, I did not satisfy myself.

You are asked, “Here is a sampling plan I just developed. Could you tell me if it has a valid statistical rationale?” I would be hard pressed to employ criteria to provide an answer. I’m seeking advice on the phrase.

For background, my degrees are in mathematics and I do a lot of work in sampling plans. I’m seeking a technical answer rather than a “warm and fuzzy”.

------------------------------
Dan O'Leary
Swanzey NH
United States
------------------------------

Maybe my position is too simplistic, but I would consider a sampling plan that is consistent with ANSI/ASQ Z1.4 or other applicable consensus standard to be based on a valid statistical rationale.  To argue otherwise suggests a problem with the standard and I don't feel a need to explain or defend the standard.  I have seen sampling plans developed by qualified statisticians that were so complex that no one performing the sampling could understand them.  While they may have been based on a valid rationale they weren't of much practical use.

   

------------------------------
Robert Athy
Director, Quality and Regulatory
Meagan Medical Inc.
Vancouver WA
United States

Original Message:
Sent: 04-23-2016 16:22
From: Dan O'Leary
Subject: Valid Sampling Plan

FDA QSR in §820.198(b) requires, “Sampling plans, when used, shall be written and based on a valid statistical rationale.”

Recently, I was asked to explain the term “valid statistical rationale”. I lapsed into a discussion of recognized consensus standards, published books, reviewing OC curves, etc. While I may (or not) have satisfied the questioner, I did not satisfy myself.

You are asked, “Here is a sampling plan I just developed. Could you tell me if it has a valid statistical rationale?” I would be hard pressed to employ criteria to provide an answer. I’m seeking advice on the phrase.

For background, my degrees are in mathematics and I do a lot of work in sampling plans. I’m seeking a technical answer rather than a “warm and fuzzy”.

------------------------------
Dan O'Leary
Swanzey NH
United States
------------------------------

In the medical device industry, the C=0 Sampling Plan is generally expected because typically knowingly accepting some defects is not considered acceptable for medical devices. If you use ANSI Z1.4 for medical devices or another FDA-regulated industry and accept on a certain level of defects, you should have very strong rationale from a risk-based perspective for why this is acceptable.

------------------------------
Randall Wheeland RAC
Medical Device QA/RA Consultant
San Antonio TX
210-978-3083
United States

Original Message:
Sent: 24-Apr-2016 14:00
From: Robert Athy
Subject: Valid Sampling Plan

Maybe my position is too simplistic, but I would consider a sampling plan that is consistent with ANSI/ASQ Z1.4 or other applicable consensus standard to be based on a valid statistical rationale.  To argue otherwise suggests a problem with the standard and I don't feel a need to explain or defend the standard.  I have seen sampling plans developed by qualified statisticians that were so complex that no one performing the sampling could understand them.  While they may have been based on a valid rationale they weren't of much practical use.

   

------------------------------
Robert Athy
Director, Quality and Regulatory
Meagan Medical Inc.
Vancouver WA
United States

It is important to note, as a clarification, that lot attribute acceptance plans partition the lot into two parts. One part is in the sample and the other is not. The inspector classifies the items in the sample as conforming or non-conforming, counts the non-conforming items, and makes a decision to accept or reject the lot.

Regardless of the accept number, one doesn’t know the number of nonconforming items in the portion not in the sample. To be specific, if c=0 and the lot is accepted there could be nonconforming items remaining in the lot. Additionally, if c=2 and the lot is accepted with 1 nonconforming item in the sample, there could be only conforming items remaining in the lot.

Appropriately disposition any nonconforming items in the sample; they are not “accepted”.

Any sampling plan includes a risk of putting nonconforming items in stock. The OC curve describes that risk. With c=0 plans, including those in Z1.4, the probability of acceptance will usually be smaller, for a given process nonconforming rate, than with plans where c>0.

The only ways to guarantee 100% conforming items, is to either use a production process with a 0% nonconformance rate or to perform 100% inspection.

------------------------------
Dan O'Leary
Swanzey NH
United States

Original Message:
Sent: 17-May-2016 01:34
From: Randall Wheeland
Subject: Valid Sampling Plan

In the medical device industry, the C=0 Sampling Plan is generally expected because typically knowingly accepting some defects is not considered acceptable for medical devices. If you use ANSI Z1.4 for medical devices or another FDA-regulated industry and accept on a certain level of defects, you should have very strong rationale from a risk-based perspective for why this is acceptable.

------------------------------
Randall Wheeland RAC
Medical Device QA/RA Consultant
San Antonio TX
210-978-3083
United States

I agree totally. Published standards are valid. Overly complex methods by a statistician are probably valid, but not practical.

This makes me rethink the question. What about simple sampling plans? For example:

Sample size is 10% of the lot, Accept on zero.

Sample size is square_root(N) + 1, where N is the lot size. Accept on zero.

Sample size is always 17 (or the whole lot for smaller lots). Accept on zero.

If presented with any of these, or similar plans, what criteria would one use to assess validity and, as a result, be satisfactory for device manufacturing?

------------------------------
Dan O'Leary
Swanzey NH
United States

Original Message:
Sent: 04-23-2016 16:22
From: Dan O'Leary
Subject: Valid Sampling Plan

FDA QSR in §820.198(b) requires, “Sampling plans, when used, shall be written and based on a valid statistical rationale.”

Recently, I was asked to explain the term “valid statistical rationale”. I lapsed into a discussion of recognized consensus standards, published books, reviewing OC curves, etc. While I may (or not) have satisfied the questioner, I did not satisfy myself.

You are asked, “Here is a sampling plan I just developed. Could you tell me if it has a valid statistical rationale?” I would be hard pressed to employ criteria to provide an answer. I’m seeking advice on the phrase.

For background, my degrees are in mathematics and I do a lot of work in sampling plans. I’m seeking a technical answer rather than a “warm and fuzzy”.

------------------------------
Dan O'Leary
Swanzey NH
United States
------------------------------

Hi Dan.

I totally agree that consensus standards would be difficult for an auditor to question regarding their validity in general.  However, there is another key point in this whole issue (and I state this not necessarily for you but for others who might be thinking about this same sort of topic):

No matter how "valid" your plan is the execution is what will determine validity as much as the "plan" does.  In other words, you should theoretically be able to use any plan that you mention.  But let's take the following example.  You "manufacture" 500 units of the same "lot" of the device.  You are using square-root-n-plus-one plan.  That means you are going to need to ensure that 24 units (SR 500 = 22.3 and I always round up in this type of plan to ensure sufficient units were checked).  Great.  So, we know we need to same 24 units.  My quality person on the production floor is responsible for collecting or noting which units should be tested for conformity. The product is running over an 8 hour shift and you turn out between 60-65 product units per hour generally.  My quality person decides to marker 8 units in hour 1; 8 units in hour 5; and 8 units in hour 7.  I have my 24 units for testing - all good, right?  Not necessarily.  Statistical sampling plans for any purpose are meant to be randomized in order to provide you with the data necessary to prove the product units manufactured from that group or lot are all conforming.  So randomization is just as essential to the question as the "plan" is.

I spent about a decade in quality in the drug world before moving to RA.  And I can tell you we had two basic sampling plans that were used for larger testing requirements (e.g. product initially failed to meet specification for a parameter).  Either we ran SRn+1 or we did 1% of the total production lot.  Basic math says that when a product lot got over 10,000 pieces we actually could test a little less by doing SRn+1 but less than 10,000 were did a few less by testing 1%.  I can tell you that this rationale survived several audits without any determination that we were out of compliance or that our rationale did not meet reasonable consensus.

------------------------------
Victor Mencarelli
Sr. Manager - Regulatory Affairs
Hain Celestial Group
United States

Original Message:
Sent: 04-24-2016 18:29
From: Dan O'Leary
Subject: Valid Sampling Plan

I agree totally. Published standards are valid. Overly complex methods by a statistician are probably valid, but not practical.

This makes me rethink the question. What about simple sampling plans? For example:

Sample size is 10% of the lot, Accept on zero.

Sample size is square_root(N) + 1, where N is the lot size. Accept on zero.

Sample size is always 17 (or the whole lot for smaller lots). Accept on zero.

If presented with any of these, or similar plans, what criteria would one use to assess validity and, as a result, be satisfactory for device manufacturing?

------------------------------
Dan O'Leary
Swanzey NH
United States

Generally in order to answer your question I'd have to go back to the
process validation. In process characterization/validation you should
have a good idea of the process variability. You can use that variation
to drive the appropriate sampling plan for lot acceptance.

"Sampling plans" for acceptance generally are not valid if there isn't
an underlying process validation to start with. A common mistake I see
is to use sampling for incoming component inspection without knowing
anything about the underlying variation.

Ginger G


Original Message------

I agree totally. Published standards are valid. Overly complex methods by a statistician are probably valid, but not practical.

This makes me rethink the question. What about simple sampling plans? For example:

Sample size is 10% of the lot, Accept on zero.

Sample size is square_root(N) + 1, where N is the lot size. Accept on zero.

Sample size is always 17 (or the whole lot for smaller lots). Accept on zero.

If presented with any of these, or similar plans, what criteria would one use to assess validity and, as a result, be satisfactory for device manufacturing?

------------------------------
Dan O'Leary
Swanzey NH
United States
------------------------------

Original Message------

FDA QSR in §820.198(b) requires, “Sampling plans, when used, shall be written and based on a valid statistical rationale.”

Recently, I was asked to explain the term “valid statistical rationale”. I lapsed into a discussion of recognized consensus standards, published books, reviewing OC curves, etc. While I may (or not) have satisfied the questioner, I did not satisfy myself.

You are asked, “Here is a sampling plan I just developed. Could you tell me if it has a valid statistical rationale?” I would be hard pressed to employ criteria to provide an answer. I’m seeking advice on the phrase.

For background, my degrees are in mathematics and I do a lot of work in sampling plans. I’m seeking a technical answer rather than a “warm and fuzzy”.

------------------------------
Dan O'Leary
Swanzey NH
United States
------------------------------

You raise an interesting point. Lot attribute sampling plans, such as Z1.4 and c=0 are one method. However, a roll of something, such as cloth or paper, may not be formed into lots, so the lot attribute sampling plans would not apply.

In a homogeneous process, I don’t see why beginning/middle/end would be statistically valid. However, this type of sampling plan needs to come under any characteristics for determining validity.

------------------------------
Dan O'Leary
Swanzey NH
United States

Original Message:
Sent: 04-23-2016 16:22
From: Dan O'Leary
Subject: Valid Sampling Plan

FDA QSR in §820.198(b) requires, “Sampling plans, when used, shall be written and based on a valid statistical rationale.”

Recently, I was asked to explain the term “valid statistical rationale”. I lapsed into a discussion of recognized consensus standards, published books, reviewing OC curves, etc. While I may (or not) have satisfied the questioner, I did not satisfy myself.

You are asked, “Here is a sampling plan I just developed. Could you tell me if it has a valid statistical rationale?” I would be hard pressed to employ criteria to provide an answer. I’m seeking advice on the phrase.

For background, my degrees are in mathematics and I do a lot of work in sampling plans. I’m seeking a technical answer rather than a “warm and fuzzy”.

------------------------------
Dan O'Leary
Swanzey NH
United States
------------------------------

I’m continuing to work on this issue of a statistically valid sampling plan. I found a very interesting article from Dr. Wayne Taylor that addresses the problem.

Dr. Taylor is well known for his work in quality assurance, statistical techniques, and medical devices. He was a major contributor to the GHTF guidance document on Process Validation.

The article, http://www.variation.com/techlib/as-7.html, discusses the issue from the point of view of the operating characteristic curve. He says, “You should document the AQLs and LTPDs of all your sampling plans.” For a continuous stream of lots, I prefer RQL as the name of the point rather than LTPD. He also says, “Sampling plans are used to make product disposition decisions. They decide which lots of product to accept and release and which lots to reject and either rework or discard.” I tend to separate the accept/reject decision from the disposition decision. This is also the approach in 820.90 on nonconforming product.

My minor quibbles notwithstanding, I believe people involved in selecting sampling plans need to understand the points he makes. The article is about attribute sampling plans, but I believe it applies equally well to variables sampling plans such as Z1.9.

------------------------------
Dan O'Leary
Swanzey NH
United States

Original Message:
Sent: 23-Apr-2016 16:22
From: Dan O'Leary
Subject: Valid Sampling Plan

FDA QSR in §820.198(b) requires, “Sampling plans, when used, shall be written and based on a valid statistical rationale.”

Recently, I was asked to explain the term “valid statistical rationale”. I lapsed into a discussion of recognized consensus standards, published books, reviewing OC curves, etc. While I may (or not) have satisfied the questioner, I did not satisfy myself.

You are asked, “Here is a sampling plan I just developed. Could you tell me if it has a valid statistical rationale?” I would be hard pressed to employ criteria to provide an answer. I’m seeking advice on the phrase.

For background, my degrees are in mathematics and I do a lot of work in sampling plans. I’m seeking a technical answer rather than a “warm and fuzzy”.

------------------------------
Dan O'Leary
Swanzey NH
United States
------------------------------

The key words in the requirement that are often overlooked are valid and rationale, which brings us to risk-based management. I've successfully approached this over the years by using a variety of statistical methods. The important thing is to ensure that whatever method is used, it validates that controls are commensurate with risk.

------------------------------
Randall Wheeland RAC
Medical Device QA/RA Consultant
San Antonio TX
210-978-3083
United States

Original Message:
Sent: 15-May-2016 17:00
From: Dan O'Leary
Subject: Valid Sampling Plan

I’m continuing to work on this issue of a statistically valid sampling plan. I found a very interesting article from Dr. Wayne Taylor that addresses the problem.

Dr. Taylor is well known for his work in quality assurance, statistical techniques, and medical devices. He was a major contributor to the GHTF guidance document on Process Validation.

The article, http://www.variation.com/techlib/as-7.html, discusses the issue from the point of view of the operating characteristic curve. He says, “You should document the AQLs and LTPDs of all your sampling plans.” For a continuous stream of lots, I prefer RQL as the name of the point rather than LTPD. He also says, “Sampling plans are used to make product disposition decisions. They decide which lots of product to accept and release and which lots to reject and either rework or discard.” I tend to separate the accept/reject decision from the disposition decision. This is also the approach in 820.90 on nonconforming product.

My minor quibbles notwithstanding, I believe people involved in selecting sampling plans need to understand the points he makes. The article is about attribute sampling plans, but I believe it applies equally well to variables sampling plans such as Z1.9.

------------------------------
Dan O'Leary
Swanzey NH
United States