Regulatory Open Forum

This message was posted by a user wishing to remain anonymous

How can one check the "effectiveness" of the training/ competency of an employee.
Is quizzing them enough?  Should the quiz records (Q&A) be maintained as a part of the training record? what if the questions are asked orally by the trainer? 

Thank you.

This is almost always the $64 million dollar question (yes, I account for inflation!).  Quizzes are a good starting point but the quiz has to be valid meaning that the answer to the questions cannot be "obvious" otherwise the quiz loses significance.  So what I have done is utilized some of my friends who are educators/teachers or trainers who do that sort of thing for a living - you know - assess understanding and knowledge.  I work with them to develop the training documentation and then provide them with the quiz I am proposing to use after the training is completed.  They can often tell me whether or not the quiz looks reasonable based on the materials and the quiz itself. 

The problem with quizzes is that you are often left with true/false or multiple choice type questions and answers.  This is because you will get such varied answers to short answer type questions that you will really need to parse through the responses to determine understanding.

Whenever you validate training or understanding in my opinion it is essential that you keep the backup documentation to show that the expected/required level of understanding has been achieved by the individual being trained.  So I would find a way to keep the actual quiz records somehow.  They don't in my opinion need to be kept in the employee's training folder or even in paper form so long as you can attest to the accuracy of the document being presented.  So scanning the actual document into an electronic file like a pdf would work just as well as keeping the actual paper but does not impose requirements to keep cases of paper documents on file.  Also, I tend to be a bit of a hoarder and keep all of the training documents (saved with the employee's name, ID number and date of completion of the training) even if you have several trainings on the same or similar SOP or topic.  This shows the auditor that you have control of your program and also have the ability to show when the initial training and any necessary retraining occurred.  So if you change an SOP you can show the employee was trained and understood the information from each specific version of the SOP that employee trained on.  If you only keep the most recent version and a question arises about something done 1 or 2 version back you have less to validate the understanding portion of the training (although you still might be able to show that the employee "attended" a training without the ability to show the employee's validated understanding of the process it could become more difficult to answer the auditor's main question which really is "did the employee understand what/how/why to do the steps and what the expected end result and documentation would look like?" when they performed that specific task).​

------------------------------
Victor Mencarelli
Director Regulatory Affairs
United States
------------------------------

Original Message:
Sent: 19-Jun-2019 11:35
From: Anonymous Member
Subject: Employee Training

This message was posted by a user wishing to remain anonymous

How can one check the "effectiveness" of the training/ competency of an employee.
Is quizzing them enough?  Should the quiz records (Q&A) be maintained as a part of the training record? what if the questions are asked orally by the trainer?

Thank you.

In addition to what Victor mentioned, you would have to careful with how you handle the quiz and marking for it. We have had one of our customer with quiz and 80% as pass percentage where FDA questioned it in one of the audits on how can you consider 80% as pass, as if it means 20% of errors can be made by them in Production. This made the organization to change their entire training process with 100% as pass percentage!

------------------------------
Loganathan Kumarasamy, MS RAC
Senior Consultant
Waukegan IL
United States
------------------------------

Original Message:
Sent: 19-Jun-2019 11:35
From: Anonymous Member
Subject: Employee Training

This message was posted by a user wishing to remain anonymous

How can one check the "effectiveness" of the training/ competency of an employee.
Is quizzing them enough?  Should the quiz records (Q&A) be maintained as a part of the training record? what if the questions are asked orally by the trainer?

Thank you.

There are many ways qualification and competency can be confirmed for employee training and giving a quiz is one method.  As mentioned by the others quizzes also need to be objective and meaningful - as stated giving a quiz with 80% pass ... is this ok for the position?  Maybe this is fine for understanding higher level procedures or SOPs.  But what about as Loganathan mentions - is it ok to have 80% acceptance rate for employees manufacturing actual finished devices?  In most companies the answer would be no.  Qualification and competency also differs based on the position, the proportion of risk they have against the finished product, and what retention needs to be made.  Personally, there are other competency methods considered in medical device companies, a good source is looking at principles of usability.  Another important part of qualification and competency is how is this maintained?  We know a person could be qualified through experience or training and they could be competent through experience or repetition of a task.  How do you know their competency is maintained at a sufficient level.  I can tell you from experience many engineering department do not maintain competency because procedures are not followed after 2 or 3 years.  Are the still competent? Some would argue no because they are not perform the task and process correctly.

------------------------------
Richard Vincins RAC
Vice President Global Regulatory Affairs
------------------------------

Original Message:
Sent: 20-Jun-2019 01:01
From: Loganathan Kumarasamy, MS RAC
Subject: Employee Training

In addition to what Victor mentioned, you would have to careful with how you handle the quiz and marking for it. We have had one of our customer with quiz and 80% as pass percentage where FDA questioned it in one of the audits on how can you consider 80% as pass, as if it means 20% of errors can be made by them in Production. This made the organization to change their entire training process with 100% as pass percentage!

------------------------------
Loganathan Kumarasamy, MS RAC
Senior Consultant
Waukegan IL
United States

Original Message:
Sent: 19-Jun-2019 11:35
From: Anonymous Member
Subject: Employee Training

This message was posted by a user wishing to remain anonymous

How can one check the "effectiveness" of the training/ competency of an employee.
Is quizzing them enough?  Should the quiz records (Q&A) be maintained as a part of the training record? what if the questions are asked orally by the trainer?

Thank you.

This message was posted by a user wishing to remain anonymous

Thank you everyone for your response. There were some really important, insightful points that i have to consider to setup up employee training.
Original Message:
Sent: 20-Jun-2019 04:51
From: Richard Vincins
Subject: Employee Training

There are many ways qualification and competency can be confirmed for employee training and giving a quiz is one method.  As mentioned by the others quizzes also need to be objective and meaningful - as stated giving a quiz with 80% pass ... is this ok for the position?  Maybe this is fine for understanding higher level procedures or SOPs.  But what about as Loganathan mentions - is it ok to have 80% acceptance rate for employees manufacturing actual finished devices?  In most companies the answer would be no.  Qualification and competency also differs based on the position, the proportion of risk they have against the finished product, and what retention needs to be made.  Personally, there are other competency methods considered in medical device companies, a good source is looking at principles of usability.  Another important part of qualification and competency is how is this maintained?  We know a person could be qualified through experience or training and they could be competent through experience or repetition of a task.  How do you know their competency is maintained at a sufficient level.  I can tell you from experience many engineering department do not maintain competency because procedures are not followed after 2 or 3 years.  Are the still competent? Some would argue no because they are not perform the task and process correctly.

------------------------------
Richard Vincins RAC
Vice President Global Regulatory Affairs

Original Message:
Sent: 20-Jun-2019 01:01
From: Loganathan Kumarasamy, MS RAC
Subject: Employee Training

In addition to what Victor mentioned, you would have to careful with how you handle the quiz and marking for it. We have had one of our customer with quiz and 80% as pass percentage where FDA questioned it in one of the audits on how can you consider 80% as pass, as if it means 20% of errors can be made by them in Production. This made the organization to change their entire training process with 100% as pass percentage!

------------------------------
Loganathan Kumarasamy, MS RAC
Senior Consultant
Waukegan IL
United States

Original Message:
Sent: 19-Jun-2019 11:35
From: Anonymous Member
Subject: Employee Training

This message was posted by a user wishing to remain anonymous

How can one check the "effectiveness" of the training/ competency of an employee.
Is quizzing them enough?  Should the quiz records (Q&A) be maintained as a part of the training record? what if the questions are asked orally by the trainer?

Thank you.

Hi there,

This is a great question, and an area we have focused on recently. Effectiveness of training, from my perspective is asking two independent questions.

1. Did the trainee receive and understand the information provided in training?
2. Does the training provide the trainee the information they need to do their job appropriately?

Using quizzes or tests answers some of both. If no one can pass the test, the training material or the test is faulty. If most people can pass the test, those who fail may need additional training or remediation.

However, this does not answer the question of Does the training provide the trainee the information they need to do their job appropriately?

To answer this, we maintain cause codes related to training in many of our sub-systems (defects, CAPA, etc.). During our trend review, we look for repeating cause codes related to training. For example, if several defects are occurring the same step of production with cause codes relating to training, we can surmise that the training for that step is ineffective, and we will take a corrective action to improve the training. 

By taking measurements both at the conclusion of training (tests) and from the outputs of our process (trending) we can say confidently that we are measuring the effectiveness of the training system as a whole.

Along the same lines, we take a risk based approach to the types of training we provide. For roles that have a more direct impact to product quality (manufacturing, servicing, etc.) we tend to use a competency based model which utilizes several teaching methods, including tests, practical exams, hands-on training, etc.

Hopefully this is helpful.

Sincerely,
Jason

------------------------------
Jason Gorman
Director, Design Quality Systems
Madison WI
United States
------------------------------

Original Message:
Sent: 19-Jun-2019 11:35
From: Anonymous Member
Subject: Employee Training

This message was posted by a user wishing to remain anonymous

How can one check the "effectiveness" of the training/ competency of an employee.
Is quizzing them enough?  Should the quiz records (Q&A) be maintained as a part of the training record? what if the questions are asked orally by the trainer?

Thank you.