Regulatory Open Forum

I work with many organisations who want to improve/streamline/automate their validation cycle. Their products range from medical devices (a lot of 'apps') to packages for the pharma industry. 

I notice that there is an 'automation spectrum': spanning from the purely manual validation process, to more and more automation involved in the release of new versions. Quite common today is the use of various platforms (Jama, Polarion, Jira) to engineer validation artefacts so that the traceability matrices can be generated automatically.

This week I had a first encounter with a company who boldly determined that their validation cycle will be 100% automated. This included the serious undertaking of providing 100% coverage by automatic tests.All the expected electronic records (validation plan, requirements specifications etc....)  will be produced in a format that is  accommodating to standards and to the inspectors expectations.

As regulatory standards and guidelines go- these are well behind:  GAMP 5 or the AAMI TIR45:2012 do not provide any reasonable help when it comes to a totally hands free validation cycle or the use of state-of-the-art devops tools.

My question therefor is:

1. what approach would inspectors take when auditing such a hands free validation cycle, in particular in regards to the test automation? 
2. what is the minimum set of control  points to design into such a cycle?

------------------------------
Rina Nir
Helping pharma and medtech make the most out of their Jira and Confluence,
Cambridge
United Kingdom
------------------------------

Hi Rina,
all sounds very interesting.  From a regulators perspective, as long as you are meeting the requirements, it is okay to use other tools.    These tools would also need to be validated for their intended use too.

with respect to your questions:
1) what approach would inspectors take when auditing such a hands free validation cycle, in particular in regards to the test automation? 
Review it like any other validation, making sure you are following procedures and that there is objective evidence that you are meeting your requirements.

2) what is the minimum set of control  points to design into such a cycle?
not sure exactly what are the control points your are referring to, but this sounds like a risk based question and something that you would want to predefine in either your procedures or your validation plan.

good luck with the project
seb


------------------------------
Sebastian Clerkin
Consultant
Cork
Ireland
------------------------------

Original Message:
Sent: 17-Mar-2018 03:19
From: Rina Nir
Subject: Handsfree validation cycle - regulatory position

I work with many organisations who want to improve/streamline/automate their validation cycle. Their products range from medical devices (a lot of 'apps') to packages for the pharma industry.

I notice that there is an 'automation spectrum': spanning from the purely manual validation process, to more and more automation involved in the release of new versions. Quite common today is the use of various platforms (Jama, Polarion, Jira) to engineer validation artefacts so that the traceability matrices can be generated automatically.

This week I had a first encounter with a company who boldly determined that their validation cycle will be 100% automated. This included the serious undertaking of providing 100% coverage by automatic tests.All the expected electronic records (validation plan, requirements specifications etc....)  will be produced in a format that is  accommodating to standards and to the inspectors expectations.

As regulatory standards and guidelines go- these are well behind:  GAMP 5 or the AAMI TIR45:2012 do not provide any reasonable help when it comes to a totally hands free validation cycle or the use of state-of-the-art devops tools.

My question therefor is:

1. what approach would inspectors take when auditing such a hands free validation cycle, in particular in regards to the test automation? 
2. what is the minimum set of control  points to design into such a cycle?

------------------------------
Rina Nir
Helping pharma and medtech make the most out of their Jira and Confluence,
Cambridge
United Kingdom
------------------------------

Hi Rina, I am not a regulator, but in my understanding validation has a strong correlation to responsibility, accountability... In case of "handsfree = brainfree" I do not believe in a success story. There is a need for "critical thinking" in validation, as mentioned in some of the standards and guidance for medical device validation. And this is difficult enough to be included completely in the validation plan.

Nevertheless, in the long term I can imagine deep learning solutions / artificial intelligence in validations at the regulator's side as well as at manufacturer's side, but that will be not next year (probably some years after driver-less cars).
But even today, it is a good starting point to automate some of the validation activities (especially the most boring activities like searching for traceability relations or for open tasks by hand / by eye / by brain, as you do it with RadBee) to enable the team some more time for "critical thinking".

Good luck with that project! Matthias.

------------------------------
Matthias Ueltzen
matthias.ueltzen@t-online.de
Sülzetal
Germany
------------------------------

Original Message:
Sent: 17-Mar-2018 03:19
From: Rina Nir
Subject: Handsfree validation cycle - regulatory position

I work with many organisations who want to improve/streamline/automate their validation cycle. Their products range from medical devices (a lot of 'apps') to packages for the pharma industry.

I notice that there is an 'automation spectrum': spanning from the purely manual validation process, to more and more automation involved in the release of new versions. Quite common today is the use of various platforms (Jama, Polarion, Jira) to engineer validation artefacts so that the traceability matrices can be generated automatically.

This week I had a first encounter with a company who boldly determined that their validation cycle will be 100% automated. This included the serious undertaking of providing 100% coverage by automatic tests.All the expected electronic records (validation plan, requirements specifications etc....)  will be produced in a format that is  accommodating to standards and to the inspectors expectations.

As regulatory standards and guidelines go- these are well behind:  GAMP 5 or the AAMI TIR45:2012 do not provide any reasonable help when it comes to a totally hands free validation cycle or the use of state-of-the-art devops tools.

My question therefor is:

1. what approach would inspectors take when auditing such a hands free validation cycle, in particular in regards to the test automation? 
2. what is the minimum set of control  points to design into such a cycle?

------------------------------
Rina Nir
Helping pharma and medtech make the most out of their Jira and Confluence,
Cambridge
United Kingdom
------------------------------

Matthias raises some good points about having a level of intelligent oversight.  If the validation protocols, scripts and test protocols have that the use of automated approaches can be very useful.  I recall 20+ years ago using automated scripts to test software and they found all sorts of issues.  The automation has a sense of "free" about it in that it can run over night and create a large body test result data.  But back to Matthias point, that data must be reviewed by individuals that understand the nature of the product and the test process to determine the validity of any conclusions.

------------------------------
Rick Muller RAC
Technical Director
Aurora CO
United States
------------------------------

Original Message:
Sent: 19-Mar-2018 17:55
From: Matthias Ueltzen
Subject: Handsfree validation cycle - regulatory position

Hi Rina, I am not a regulator, but in my understanding validation has a strong correlation to responsibility, accountability... In case of "handsfree = brainfree" I do not believe in a success story. There is a need for "critical thinking" in validation, as mentioned in some of the standards and guidance for medical device validation. And this is difficult enough to be included completely in the validation plan.

Nevertheless, in the long term I can imagine deep learning solutions / artificial intelligence in validations at the regulator's side as well as at manufacturer's side, but that will be not next year (probably some years after driver-less cars).
But even today, it is a good starting point to automate some of the validation activities (especially the most boring activities like searching for traceability relations or for open tasks by hand / by eye / by brain, as you do it with RadBee) to enable the team some more time for "critical thinking".

Good luck with that project! Matthias.

------------------------------
Matthias Ueltzen
matthias.ueltzen@t-online.de
Sülzetal
Germany

Original Message:
Sent: 17-Mar-2018 03:19
From: Rina Nir
Subject: Handsfree validation cycle - regulatory position

I work with many organisations who want to improve/streamline/automate their validation cycle. Their products range from medical devices (a lot of 'apps') to packages for the pharma industry.

I notice that there is an 'automation spectrum': spanning from the purely manual validation process, to more and more automation involved in the release of new versions. Quite common today is the use of various platforms (Jama, Polarion, Jira) to engineer validation artefacts so that the traceability matrices can be generated automatically.

This week I had a first encounter with a company who boldly determined that their validation cycle will be 100% automated. This included the serious undertaking of providing 100% coverage by automatic tests.All the expected electronic records (validation plan, requirements specifications etc....)  will be produced in a format that is  accommodating to standards and to the inspectors expectations.

As regulatory standards and guidelines go- these are well behind:  GAMP 5 or the AAMI TIR45:2012 do not provide any reasonable help when it comes to a totally hands free validation cycle or the use of state-of-the-art devops tools.

My question therefor is:

1. what approach would inspectors take when auditing such a hands free validation cycle, in particular in regards to the test automation? 
2. what is the minimum set of control  points to design into such a cycle?

------------------------------
Rina Nir
Helping pharma and medtech make the most out of their Jira and Confluence,
Cambridge
United Kingdom
------------------------------

Thanks @Rick Muller, @Matthias Ueltzen, @Sebastian Clerkin for your replies,

Currently the automated tests are designed mostly by human. I say mostly because we will also apply some core analyzers to ensure certain quality guidelines of the code are enforced.
In terms of the controls, which you have suggested should be in place​​​, we considered to have:
1. Design review for the automated tests (risk based approach- so probably not 100% review)
2. Periodic manual review of test results - during the development cycle
3. Human review (and sign-up) of the test results report before official release.

Does that sound sufficient? is that excessive or 'just enough'?

------------------------------
Rina Nir
Cambridge
United Kingdom
------------------------------

Original Message:
Sent: 20-Mar-2018 09:55
From: Rick Muller
Subject: Handsfree validation cycle - regulatory position

Matthias raises some good points about having a level of intelligent oversight.  If the validation protocols, scripts and test protocols have that the use of automated approaches can be very useful.  I recall 20+ years ago using automated scripts to test software and they found all sorts of issues.  The automation has a sense of "free" about it in that it can run over night and create a large body test result data.  But back to Matthias point, that data must be reviewed by individuals that understand the nature of the product and the test process to determine the validity of any conclusions.

------------------------------
Rick Muller RAC
Technical Director
Aurora CO
United States

Original Message:
Sent: 19-Mar-2018 17:55
From: Matthias Ueltzen
Subject: Handsfree validation cycle - regulatory position

Hi Rina, I am not a regulator, but in my understanding validation has a strong correlation to responsibility, accountability... In case of "handsfree = brainfree" I do not believe in a success story. There is a need for "critical thinking" in validation, as mentioned in some of the standards and guidance for medical device validation. And this is difficult enough to be included completely in the validation plan.

Nevertheless, in the long term I can imagine deep learning solutions / artificial intelligence in validations at the regulator's side as well as at manufacturer's side, but that will be not next year (probably some years after driver-less cars).
But even today, it is a good starting point to automate some of the validation activities (especially the most boring activities like searching for traceability relations or for open tasks by hand / by eye / by brain, as you do it with RadBee) to enable the team some more time for "critical thinking".

Good luck with that project! Matthias.

------------------------------
Matthias Ueltzen
matthias.ueltzen@t-online.de
Sülzetal
Germany

Original Message:
Sent: 17-Mar-2018 03:19
From: Rina Nir
Subject: Handsfree validation cycle - regulatory position

I work with many organisations who want to improve/streamline/automate their validation cycle. Their products range from medical devices (a lot of 'apps') to packages for the pharma industry.

I notice that there is an 'automation spectrum': spanning from the purely manual validation process, to more and more automation involved in the release of new versions. Quite common today is the use of various platforms (Jama, Polarion, Jira) to engineer validation artefacts so that the traceability matrices can be generated automatically.

This week I had a first encounter with a company who boldly determined that their validation cycle will be 100% automated. This included the serious undertaking of providing 100% coverage by automatic tests.All the expected electronic records (validation plan, requirements specifications etc....)  will be produced in a format that is  accommodating to standards and to the inspectors expectations.

As regulatory standards and guidelines go- these are well behind:  GAMP 5 or the AAMI TIR45:2012 do not provide any reasonable help when it comes to a totally hands free validation cycle or the use of state-of-the-art devops tools.

My question therefor is:

1. what approach would inspectors take when auditing such a hands free validation cycle, in particular in regards to the test automation? 
2. what is the minimum set of control  points to design into such a cycle?

------------------------------
Rina Nir
Helping pharma and medtech make the most out of their Jira and Confluence,
Cambridge
United Kingdom
------------------------------

Hi Rina,

In addition to what you have mentioned above, it is important that the automation tool or the system itself follows the SLC life cycle. As in, it should have been validated at first and a proper change control mechanism should be exercised. Ex., your protocols and results becomes an auditable record and if you are leveraging e-signature, then the system should comply with Part 11.  Hence, proper access controls should be put in place with procedures on system administration.

As you have indicated, code review  and test script coverage should be reviewed for adequacy, correctness and completeness.


------------------------------
Loganathan Kumarasamy
Senior Consultant
Zifo RnD Solutions
Waukegan IL
United States
------------------------------

Original Message:
Sent: 21-Mar-2018 03:56
From: Rina Nir
Subject: Handsfree validation cycle - regulatory position

Thanks @Rick Muller, @Matthias Ueltzen, @Sebastian Clerkin for your replies,

Currently the automated tests are designed mostly by human. I say mostly because we will also apply some core analyzers to ensure certain quality guidelines of the code are enforced.
In terms of the controls, which you have suggested should be in place​​​, we considered to have:
1. Design review for the automated tests (risk based approach- so probably not 100% review)
2. Periodic manual review of test results - during the development cycle
3. Human review (and sign-up) of the test results report before official release.

Does that sound sufficient? is that excessive or 'just enough'?

------------------------------
Rina Nir
Cambridge
United Kingdom

Original Message:
Sent: 20-Mar-2018 09:55
From: Rick Muller
Subject: Handsfree validation cycle - regulatory position

Matthias raises some good points about having a level of intelligent oversight.  If the validation protocols, scripts and test protocols have that the use of automated approaches can be very useful.  I recall 20+ years ago using automated scripts to test software and they found all sorts of issues.  The automation has a sense of "free" about it in that it can run over night and create a large body test result data.  But back to Matthias point, that data must be reviewed by individuals that understand the nature of the product and the test process to determine the validity of any conclusions.

------------------------------
Rick Muller RAC
Technical Director
Aurora CO
United States

Original Message:
Sent: 19-Mar-2018 17:55
From: Matthias Ueltzen
Subject: Handsfree validation cycle - regulatory position

Hi Rina, I am not a regulator, but in my understanding validation has a strong correlation to responsibility, accountability... In case of "handsfree = brainfree" I do not believe in a success story. There is a need for "critical thinking" in validation, as mentioned in some of the standards and guidance for medical device validation. And this is difficult enough to be included completely in the validation plan.

Nevertheless, in the long term I can imagine deep learning solutions / artificial intelligence in validations at the regulator's side as well as at manufacturer's side, but that will be not next year (probably some years after driver-less cars).
But even today, it is a good starting point to automate some of the validation activities (especially the most boring activities like searching for traceability relations or for open tasks by hand / by eye / by brain, as you do it with RadBee) to enable the team some more time for "critical thinking".

Good luck with that project! Matthias.

------------------------------
Matthias Ueltzen
matthias.ueltzen@t-online.de
Sülzetal
Germany

Original Message:
Sent: 17-Mar-2018 03:19
From: Rina Nir
Subject: Handsfree validation cycle - regulatory position

I work with many organisations who want to improve/streamline/automate their validation cycle. Their products range from medical devices (a lot of 'apps') to packages for the pharma industry.

I notice that there is an 'automation spectrum': spanning from the purely manual validation process, to more and more automation involved in the release of new versions. Quite common today is the use of various platforms (Jama, Polarion, Jira) to engineer validation artefacts so that the traceability matrices can be generated automatically.

This week I had a first encounter with a company who boldly determined that their validation cycle will be 100% automated. This included the serious undertaking of providing 100% coverage by automatic tests.All the expected electronic records (validation plan, requirements specifications etc....)  will be produced in a format that is  accommodating to standards and to the inspectors expectations.

As regulatory standards and guidelines go- these are well behind:  GAMP 5 or the AAMI TIR45:2012 do not provide any reasonable help when it comes to a totally hands free validation cycle or the use of state-of-the-art devops tools.

My question therefor is:

1. what approach would inspectors take when auditing such a hands free validation cycle, in particular in regards to the test automation? 
2. what is the minimum set of control  points to design into such a cycle?

------------------------------
Rina Nir
Helping pharma and medtech make the most out of their Jira and Confluence,
Cambridge
United Kingdom
------------------------------