Regulatory Open Forum

Hi All,

A part of my company's SaMD (Class II, moderate LOC) product is a service that is used to create an environment for launching of algorithms. The verification for this part is done via unit verification (code reviews, architecture reviews, walkthroughs, and unit testing), integration verification, and functional verification.

For this service, code development takes place in a separate feature development branch created by a developer and forked off of the main code branch which is called 'develop' branch in the repository. After the development branch code successfully goes through code reviews and testing, it is merged into the 'develop' branch. At the time of scheduled release, the  'develop' branch goes through another set of automated testing in Bitbucket Pipeline. After that the Release branch is created and promoted into the production environment.

All verification testing is done before the code is released into production environment, which is the part I have slight doubts about. Is that an accepted practice when it comes to the US FDA? Or will functional verification be required after code is released into production environment?

I am kind of okay with this approach, albeit with some apprehension, as there will be integration and system-level verification for the entire product carried out. However, I would appreciate any opinions and guidance here as this is the first SaMD product I'm working on.

Thanks!

------------------------------
Best,

Akshay Kulkarni
------------------------------

Hi Akshay,

just a quick question.
>> What do you consider as production environment for a SaMD? For me, after successful testing of a SaMD, the next process step is deployment (examples upload to an app store, activation of a web site).
The act of deployment should not impact the SaMD itself, hence I can only see a functional test that the SaMD (as a whole) has been successfully deployed.

------------------------------
Uwe Zeller | Regulatory Affairs / Risk Management Consultant
Biberach an der Riß, Germany
------------------------------

Original Message:
Sent: 01-Apr-2022 15:15
From: Akshay Kulkarni
Subject: Software Verification

Hi All,

A part of my company's SaMD (Class II, moderate LOC) product is a service that is used to create an environment for launching of algorithms. The verification for this part is done via unit verification (code reviews, architecture reviews, walkthroughs, and unit testing), integration verification, and functional verification.

For this service, code development takes place in a separate feature development branch created by a developer and forked off of the main code branch which is called 'develop' branch in the repository. After the development branch code successfully goes through code reviews and testing, it is merged into the 'develop' branch. At the time of scheduled release, the  'develop' branch goes through another set of automated testing in Bitbucket Pipeline. After that the Release branch is created and promoted into the production environment.

All verification testing is done before the code is released into production environment, which is the part I have slight doubts about. Is that an accepted practice when it comes to the US FDA? Or will functional verification be required after code is released into production environment?

I am kind of okay with this approach, albeit with some apprehension, as there will be integration and system-level verification for the entire product carried out. However, I would appreciate any opinions and guidance here as this is the first SaMD product I'm working on.

Thanks!

------------------------------
Best,

Akshay Kulkarni
------------------------------

In general, the FDA does not specify what, when and how of any testing or SDLC activities in general; Design Control and several guidance documents provide requirements and information about various activities, but each manufacturer decides how these requirements fit into their development process.

With that, all testing should be successfully completed before the software is released to a production environment. I assume that by production environment you mean it's available for customers to use, unless there's another step.
I would also think you have some sort of QA environment that is equivalent to your Prod where you can test all functionality before you push it out to Prod.

Many companies perform some post-deployment checks (sanity testing), which may include a subset of functional testing, to ensure everything propagated properly into production environment, but complete verification and validation must be performed before the release.

------------------------------
Michael Zagorski RAC
Director of Regulatory Affairs
Pittsburgh PA
------------------------------

Original Message:
Sent: 01-Apr-2022 15:15
From: Akshay Kulkarni
Subject: Software Verification

Hi All,

A part of my company's SaMD (Class II, moderate LOC) product is a service that is used to create an environment for launching of algorithms. The verification for this part is done via unit verification (code reviews, architecture reviews, walkthroughs, and unit testing), integration verification, and functional verification.

For this service, code development takes place in a separate feature development branch created by a developer and forked off of the main code branch which is called 'develop' branch in the repository. After the development branch code successfully goes through code reviews and testing, it is merged into the 'develop' branch. At the time of scheduled release, the  'develop' branch goes through another set of automated testing in Bitbucket Pipeline. After that the Release branch is created and promoted into the production environment.

All verification testing is done before the code is released into production environment, which is the part I have slight doubts about. Is that an accepted practice when it comes to the US FDA? Or will functional verification be required after code is released into production environment?

I am kind of okay with this approach, albeit with some apprehension, as there will be integration and system-level verification for the entire product carried out. However, I would appreciate any opinions and guidance here as this is the first SaMD product I'm working on.

Thanks!

------------------------------
Best,

Akshay Kulkarni
------------------------------

Hi Akshay,

Although the verification and validation you perform prior to release will be acceptable for design controls, the best practice is to perform smoke tests (or equivalent) once the code has been deployed to Prod. If you go down this path, the list of items you test as part of the smoke test and associated acceptance criteria should be defined upfront. 

It's also not a bad idea to include smoke tests as part of your Dev environment (and any other environments, like QA). This way there should be no surprises when the smoke test in run on Prod.

Hope this helps.

------------------------------
Richie Christian
Head of Quality and Regulatory
Auckland
New Zealand
------------------------------

Original Message:
Sent: 01-Apr-2022 15:15
From: Akshay Kulkarni
Subject: Software Verification

Hi All,

A part of my company's SaMD (Class II, moderate LOC) product is a service that is used to create an environment for launching of algorithms. The verification for this part is done via unit verification (code reviews, architecture reviews, walkthroughs, and unit testing), integration verification, and functional verification.

For this service, code development takes place in a separate feature development branch created by a developer and forked off of the main code branch which is called 'develop' branch in the repository. After the development branch code successfully goes through code reviews and testing, it is merged into the 'develop' branch. At the time of scheduled release, the  'develop' branch goes through another set of automated testing in Bitbucket Pipeline. After that the Release branch is created and promoted into the production environment.

All verification testing is done before the code is released into production environment, which is the part I have slight doubts about. Is that an accepted practice when it comes to the US FDA? Or will functional verification be required after code is released into production environment?

I am kind of okay with this approach, albeit with some apprehension, as there will be integration and system-level verification for the entire product carried out. However, I would appreciate any opinions and guidance here as this is the first SaMD product I'm working on.

Thanks!

------------------------------
Best,

Akshay Kulkarni
------------------------------

Hi All,

Thank you for replying! I should have been more clear - this part is not the entire product, but acts as a platform for a software which launches algorithms.
The production environment mentioned is just for this part, not that it is available for customers in any fashion.

Based off of the answers provided, I believe that this testing is acceptable in itself, and the final product should have robust validation testing and sanity/smoke tests as well. Is that an accurate capture?

------------------------------
Best,

Akshay Kulkarni
------------------------------

Original Message:
Sent: 01-Apr-2022 15:15
From: Akshay Kulkarni
Subject: Software Verification

Hi All,

A part of my company's SaMD (Class II, moderate LOC) product is a service that is used to create an environment for launching of algorithms. The verification for this part is done via unit verification (code reviews, architecture reviews, walkthroughs, and unit testing), integration verification, and functional verification.

For this service, code development takes place in a separate feature development branch created by a developer and forked off of the main code branch which is called 'develop' branch in the repository. After the development branch code successfully goes through code reviews and testing, it is merged into the 'develop' branch. At the time of scheduled release, the  'develop' branch goes through another set of automated testing in Bitbucket Pipeline. After that the Release branch is created and promoted into the production environment.

All verification testing is done before the code is released into production environment, which is the part I have slight doubts about. Is that an accepted practice when it comes to the US FDA? Or will functional verification be required after code is released into production environment?

I am kind of okay with this approach, albeit with some apprehension, as there will be integration and system-level verification for the entire product carried out. However, I would appreciate any opinions and guidance here as this is the first SaMD product I'm working on.

Thanks!

------------------------------
Best,

Akshay Kulkarni
------------------------------

Hi Akshay,

Not knowing what your SaMD does, it is quite difficult to say exactly what you should do. In general, it all depends on your risk assessment, as you have probably heard before. For example, does this component directly impact the patient's health and safety? From what you have said, you suggested that it would rather be indirect. In that case, the testing you do would be sufficient if you say that you also test the product as a whole at each release.


------------------------------
Hao Wang
Computer System Validation and Compliance Specialist
C-realize IT Services
www.c-realize.com
------------------------------

Original Message:
Sent: 04-Apr-2022 12:55
From: Akshay Kulkarni
Subject: Software Verification

Hi All,

Thank you for replying! I should have been more clear - this part is not the entire product, but acts as a platform for a software which launches algorithms.
The production environment mentioned is just for this part, not that it is available for customers in any fashion.

Based off of the answers provided, I believe that this testing is acceptable in itself, and the final product should have robust validation testing and sanity/smoke tests as well. Is that an accurate capture?

------------------------------
Best,

Akshay Kulkarni

Original Message:
Sent: 01-Apr-2022 15:15
From: Akshay Kulkarni
Subject: Software Verification

Hi All,

A part of my company's SaMD (Class II, moderate LOC) product is a service that is used to create an environment for launching of algorithms. The verification for this part is done via unit verification (code reviews, architecture reviews, walkthroughs, and unit testing), integration verification, and functional verification.

For this service, code development takes place in a separate feature development branch created by a developer and forked off of the main code branch which is called 'develop' branch in the repository. After the development branch code successfully goes through code reviews and testing, it is merged into the 'develop' branch. At the time of scheduled release, the  'develop' branch goes through another set of automated testing in Bitbucket Pipeline. After that the Release branch is created and promoted into the production environment.

All verification testing is done before the code is released into production environment, which is the part I have slight doubts about. Is that an accepted practice when it comes to the US FDA? Or will functional verification be required after code is released into production environment?

I am kind of okay with this approach, albeit with some apprehension, as there will be integration and system-level verification for the entire product carried out. However, I would appreciate any opinions and guidance here as this is the first SaMD product I'm working on.

Thanks!

------------------------------
Best,

Akshay Kulkarni
------------------------------

Thanks, Hao!

Your interpretation is correct, there is no direct impact to patient health and safety.

------------------------------
Best,

Akshay Kulkarni
------------------------------

Original Message:
Sent: 05-Apr-2022 04:52
From: Hao Wang
Subject: Software Verification

Hi Akshay,

Not knowing what your SaMD does, it is quite difficult to say exactly what you should do. In general, it all depends on your risk assessment, as you have probably heard before. For example, does this component directly impact the patient's health and safety? From what you have said, you suggested that it would rather be indirect. In that case, the testing you do would be sufficient if you say that you also test the product as a whole at each release.


------------------------------
Hao Wang
Computer System Validation and Compliance Specialist
C-realize IT Services
www.c-realize.com

Original Message:
Sent: 04-Apr-2022 12:55
From: Akshay Kulkarni
Subject: Software Verification

Hi All,

Thank you for replying! I should have been more clear - this part is not the entire product, but acts as a platform for a software which launches algorithms.
The production environment mentioned is just for this part, not that it is available for customers in any fashion.

Based off of the answers provided, I believe that this testing is acceptable in itself, and the final product should have robust validation testing and sanity/smoke tests as well. Is that an accurate capture?

------------------------------
Best,

Akshay Kulkarni

Original Message:
Sent: 01-Apr-2022 15:15
From: Akshay Kulkarni
Subject: Software Verification

Hi All,

A part of my company's SaMD (Class II, moderate LOC) product is a service that is used to create an environment for launching of algorithms. The verification for this part is done via unit verification (code reviews, architecture reviews, walkthroughs, and unit testing), integration verification, and functional verification.

For this service, code development takes place in a separate feature development branch created by a developer and forked off of the main code branch which is called 'develop' branch in the repository. After the development branch code successfully goes through code reviews and testing, it is merged into the 'develop' branch. At the time of scheduled release, the  'develop' branch goes through another set of automated testing in Bitbucket Pipeline. After that the Release branch is created and promoted into the production environment.

All verification testing is done before the code is released into production environment, which is the part I have slight doubts about. Is that an accepted practice when it comes to the US FDA? Or will functional verification be required after code is released into production environment?

I am kind of okay with this approach, albeit with some apprehension, as there will be integration and system-level verification for the entire product carried out. However, I would appreciate any opinions and guidance here as this is the first SaMD product I'm working on.

Thanks!

------------------------------
Best,

Akshay Kulkarni
------------------------------