Regulatory Open Forum

This message was posted by a user wishing to remain anonymous

Someone in my company keeps asking us to create requirements and run unit testing on a statistical test (T-square test) for our algorithm, which is silly. The algorithm designer refuses to on the grounds that the T-square test is one of four tests that asks the question "how close is the test spectrum to the calibration spectrum? In other words, how well is the test spectrum described by the model developed with the calibration spectrum?

All four tests are used in combination to improve confidence in the final decision and to understand the source of the difference between the measured test spectrum and the model. According to him, none of these tests are requirements, they are; rather, design choices to meet the user accuracy requirement.

I try following the IEC 62304 standard and we do a good job when it comes to documenting the lifecycle of our software. However, the lifecycle of the algorithm is not well understood. Do we need requirements and unit tests for every function that the <g class="gr_ gr_1097 gr-alert gr_spell gr_inline_cards gr_run_anim ContextualSpelling" id="1097" data-gr-id="1097">algo</g> performs? Do we have to create requirements and unit testing around the T-square tests?  Please help!

First, step back from any regulatory requirements and ask a practical question. I'll infer that the t-test is the appropriate test in this application. The question is whether or not your software correctly implements the test. The appropriate method, in my opinion, is to write a specification for the test, write some positive and negative test case, and ensure the test performs correctly. The specification could be as simple as the maximum number of data points, the format of the data (traditional base 10 notation, scientific notation, etc.) and the interface. You could run either black box or white box testing. The test does not need to be complicated.

Remember that these tests can uncover problems in the software. They are not tests of the designer's statistical prowess.

On the regulatory side, you didn't state the type of product, I infer a device since you cite 62304. You didn't say where the software is located, so I infer it is in the device as opposed to, say, test software in the development lab. I think that failure to write the specification and test the software violates the requirements for software validation in 820.30(g).

------------------------------
Dan O'Leary
Swanzey NH
United States
------------------------------

Original Message:
Sent: 18-Jul-2018 03:08
From: Anonymous Member
Subject: Unit Testing/Requirements for Algorithm?

This message was posted by a user wishing to remain anonymous

Someone in my company keeps asking us to create requirements and run unit testing on a statistical test (T-square test) for our algorithm, which is silly. The algorithm designer refuses to on the grounds that the T-square test is one of four tests that asks the question "how close is the test spectrum to the calibration spectrum? In other words, how well is the test spectrum described by the model developed with the calibration spectrum?

All four tests are used in combination to improve confidence in the final decision and to understand the source of the difference between the measured test spectrum and the model. According to him, none of these tests are requirements, they are; rather, design choices to meet the user accuracy requirement.

I try following the IEC 62304 standard and we do a good job when it comes to documenting the lifecycle of our software. However, the lifecycle of the algorithm is not well understood. Do we need requirements and unit tests for every function that the <g class="gr_ gr_1097 gr-alert gr_spell gr_inline_cards gr_run_anim ContextualSpelling" id="1097" data-gr-id="1097">algo</g> performs? Do we have to create requirements and unit testing around the T-square tests?  Please help!

------------------------------

------------------------------

Original Message:
Sent: 19-Jul-2018 08:12
From: Dan O'Leary
Subject: Unit Testing/Requirements for Algorithm?

First, step back from any regulatory requirements and ask a practical question. I'll infer that the t-test is the appropriate test in this application. The question is whether or not your software correctly implements the test. The appropriate method, in my opinion, is to write a specification for the test, write some positive and negative test case, and ensure the test performs correctly. The specification could be as simple as the maximum number of data points, the format of the data (traditional base 10 notation, scientific notation, etc.) and the interface. You could run either black box or white box testing. The test does not need to be complicated.

Remember that these tests can uncover problems in the software. They are not tests of the designer's statistical prowess.

On the regulatory side, you didn't state the type of product, I infer a device since you cite 62304. You didn't say where the software is located, so I infer it is in the device as opposed to, say, test software in the development lab. I think that failure to write the specification and test the software violates the requirements for software validation in 820.30(g).

------------------------------
Dan O'Leary
Swanzey NH
United States

Original Message:
Sent: 18-Jul-2018 03:08
From: Anonymous Member
Subject: Unit Testing/Requirements for Algorithm?

This message was posted by a user wishing to remain anonymous

Someone in my company keeps asking us to create requirements and run unit testing on a statistical test (T-square test) for our algorithm, which is silly. The algorithm designer refuses to on the grounds that the T-square test is one of four tests that asks the question "how close is the test spectrum to the calibration spectrum? In other words, how well is the test spectrum described by the model developed with the calibration spectrum?

All four tests are used in combination to improve confidence in the final decision and to understand the source of the difference between the measured test spectrum and the model. According to him, none of these tests are requirements, they are; rather, design choices to meet the user accuracy requirement.

I try following the IEC 62304 standard and we do a good job when it comes to documenting the lifecycle of our software. However, the lifecycle of the algorithm is not well understood. Do we need requirements and unit tests for every function that the <g class="gr_ gr_62 gr-alert gr_spell gr_inline_cards gr_run_anim ContextualSpelling" id="62" data-gr-id="62">algo</g> performs? Do we have to create requirements and unit testing around the T-square tests?  Please help!

The short answer is yes. You need to test each statistical method in your software. The tests should involve correct calculation as well as error handling. Without these tests, you do not know that your software operates correctly.

The phrase "tests developed elsewhere" raises another issue. I could be that the tests you use are standard statistical tests for this application and you wrote the code to implement them. The other case is that you are using code developed by somebody else and incorporating it into your system. The second situation raises a few regulatory issues. The use of OTS software is covered by an FDA guidance document. The guidance documents on cybersecurity also include provisions to ensure the code is "clean". In terms of 62304, you have SOUP, so you need to be sure to account for it correctly.

------------------------------
Dan O'Leary
Swanzey NH
United States
------------------------------

Original Message:
Sent: 18-Jul-2018 03:08
From: Anonymous Member
Subject: Unit Testing/Requirements for Algorithm?

This message was posted by a user wishing to remain anonymous

Someone in my company keeps asking us to create requirements and run unit testing on a statistical test (T-square test) for our algorithm, which is silly. The algorithm designer refuses to on the grounds that the T-square test is one of four tests that asks the question "how close is the test spectrum to the calibration spectrum? In other words, how well is the test spectrum described by the model developed with the calibration spectrum?

All four tests are used in combination to improve confidence in the final decision and to understand the source of the difference between the measured test spectrum and the model. According to him, none of these tests are requirements, they are; rather, design choices to meet the user accuracy requirement.

I try following the IEC 62304 standard and we do a good job when it comes to documenting the lifecycle of our software. However, the lifecycle of the algorithm is not well understood. Do we need requirements and unit tests for every function that the <g class="gr_ gr_1097 gr-alert gr_spell gr_inline_cards gr_run_anim ContextualSpelling" id="1097" data-gr-id="1097">algo</g> performs? Do we have to create requirements and unit testing around the T-square tests?  Please help!