Regulatory Open Forum

Dear colleagues, hello!👋

One of the most challenging exercise during risk management of a medical device is the evaluation of overall residual risk relative to the benefits of its intended use.

This is because there is no cookie-cutter formula and this evaluation generally involves a lot of (clinical) judgment.

When it comes to benefit-risk assessment of an AI/ML enabled device, this exercise is even more challenging because of so many unknowns associated with this rapidly evolving technology.

What if we could see some real world examples?

That is why, I took a shot at "reverse engineering" the Benefit-Risk assessment worksheet from the FDA in one of their guidance documents.

This device is intended to detect early signs of diabetic retinopathy using images collected in a routine exam. Pretty cool!

Here is my final output - I would love to hear your thoughts. (This is only intended for educational purposes and by no means suggests that it might be sufficient for FDA approval/clearance).

If you want to see my step-by-step approach, click on the image below:

Benefit-risk assessment for a vision screening device

Please share your comments and suggestions for improvement below.

Best regards

Naveen,

Agree the benefit vs risk trade-off is a complex terrain to cross-no doubt about it .

Also agree this gets even more complicated when AI/ML is factored in on its own!

 RAG technology  (Retrieval Augmented Generating) with browsers ( Microsoft Bing and Google ) helps some by publishing Internet links it discovers to augment the machine learning prompt replies from ChatGPT LLM  on Bing for example but still has a long way to go to render that sound clinical judgment on benefits vs risk for devices. At best what we get is a response on the benefit vs risk challenge that amounts to 'likely' also!

 We  wondered what apart from FDA Guidance PDFs ( that included your  assessment worksheet)  the US FDA publishes on this benefit vs risk trade off for medical devices.

See attachment on the results from our SmartSearch+ quest on --> (benefit* risk*)  NEAR 'device*'  search ---

Interesting discovery- it turns out there's a plethora of FDA 21 CFRs  out there though not necessarily on retinal diagnostics for diabetic retinopathy that deal with the benefit vs risk  from  devices  conundrum that are at least worthy for quick reviews to gain even more HUMAN clinical insights on actual FDA regulatory contents.

It's why we came up with a hybrid approach, i.e. part AI/ML AND  part full-text on actual FDA contents searching on not just  mere FDA Guidance PDFs but that  includes full-text searching on FDA's 21 CFRs as well. 'Hybrid' is what SmartSearch+ was engineered to achieve.

Put another way, a 'dueling banjo' approach between human and artificial intelligence back and forth searching still goes a long way.

Ram B

Dear colleagues, hello!👋

One of the most challenging exercise during risk management of a medical device is the evaluation of overall residual risk relative to the benefits of its intended use.

This is because there is no cookie-cutter formula and this evaluation generally involves a lot of (clinical) judgment.

When it comes to benefit-risk assessment of an AI/ML enabled device, this exercise is even more challenging because of so many unknowns associated with this rapidly evolving technology.

What if we could see some real world examples?

That is why, I took a shot at "reverse engineering" the Benefit-Risk assessment worksheet from the FDA in one of their guidance documents.

This device is intended to detect early signs of diabetic retinopathy using images collected in a routine exam. Pretty cool!

Here is my final output - I would love to hear your thoughts. (This is only intended for educational purposes and by no means suggests that it might be sufficient for FDA approval/clearance).

If you want to see my step-by-step approach, click on the image below:

Benefit-risk assessment for a vision screening device

Please share your comments and suggestions for improvement below.

Best regards

Hi Ram

Thanks for your comment - these are very interesting points.

Here is the reference to the FDA guidance document on this topic.

Factors to Consider When Making Benefit-Risk Determinations in Medical Device Premarket Approval and De Novo Classifications

You can find it here.

------------------------------
Naveen Agarwal, Ph.D.
Problem Solver | Knowledge Sharer.
Let's Talk Risk!
@https://naveenagarwalphd.substack.com/

Original Message:
Sent: 24-Jan-2024 05:28
From: Ram Balani
Subject: Benefit-risk assessment of an AI/ML enabled medical device

Naveen,

Agree the benefit vs risk trade-off is a complex terrain to cross-no doubt about it .

Also agree this gets even more complicated when AI/ML is factored in on its own!

 RAG technology  (Retrieval Augmented Generating) with browsers ( Microsoft Bing and Google ) helps some by publishing Internet links it discovers to augment the machine learning prompt replies from ChatGPT LLM  on Bing for example but still has a long way to go to render that sound clinical judgment on benefits vs risk for devices. At best what we get is a response on the benefit vs risk challenge that amounts to 'likely' also!

 We  wondered what apart from FDA Guidance PDFs ( that included your  assessment worksheet)  the US FDA publishes on this benefit vs risk trade off for medical devices.

See attachment on the results from our SmartSearch+ quest on --> (benefit* risk*)  NEAR 'device*'  search ---

Interesting discovery- it turns out there's a plethora of FDA 21 CFRs  out there though not necessarily on retinal diagnostics for diabetic retinopathy that deal with the benefit vs risk  from  devices  conundrum that are at least worthy for quick reviews to gain even more HUMAN clinical insights on actual FDA regulatory contents.

It's why we came up with a hybrid approach, i.e. part AI/ML AND  part full-text on actual FDA contents searching on not just  mere FDA Guidance PDFs but that  includes full-text searching on FDA's 21 CFRs as well. 'Hybrid' is what SmartSearch+ was engineered to achieve.

Put another way, a 'dueling banjo' approach between human and artificial intelligence back and forth searching still goes a long way.

Ram B

Dear colleagues, hello!👋

One of the most challenging exercise during risk management of a medical device is the evaluation of overall residual risk relative to the benefits of its intended use.

This is because there is no cookie-cutter formula and this evaluation generally involves a lot of (clinical) judgment.

When it comes to benefit-risk assessment of an AI/ML enabled device, this exercise is even more challenging because of so many unknowns associated with this rapidly evolving technology.

What if we could see some real world examples?

That is why, I took a shot at "reverse engineering" the Benefit-Risk assessment worksheet from the FDA in one of their guidance documents.

This device is intended to detect early signs of diabetic retinopathy using images collected in a routine exam. Pretty cool!

Here is my final output - I would love to hear your thoughts. (This is only intended for educational purposes and by no means suggests that it might be sufficient for FDA approval/clearance).

If you want to see my step-by-step approach, click on the image below:

Benefit-risk assessment for a vision screening device

Please share your comments and suggestions for improvement below.

Best regards