Hi all - my first entry into this forum, look forward to many interesting discussions.
Some new perspective on the summary document: it would be good to distinguish the deliverables of this group (guidance document, for example) from similar deliverables that are out there already or are under preparation by other groups with similar interests (ASME V&V40, for example).
First term goals seem reasonable. Three seem focused on open communication, which is an important but soft goal. The other two are more concrete - we'll deliver a document that people may or may not follow; as well as a model certification process (which also sounds like a document in the end). Generating these deliverables is one thing; ensuring they have traction in the larger community once delivered is another thing. Are there thoughts on how to address this latter point?
Review of summary document providing Committee overview
-
Jeff Bischoff
- Posts: 4
- Joined: Wed Nov 15, 2006 3:05 pm
-
Lealem Mulugeta
- Posts: 42
- Joined: Tue Dec 21, 2010 11:03 am
Re: Review of summary document providing Committee overview
Jeff,
RE: distinguish the deliverables from similar deliverables that are out there already or are under preparation by other groups with similar interests (ASME V&V40, for example).
Although there might be some overlaps between ASME and CPMS deliverable, the deliverable of this CPMS will be different from the ASME V&V40 or other similar processes in several key aspects. We were very careful to ensure that the CPMS's deliverables were needed and unique (why reinvent the wheel?). Some key aspects that make the CPMS deliverables unique to other efforts are:
1. The CPMS is focused on the good practices for translation of healthcare modeling and simulation research in general to clinical practice can be filled. Currently there are many researchers funded by the NIH and other organizations to develop multiscale models of various biological processes, drug interactions, disease states, organ function, and many more with the end goal of impacting the medical field by advancing the state of knowledge, and most importantly improving therapeutics and clinical interventions. However, it is clear that there is a large gap to be filled with regards translating these models from the researchers' labs to the clinic to inform medical practice. This is what the Committee will be committed to help solve.
2. The V&V40 task is focused on (at least currently) on setting guidelines/standard practices for V&V and credibility assessment of medical device models developed and applied by biomedical device manufacturers as part of their regulatory application submissions. In addition, the ASME guideline/standard is also more focused on industry application. The CPMS, on the other hand focused on bridging the gap between researchers and clinical practice, and we are not focused on a single medical application or area (e.g. medical devices). We are focused on M&S that have applicability to improving Healthcare, which also includes medical devices but still looking at translation between research and the clinic/industry.
3. Given that the CPMS is not a industry standards development organization like the ASME, the end products are not likely to be viewed at the same level as the V&V standards the ASME develops. The guidelines we will establish are intended to support researchers develop their M&S so that they can develop credibility within the research and clinical world to allow for smoother transition to the clinical setting. In addition, if and when the models are at a state to be "certified" by regulatory body, most of the necessary work with regards to to V&V, UQ, documentation, etc. will have been done already. So in a sense, the CPMS's products are complementary to V&V40, but not a duplication.
There are several people involved in the CPMS that also have a stake in V&V40. So between all of us, we should work to ensure that the two groups are complementary, and not a duplication of each other. In fact, the above points are major points I stressed the entire time I was working with the IMAG/NIH to establish the CPMS. So this is something we are acutely aware of, and I am counting on you and a few other who are working with both V&V40 and CPMS to ensure we keep the two deliverable separate and yet complementary to each other
RE: Ensuring they (credible practice guidelines) have traction in the larger community once delivered
Ah yes, this is key if the guidelines are to be meaningful to healthcare as we envision it.
As I see it, the way we are going to gain traction is by having the stakeholders on your side from the beginning. Namely the the agencies that fund the M&S researchers, and the M&S research community. Given that IMAG (9 agencies) and MSM (consortium of multiscale modeling researchers funded by IMAG agencies) instigated and strongly support our work, we have the main stakeholders on our side already. However, it is imperative that we continue to involve the IMAG and MSM so that they continue to support and endorse our deliverable. In an ideal scenario also, we would have the IMAG agencies recommend/request that proposers use credible practices in M&S such as the one established by the CPMS. This may sound a bit outrageous to some, but independent of the IMAG, NASA has started requesting that proposers use NASA-STD-7009 or equivalent methods for M&S credibility assessment. I have also heard similar ideas being floated around among IMAG participants.
Phew, excuse the long winded message. But I hope that answers most of your questions. If three is anything I am not clear about, don't hesitate to ask.
RE: distinguish the deliverables from similar deliverables that are out there already or are under preparation by other groups with similar interests (ASME V&V40, for example).
Although there might be some overlaps between ASME and CPMS deliverable, the deliverable of this CPMS will be different from the ASME V&V40 or other similar processes in several key aspects. We were very careful to ensure that the CPMS's deliverables were needed and unique (why reinvent the wheel?). Some key aspects that make the CPMS deliverables unique to other efforts are:
1. The CPMS is focused on the good practices for translation of healthcare modeling and simulation research in general to clinical practice can be filled. Currently there are many researchers funded by the NIH and other organizations to develop multiscale models of various biological processes, drug interactions, disease states, organ function, and many more with the end goal of impacting the medical field by advancing the state of knowledge, and most importantly improving therapeutics and clinical interventions. However, it is clear that there is a large gap to be filled with regards translating these models from the researchers' labs to the clinic to inform medical practice. This is what the Committee will be committed to help solve.
2. The V&V40 task is focused on (at least currently) on setting guidelines/standard practices for V&V and credibility assessment of medical device models developed and applied by biomedical device manufacturers as part of their regulatory application submissions. In addition, the ASME guideline/standard is also more focused on industry application. The CPMS, on the other hand focused on bridging the gap between researchers and clinical practice, and we are not focused on a single medical application or area (e.g. medical devices). We are focused on M&S that have applicability to improving Healthcare, which also includes medical devices but still looking at translation between research and the clinic/industry.
3. Given that the CPMS is not a industry standards development organization like the ASME, the end products are not likely to be viewed at the same level as the V&V standards the ASME develops. The guidelines we will establish are intended to support researchers develop their M&S so that they can develop credibility within the research and clinical world to allow for smoother transition to the clinical setting. In addition, if and when the models are at a state to be "certified" by regulatory body, most of the necessary work with regards to to V&V, UQ, documentation, etc. will have been done already. So in a sense, the CPMS's products are complementary to V&V40, but not a duplication.
There are several people involved in the CPMS that also have a stake in V&V40. So between all of us, we should work to ensure that the two groups are complementary, and not a duplication of each other. In fact, the above points are major points I stressed the entire time I was working with the IMAG/NIH to establish the CPMS. So this is something we are acutely aware of, and I am counting on you and a few other who are working with both V&V40 and CPMS to ensure we keep the two deliverable separate and yet complementary to each other
RE: Ensuring they (credible practice guidelines) have traction in the larger community once delivered
Ah yes, this is key if the guidelines are to be meaningful to healthcare as we envision it.
As I see it, the way we are going to gain traction is by having the stakeholders on your side from the beginning. Namely the the agencies that fund the M&S researchers, and the M&S research community. Given that IMAG (9 agencies) and MSM (consortium of multiscale modeling researchers funded by IMAG agencies) instigated and strongly support our work, we have the main stakeholders on our side already. However, it is imperative that we continue to involve the IMAG and MSM so that they continue to support and endorse our deliverable. In an ideal scenario also, we would have the IMAG agencies recommend/request that proposers use credible practices in M&S such as the one established by the CPMS. This may sound a bit outrageous to some, but independent of the IMAG, NASA has started requesting that proposers use NASA-STD-7009 or equivalent methods for M&S credibility assessment. I have also heard similar ideas being floated around among IMAG participants.
Phew, excuse the long winded message. But I hope that answers most of your questions. If three is anything I am not clear about, don't hesitate to ask.
-
Jacob Barhak
- Posts: 64
- Joined: Wed Apr 17, 2013 4:14 pm
Changes to the summary document providing Committee overview
Hi Jerry,
At Ahmet's request in the meeting I went through discussions and created a slide by slide list of changes. Please go though those changes and make sure that I did not miss anything and that I am representing well the consensus reached.
Martin had other less specific comments that I could not translate to changes in the slides. you may want to communicate with him to see if there is anything he feels strongly about that is not in this list.
This is open so others can pitch in to avoid critical errors. This was open for quite a while so by now everyone should have had a chance to comment. So it is unlikely we get major changes. If you do choose to comment, please make sure your contribution is known to Jerry and does not confuse him since this list of changes is in his hands now.
I hope you find it in good shape
Jacob
####### List of Changes ########
Slide 7: Need, under Clinical Urgency:
- Change the first bullet
From:
* There is a pressing need to utilize computational modeling & simulation to support clinical research and decision making in healthcare.
To :
* Modeling and simulation offers the capabilities to potentially expedite and increase the efficiency of healthcare delivery by supporting clinical research and decision making
- Change the second bullet
From:
* There is a gap in mechanisms or processes for translating computational models to the clinical practice.
To:
* ]There is a gap in mechanisms or processes for translating computational research models to the clinical practice.
- Add a new third bullet:
* Current computing technology can now replace many human tasks and decisions. It is important that the ability of computers is neither exaggerated nor diminished. It is important to gage this transition of tasks from human to machine in a manner that will be most efficient while diminishing negative phenomena. Establishing the credibility level of models will help smooth this transition.
Slide 8: Need, Under the long list of Scattered activities:
Add the following bullet at the end of the slide:
* Sensitivity Analysis / Results Robustness
Slide 10: Charge
Add a new bullet:
* Identify and promote innovative game changing technologies establishing model credibility
Slide 11 Charge - Adopt a consistent terminology:
Add 4 bullets to the long list of vocabulary:
* Abstraction
* Assumption
* Intended Use
* Referent
Slide 12: Charge - Propose guidelines and procedures for credible practice:
Add the bullet
* Move towards models that directly tie claims to results
The consensus with Ahmet was: Proposing guidelines and procedures for credible practice is a step towards endorsing models that directly tie claims to results – please feel free to rephrase/omit
Slide 14: Charge - promote good practice:
Add a new bullet:
* Reward Self Criticism: Suggest methods and promote cultures and environments that allow admitting failure to speed up the development cycle
Add a new Slide after Slide 14 with the Title Charge:
Sub title in bold is:
Identify and promote innovative game changing technologies establishing model credibility
Under this title add the following bullets:
* Engage with modelers and accumulate technologies in a list
* Identify technologies that are successful in one modeling field and check if those are applicable in other modeling fields.
* Assess possible benefits of each technology from certain to highly speculative.
* Disseminate the list of technologies and findings with the modelers and modeling community.
*** Please feel free to modify these since these points have not been reviewed by others and if you feel it is too cumbersome, you can omit and simplify.
Slide 21: Participation:
Please add the following bullets to the multidisciplinary background list at the end of the slide:
* Scientists
* Physicists
* Physicians
* Pharmacists
At Ahmet's request in the meeting I went through discussions and created a slide by slide list of changes. Please go though those changes and make sure that I did not miss anything and that I am representing well the consensus reached.
Martin had other less specific comments that I could not translate to changes in the slides. you may want to communicate with him to see if there is anything he feels strongly about that is not in this list.
This is open so others can pitch in to avoid critical errors. This was open for quite a while so by now everyone should have had a chance to comment. So it is unlikely we get major changes. If you do choose to comment, please make sure your contribution is known to Jerry and does not confuse him since this list of changes is in his hands now.
I hope you find it in good shape
Jacob
####### List of Changes ########
Slide 7: Need, under Clinical Urgency:
- Change the first bullet
From:
* There is a pressing need to utilize computational modeling & simulation to support clinical research and decision making in healthcare.
To :
* Modeling and simulation offers the capabilities to potentially expedite and increase the efficiency of healthcare delivery by supporting clinical research and decision making
- Change the second bullet
From:
* There is a gap in mechanisms or processes for translating computational models to the clinical practice.
To:
* ]There is a gap in mechanisms or processes for translating computational research models to the clinical practice.
- Add a new third bullet:
* Current computing technology can now replace many human tasks and decisions. It is important that the ability of computers is neither exaggerated nor diminished. It is important to gage this transition of tasks from human to machine in a manner that will be most efficient while diminishing negative phenomena. Establishing the credibility level of models will help smooth this transition.
Slide 8: Need, Under the long list of Scattered activities:
Add the following bullet at the end of the slide:
* Sensitivity Analysis / Results Robustness
Slide 10: Charge
Add a new bullet:
* Identify and promote innovative game changing technologies establishing model credibility
Slide 11 Charge - Adopt a consistent terminology:
Add 4 bullets to the long list of vocabulary:
* Abstraction
* Assumption
* Intended Use
* Referent
Slide 12: Charge - Propose guidelines and procedures for credible practice:
Add the bullet
* Move towards models that directly tie claims to results
The consensus with Ahmet was: Proposing guidelines and procedures for credible practice is a step towards endorsing models that directly tie claims to results – please feel free to rephrase/omit
Slide 14: Charge - promote good practice:
Add a new bullet:
* Reward Self Criticism: Suggest methods and promote cultures and environments that allow admitting failure to speed up the development cycle
Add a new Slide after Slide 14 with the Title Charge:
Sub title in bold is:
Identify and promote innovative game changing technologies establishing model credibility
Under this title add the following bullets:
* Engage with modelers and accumulate technologies in a list
* Identify technologies that are successful in one modeling field and check if those are applicable in other modeling fields.
* Assess possible benefits of each technology from certain to highly speculative.
* Disseminate the list of technologies and findings with the modelers and modeling community.
*** Please feel free to modify these since these points have not been reviewed by others and if you feel it is too cumbersome, you can omit and simplify.
Slide 21: Participation:
Please add the following bullets to the multidisciplinary background list at the end of the slide:
* Scientists
* Physicists
* Physicians
* Pharmacists
-
Martin Steele
- Posts: 37
- Joined: Tue Apr 23, 2013 9:52 am
Re: Review of summary document providing Committee overview
So, how does one communicate openly about an M&S?jeffbischoff wrote:Open Communication
How does one communicate credibility of practice in M&S? (I've addressed this previously)
Should this Committee:
- Institute methods for more completely communicating the credibility of an M&S Practice?
- Develop a set of expectations for credible M&S Practice?
As for "Model Certification" - what does that get you?
Caution: One can have a "certified model" and still use it inappropriately or get unsatisfactory results.
This is certainly challenging. One possible method is to select some pilot M&S projects (good & bad examples) from various venues to illustrate the value of Credible M&S Practices and/or the M&S Certification Process, then make sure they are given broad exposure (meetings, conferences, workshops, etc.)jeffbischoff wrote:ensuring they [the products] have traction in the larger community
-
Martin Steele
- Posts: 37
- Joined: Tue Apr 23, 2013 9:52 am
Re: Changes to the summary document providing Committee over
Consider adding "improve the quality" to this change, as in:jbarhak wrote:
####### List of Changes ########
Slide 7: Need, under Clinical Urgency:
- Change the first bullet
From:
* There is a pressing need to utilize computational modeling & simulation to support clinical research and decision making in healthcare.
To :
* Modeling and simulation offers the capabilities to potentially expedite and increase the efficiency of healthcare delivery by supporting clinical research and decision making
Modeling and simulation offers the capabilities to potentially expedite, improve the quality, and increase the efficiency of healthcare delivery by supporting clinical research and decision making
-
Martin Steele
- Posts: 37
- Joined: Tue Apr 23, 2013 9:52 am
Re: Review of summary document providing Committee overview
Can we get this document to assist/inform our purpose?lealem wrote:ASME V&V40
It would be helpful to see a definition for this term.lealem wrote:multiscale models
-
Tina Morrison
- Posts: 6
- Joined: Mon May 07, 2007 4:35 pm
Re: Review of summary document providing Committee overview
Hello Everyone,
It might take me sometime to catch up with all the comments regarding the Committee overview. Over the next four weeks, I'll be busy prepping the FDA workshop on computational models and validation. And then I'll have some free time to be more involved. In the meantime, I added my comments to the PDF - sorry they're not posted nicely on the forum as many of you have done.
I'm looking forward to the work this group will produce.
It might take me sometime to catch up with all the comments regarding the Committee overview. Over the next four weeks, I'll be busy prepping the FDA workshop on computational models and validation. And then I'll have some free time to be more involved. In the meantime, I added my comments to the PDF - sorry they're not posted nicely on the forum as many of you have done.
I'm looking forward to the work this group will produce.
- Attachments
-
- committee_summary - TMM comments.pdf
- (284.86 KiB) Downloaded 119 times
-
Jacob Barhak
- Posts: 64
- Joined: Wed Apr 17, 2013 4:14 pm
Re: Review of summary document providing Committee overview
Hello Everybody,
You may have seen the list of changes to the committee summary document.
At the phone meeting I was asked to approve this list of changes with the entire committee.
Please consider the post I made on May 16 and the responses Martin made on May 17, and Tina on May 20. You will need to download Tina's PDF file to see her suggested changes in red and in inserted comments. She raised some questions as well which you may continue discussing, and had some comments, yet I am referring only to the actual changes in text that add new words/phrases to the slides:
Slide 7: add Engineering before mathematical
Slide 8: add especially before complicated
Slide 8: extend uncertainty estimation by adding and propagation
Slide 9: add complete after inclusive
Slide 11: add the phrase and provide definitions
Slide 11: add qualification to the list
Martin asked to add Improve quality to slide 7
Please reply to the forum and choose from:
Accept changes
Reject changes and explain why
I will start by accepting all changes suggested.
If you you do not respond by the next team meeting, the assumption will be that you choose to abstain.
I look forward to your responses.
You may have seen the list of changes to the committee summary document.
At the phone meeting I was asked to approve this list of changes with the entire committee.
Please consider the post I made on May 16 and the responses Martin made on May 17, and Tina on May 20. You will need to download Tina's PDF file to see her suggested changes in red and in inserted comments. She raised some questions as well which you may continue discussing, and had some comments, yet I am referring only to the actual changes in text that add new words/phrases to the slides:
Slide 7: add Engineering before mathematical
Slide 8: add especially before complicated
Slide 8: extend uncertainty estimation by adding and propagation
Slide 9: add complete after inclusive
Slide 11: add the phrase and provide definitions
Slide 11: add qualification to the list
Martin asked to add Improve quality to slide 7
Please reply to the forum and choose from:
Accept changes
Reject changes and explain why
I will start by accepting all changes suggested.
If you you do not respond by the next team meeting, the assumption will be that you choose to abstain.
I look forward to your responses.
-
Martin Steele
- Posts: 37
- Joined: Tue Apr 23, 2013 9:52 am
Re: Review of summary document providing Committee overview
Jacob, This comment assumes the new version of the Summary Document has not been completed.
Chart 4 – “Modeling” is an activity, not a “thing”
The first words in the definition “specifically computational modeling” are fine as setting the context.
The way the rest of the definition reads “virtual representation of system(s) of interest …” could easily be attributed to the term “model.” “Modeling,” on the other hand, is the activity of building, making, or showing the system representation. So, I recommend one of the following:
Chart 4 – “Modeling” is an activity, not a “thing”
The first words in the definition “specifically computational modeling” are fine as setting the context.
The way the rest of the definition reads “virtual representation of system(s) of interest …” could easily be attributed to the term “model.” “Modeling,” on the other hand, is the activity of building, making, or showing the system representation. So, I recommend one of the following:
- To minimize changes to the definition on Chart 4: “virtually representing systems of interest …”
- Slightly longer option: “the activity of building, making, or showing the virtual representation of system(s) of interest …”
-
Lealem Mulugeta
- Posts: 42
- Joined: Tue Dec 21, 2010 11:03 am
Re: Review of summary document providing Committee overview
Hi Martin,
I just noticed that I had forgotten to the following questions.
"Multiscale biomedical modeling uses mathematics and computation to represent and simulate a physiological system at more than one biological scale. Biological scales include atomic, molecular, molecular complexes, sub-cellular, cellular, multi-cell systems, tissue, organ, multi-organ systems, organism, population, and behavior. These multiscale biomedical models may also include dynamical processes which span multiple time and length scales."
Multiscale Modeling may be a term we might want to clearly define since it is likely to come up again and I've noticed some discussions of what it really means among other researchers. I will post it on the thread that was posted for suggested glossary of terms.
I just noticed that I had forgotten to the following questions.
It is not available for wider distribution yet. But I will make sure you get a copy as soon as it is.mjsteele wrote:mjsteele wrote:Can we get this document to assist/inform our purpose?
As far as I know the current working definition of the Interagency Modeling and Analysis Group is as follows:mjsteele wrote:mjsteele wrote:It would be helpful to see a definition for this term "multiscale model".
"Multiscale biomedical modeling uses mathematics and computation to represent and simulate a physiological system at more than one biological scale. Biological scales include atomic, molecular, molecular complexes, sub-cellular, cellular, multi-cell systems, tissue, organ, multi-organ systems, organism, population, and behavior. These multiscale biomedical models may also include dynamical processes which span multiple time and length scales."
Multiscale Modeling may be a term we might want to clearly define since it is likely to come up again and I've noticed some discussions of what it really means among other researchers. I will post it on the thread that was posted for suggested glossary of terms.