The Chairs encouraged me to start this thread that demonstrates the need for the CPMS committee.
Instead of writing a single long text I decided to break this discussion thread into several smaller posts. Each one will show the need of our work.
I will try to be as objective as I can and reference as many facts as I can. Yet some posts will bring my interpretations to what I witness and perhaps will be less objective - please see those as my opinion.
You are welcome to join and post your contributions or comments to this thread. If you can bring objective evidence to back up your posts - great, otherwise state that this is your opinion and explain as best you can.
I hope these posts will help shape our work.
Need for the committee
-
Jacob Barhak
- Posts: 64
- Joined: Wed Apr 17, 2013 4:14 pm
-
Jacob Barhak
- Posts: 64
- Joined: Wed Apr 17, 2013 4:14 pm
The Need observed from the DDMoRe publication
With regards to the need and importance of this committee your attention is drawn towards the following paper:
White Paper: Landscape on Technical and Conceptual Requirements and Competence Framework in Drug/Disease Modeling and Simulation
http://www.nature.com/psp/journal/v2/n5 ... 1316a.html
In the paper there are several points of interest that demonstrate the importance and need for our contribution.
1. Size of move forward: Drug Disease Model Resources (DDMoRe) consortium is one of projects under the knowledge management pillar of Innovaive Medicines Initiative which was approved by the EU - this umbrella initiative is sized in the paper at 2 billion Euro.
2. Need for education: The paper clearly states that there is a need for education in this field - no consensus about knowledge or skills. Our committee has a broad multidisciplinary view and can help map educational directions.
3. Demand for guidelines: The paper states that "currently a code of best practices is lacking". I believe that the output of our committee will fit this missing element.
Even though this consortium is in a different continent, its findings in the paper are relevant to the committee.
You can find additional information on the consortium in the following web site:
http://www.ddmore.eu/
It will be nice to get in contact with this consortium to avoid effort duplication. I tried making contact a few weeks ago, yet still waiting for reply - if any committee member or advisory member can help connect it would be beneficial.
White Paper: Landscape on Technical and Conceptual Requirements and Competence Framework in Drug/Disease Modeling and Simulation
http://www.nature.com/psp/journal/v2/n5 ... 1316a.html
In the paper there are several points of interest that demonstrate the importance and need for our contribution.
1. Size of move forward: Drug Disease Model Resources (DDMoRe) consortium is one of projects under the knowledge management pillar of Innovaive Medicines Initiative which was approved by the EU - this umbrella initiative is sized in the paper at 2 billion Euro.
2. Need for education: The paper clearly states that there is a need for education in this field - no consensus about knowledge or skills. Our committee has a broad multidisciplinary view and can help map educational directions.
3. Demand for guidelines: The paper states that "currently a code of best practices is lacking". I believe that the output of our committee will fit this missing element.
Even though this consortium is in a different continent, its findings in the paper are relevant to the committee.
You can find additional information on the consortium in the following web site:
http://www.ddmore.eu/
It will be nice to get in contact with this consortium to avoid effort duplication. I tried making contact a few weeks ago, yet still waiting for reply - if any committee member or advisory member can help connect it would be beneficial.
-
Jacob Barhak
- Posts: 64
- Joined: Wed Apr 17, 2013 4:14 pm
The Government Thinks Modeling and Simulation is Important
One visible title in the MODSIM world conference was: U.S. Congress has declared M&S a National Critical Technology. See: http://www.modsimworldconference.com/
This title came from legislation that was produced by the modeling and simulation caucus. Here is a link to the web site:
http://forbes.house.gov/biography/mscaucus.htm
The leader of the caucus is congressman J. Randy Forbes from Virginia, yet there are representatives from many other states - you can find a list on the web site.
If the government declares Modeling and Simulation as a critical technology with such support then our work here is of value for sure.
This title came from legislation that was produced by the modeling and simulation caucus. Here is a link to the web site:
http://forbes.house.gov/biography/mscaucus.htm
The leader of the caucus is congressman J. Randy Forbes from Virginia, yet there are representatives from many other states - you can find a list on the web site.
If the government declares Modeling and Simulation as a critical technology with such support then our work here is of value for sure.
-
Jacob Barhak
- Posts: 64
- Joined: Wed Apr 17, 2013 4:14 pm
Re: Need for the committee
Another important reason for the need of credible practice guide is the alternative of not having one.
Without credible practice guidelines models will have adoptions issues. If there is no set standard, good models may be rejected because of novelty. Moreover early technology adopters who experienced the growing pain may become discouraged and avoid the use of models who can help them.
A good example for such a case where the computational model role is being diminished can be seen in the following link:
http://www.linkedin.com/groups/Technolo ... na_4158822
In this example, it seems users ran into solvable technological issues, yet dismissed modeling technology and are about to issue guidelines that diminish the use if computational models.
If there were checks and guidelines to fit models for their use and calm users, then the extra effort to adopt new technology would be more readily spent and as a consequence decision quality would be improved with the use of credible models.
Without credible practice guidelines models will have adoptions issues. If there is no set standard, good models may be rejected because of novelty. Moreover early technology adopters who experienced the growing pain may become discouraged and avoid the use of models who can help them.
A good example for such a case where the computational model role is being diminished can be seen in the following link:
http://www.linkedin.com/groups/Technolo ... na_4158822
In this example, it seems users ran into solvable technological issues, yet dismissed modeling technology and are about to issue guidelines that diminish the use if computational models.
If there were checks and guidelines to fit models for their use and calm users, then the extra effort to adopt new technology would be more readily spent and as a consequence decision quality would be improved with the use of credible models.
Re: Need for the committee
I would add that if one looks under the covers in DOD and NASA, they will find guidelines for M&S firmly rooted in the work of some of the first computer/mathematical modelers who had enough computing power (duel processors) to even try to execute code for a simulation based on a model.
There are two strong sets of evidence to consider when looking at them. First, NASA has just majorly reviewed and updated theirs. It took time and was expensive. It was driven in part by failures in which models played a part some of which were catastrophic at least programmatically. But they also had had so many successes that lead to decided that it was worth keeping them albeit in updated more contemporary modeling terms. I suspect that the core rules were still the same.
Second, If you look at DOD B$$$$ have been and is being spent for modeling and simaliton, there is a great deal of guidance in the files. Important to note that in the files does not mean it is always used. The current document on VV&A (in fact the need for Independent VV&A ) has evolved for over 40+ years. I suggest that the best evidence for the need for such guides is that in DOD, where the acquisition system is quite independent for the "governing" systems there have been great wastes of $ on failed modeling and simulation investment. Although the earliest very complex integrated models were for war gaming simulators or flight training systems whose creators knew that some discipline was essential. This was especially true if models were to ever be integrated into systems of models such as in the naval war game systems (NWGS and E (enhanced) NWGS) or a complete simulated aircraft. I will suggest that unlike NASA in the context above, little was ever seen of failures of DOD simulation but that many if not most failed in large part because the acquisition program managers chose, often quite explicitly, not to accept the cost or time it would have taken to following guidance. \
None the less, the discovery within the bio domain of M&S that lack of some shared discipline posed more risk to the future of the technology than was healthy is just a discovery of something that the for runners in M&S discovered or just expected from their earliest beginnings. Maybe and analog of their need to discipline their code to survive with other users in a MEG of core memory and 64K processor.
a few links among 5000 + hits on Google DMSO VV&A
VV&A Recommended Practices Guide - Modeling & Simulation ...
http://www.msco.mil/VVA_RPG.html
The Recommended Practices Guide (RPG) describes these VV&A processes from a number of perspectives. The RPG covers the different roles and ...
[PDF]
U.S. Coast Guard
http://www.uscg.mil/directives/ci/5000- ... 00_40.pdf
Dec 22, 2006 - (b) Defense Modeling and Simulation Office (DMSO) VV&A Recommended Practices Guide. Build 2.5,
http://vva.dmso.mil.
Revisiting NASA's Recommended Practices Guide for Verification ...
http://www.aegistg.com/.../Revisiting%2 ... 0VV&A%...
Recommended Practices Guide for Verification, Validation and Accreditation ... According to the DMSO VV&A RPG, “M&S credibility is measured by verification ...[PDF]
The Use of M&S VV&A as a Risk Mitigation Strategy in Defense ...
http://www.scs.org/pubs/jdms/vol2num4/K ... 9-216.pdf
Summary: Justification is that if lots of people who live with modeling and simulation did not think rules were essential there would not have be MILLIONS, Maybe BILLIONS of dollars spent trying to codify them. So if bio sim does not have them (yet) they will and at least at the top level they will likely be the same ones DOD and NASA already have.
There are two strong sets of evidence to consider when looking at them. First, NASA has just majorly reviewed and updated theirs. It took time and was expensive. It was driven in part by failures in which models played a part some of which were catastrophic at least programmatically. But they also had had so many successes that lead to decided that it was worth keeping them albeit in updated more contemporary modeling terms. I suspect that the core rules were still the same.
Second, If you look at DOD B$$$$ have been and is being spent for modeling and simaliton, there is a great deal of guidance in the files. Important to note that in the files does not mean it is always used. The current document on VV&A (in fact the need for Independent VV&A ) has evolved for over 40+ years. I suggest that the best evidence for the need for such guides is that in DOD, where the acquisition system is quite independent for the "governing" systems there have been great wastes of $ on failed modeling and simulation investment. Although the earliest very complex integrated models were for war gaming simulators or flight training systems whose creators knew that some discipline was essential. This was especially true if models were to ever be integrated into systems of models such as in the naval war game systems (NWGS and E (enhanced) NWGS) or a complete simulated aircraft. I will suggest that unlike NASA in the context above, little was ever seen of failures of DOD simulation but that many if not most failed in large part because the acquisition program managers chose, often quite explicitly, not to accept the cost or time it would have taken to following guidance. \
None the less, the discovery within the bio domain of M&S that lack of some shared discipline posed more risk to the future of the technology than was healthy is just a discovery of something that the for runners in M&S discovered or just expected from their earliest beginnings. Maybe and analog of their need to discipline their code to survive with other users in a MEG of core memory and 64K processor.
a few links among 5000 + hits on Google DMSO VV&A
VV&A Recommended Practices Guide - Modeling & Simulation ...
http://www.msco.mil/VVA_RPG.html
The Recommended Practices Guide (RPG) describes these VV&A processes from a number of perspectives. The RPG covers the different roles and ...
[PDF]
U.S. Coast Guard
http://www.uscg.mil/directives/ci/5000- ... 00_40.pdf
Dec 22, 2006 - (b) Defense Modeling and Simulation Office (DMSO) VV&A Recommended Practices Guide. Build 2.5,
http://vva.dmso.mil.
Revisiting NASA's Recommended Practices Guide for Verification ...
http://www.aegistg.com/.../Revisiting%2 ... 0VV&A%...
Recommended Practices Guide for Verification, Validation and Accreditation ... According to the DMSO VV&A RPG, “M&S credibility is measured by verification ...[PDF]
The Use of M&S VV&A as a Risk Mitigation Strategy in Defense ...
http://www.scs.org/pubs/jdms/vol2num4/K ... 9-216.pdf
Summary: Justification is that if lots of people who live with modeling and simulation did not think rules were essential there would not have be MILLIONS, Maybe BILLIONS of dollars spent trying to codify them. So if bio sim does not have them (yet) they will and at least at the top level they will likely be the same ones DOD and NASA already have.