John Rice - about DoD:
~~~~~~~~~
They have ancient manuscripts from the founding fathers of computer simulation who knew much of what we are trying to figure out about the use of simulation.
...
they are thing to consider as you all try to bring a little structure and discipline to biomed related simulation before it get out of hand and hurts someone which will lead to REGULATION imposed from outside instead of discipline from within.
~~~~~~~~~
Tony Hunt - In Reply to John Rice:
~~~~~~~~~
All is not lost. A significant DOD M&S thought leader (at Rand Corp.) was (is) Paul Davis (and his co-workers). If you trace forward the branching citation tree from this paper you encounter a lot of good stuff. Some of it is only in Rand Corp. documents.
Davis, Paul K., and Richard Hillestad. "Families of models that cross levels of resolution: Issues for design, calibration and management." Proceedings of the 25th conference on Winter simulation. ACM, 1993. Paul grappled with many of our current issues 20 years ago.
Paul was always concerned with credibility issues.
Another, of course, is Bernard P. Zeigler. Although hard to read, his books, again, cover (talk about) most of our issues. BTW: his co-author on many works is doing a sabbatical with me this academic year).
~~~~~~~~~
Tony also attached this document to the message: BEST PRACTICES FOR THE DEVELOPMENT OF MODELS AND SIMULATIONS Final Report. http://www.msco.mil/documents/10-S-2_26 ... %20(3).pdf
Jacob Barhak in Reply to Tony Hunt:
~~~~~~~~~
From a brief look at the text it seems that our committee should look at Appendix B in BEST PRACTICES FOR THE DEVELOPMENT OF MODELS AND SIMULATIONS Final Report, June 2010. This appendix should be merged with our best practice rule candidates when we discuss the next version of our best practice rules.
~~~~~~~~~
John Rice in Reply to Tony Hunt:
~~~~~~~~~
So Tony, your figure from the paper which suggests that the variables effecting difficulty achieving complex, shareable, composible models are much greater in bio but I think the two bottom line arrows pointing in opposite directions may be misleading in that even if physics of ships, weapons, blast against materials and designs flight. radar, sonar, water, wind, etc are more observable and open to validation, integrating them given the state of knowledge of computer modeling and lack of computers, required creative genius, tenacity, knowledge of the calculus that the bio med community will need. But the "physics folks" did it and only now do we want to point to how easy it was. I think that the M&S community needs to come together and share all across the domains of use since surely sometime soon we will want to integrate their models which will be made harder if we don't start now to follow the same "top" rules.
~~~~~~~~~
Tony Hunt in response to Jacob - about Paul Davis:
~~~~~~~~~
Paul visited with my group in '99 & '00, as I was getting started down this new path. I paid close attention to his lessons. To me, it was obvious that the M&S issues he (et al.) identified transcended our very different domains. He correctly anticipated that we (my group) would face punishing "headwinds" within the reductionist-dominated biomedical M&S community.
~~~~~~~~~
Tony Hunt in Reply to John Rice:
~~~~~~~~~
A problem that Paul faced (& those in DoD, esp. the Navy Postgraduate School) and vastly underestimated was how users were quick to reify the models. Paul recognized the danger in that, but no plan to stop it.
My diagram is an effort to resist that urge. As soon as a user (or a stakeholder, or even the modeler) reifies a model, they enter a rabbit hole that damages--if not destroys--model credibility.
~~~~~~~~~
Lealem Mulugeta - about DoD and NASA:
~~~~~~~~~
I agree with the DoD’s best practices recommendations. It is not surprising that they came to the list in the appendix, because everything that they have in there is exactly what we do here at NASA. When NASA-STD-7009 was established in 2008, it was due to the lessons of the Columbia accident, which was believed may have been partially due to poor practices in how M&S were used for analyzing debris strike on the shuttle thermal protection system. So, the Columbia investigation recommended the establishment of M&S good practices/standard… the same good practice lessons listed in the appendix of the document Jacob mentions below. NASA basically took all of this and boiled it down to the NASA standard. If you read through the NASA standard 7009, as well as the handbook and guidance documents that should are coming out soon, you’ll see that what NASA’s work and DoD’s findings are, in many ways, mirror reflections of each other.
~~~~~~~~~
Lealem Mulugeta - about differences in language between disciplines:
~~~~~~~~~
As Tony kind of said it in so many ways, most researchers, biologists/physiologists and clinicians kind of look at you funny. What I found very interesting, however, is that their response is not because they disagree with our message. They come across as resistant because they simply don’t understand due to the simple fact that the language in most of these best practices documents tend to be different from what they are used to. Consequently, my colleagues and I have learned to restate these best practices to our end users (physiologists, clinicians, etc) in a language they understand. Fortunately, the Committee has already recognized the importance of using consistent and discipline specific terminologies. So I find it very encouraging that we are converging in our ideas to establish a “Credible Practice Guideline for M&S in Healthcare” that can be understood and useful to the greater clinical/medical research and practice community.
~~~~~~~~~
Jacob Barhak in reply to Lealem Mulugeta:
~~~~~~~~~
And Lealem, yes I do agree that language is the barrier. And the fact that you chose to add the word simple to the title of practices is important. If we simplify the language we define the rules we may expand our common base. If we use simple enough rules we may not need a glossary. The challenge is narrowing down the definitions so these will be accepted easily by multiple communities.
~~~~~~~~~
Ahmet Erdemir:
~~~~~~~~~
I think at the foundations of the funny looks in the MSM crowd is related to the diversity of the group. There are folks in bio* communities who would like to utilize M&S directly for field practice, similar to Defense and NASA folks like to do to support decision making. For this group, establishing good practice guidelines, standards (sometimes rather stringent) is a necessity simply due to potential consequences. On the other hand, there are also folks in bio* communities who would like to utilize M&S to support thought experiments; to generate hypothesis for example or to support their heuristic understanding of a system with potentially more objective processes of M&S. For this group, establishing (or conforming to) good practice guidelines, and standards is not too critical (and for many perceived as cumbersome). Yet in my opinion even if you do exploratory modeling, awareness of good practice can be very useful to prevent misinterpretations. I believe the NASA standard tried to be inclusive of these different cultures, similar to ASME V&V efforts in use of M&S in medical devices, call it "risk and impact" or "influence and consequence".
~~~~~~~~~
Jacob Barhak in Reply to Ahmet Erdemir:
~~~~~~~~~
You have a great point. This is a good observation - I would phrase it as bio modelers tend to want "exploration freedom" while engineers tend to be more result oriented. Although you can find contradictory examples this observation fits my experience.
~~~~~~~~~
John Rice in Reply to Ahmet Erdemir:
~~~~~~~~~
NASA did a marvelous job dealing with simulation issues following their high vis adverse events. But then NASA had been VERY PUBLIC with their simulation work and probably deserves 90% of the credit for educating and even exciting the PUBLIC about simulation. The saving of APOLLO demonstrated the role if simulation in many ways.
Aviation has followed actively showing value if simulation so much so that serious and want-a-be pilots have simulators at home.
So now what. And how does Bio play out. Or for us more important how could bio play in?! With defense, space, & aviation maybe bio-med (together) makes the critical mass or catalyst the M&S community needs to get an international cross domain professional movement centered on simple shared guidelines WHICH (back to where this started) recognize the research into M&S as a technology of science and inevitable tool of the needed new Scientific Method" (not unlike the microscope was) when the scientist have the exploration labs but where the PRODUCTS of both the exploration and the tool are disciplined by their community of tool makers producer and users.
~~~~~~~~~
Lealem Mulugeta in Reply to Ahmet Erdemir:
~~~~~~~~~
I think you are spot on regarding the diversity of the MSM community and how it can makes room for miscommunication, but it is not unique to the MSM community. In fact, I’ve found myself in a funny discussion with another lead bio-computational scientist here at NASA regarding the issue of validation methods. In the end, it turned out that there was no disagreement between us. What caused the perceived disagreement was due to the differences in our modeling and analysis approach, and the intended use of the two different models. But once we finally stepped back and looked at our particular applications, influencing disciplines etc., it made perfect sense and we were indeed saying the same thing. We just had to be willing to see things from each other’s particular areas of interests.
As NASA has worked to apply its standard to the bio computation realm, the diversity of disciplines and use cases (e.g. clinical decision, what-if scenario analyses to gain insight on unknown mechanism, hypothesis testing of potential mechanisms, etc) continues to grow. Therefore, it has become essential to guide this community on appropriate levels of verification, validation and credibility assessment practices. This is why we are developing the guidance document (with understandable terminology) as a "how to" for 7009. It’s tricky business, but much needed since most of what we work on at NASA has high consequence, as you pointed out.
Secondly, I am in partial agreement regarding your point about how conforming to practice guidelines for cases where models are being used for hypothesis testing or to support experiments. The condition I would agree with this point are:
- The data from the model are NOT major influencing factor in the conclusions to be drawn regarding the experiment or hypothesis being tested (i.e. you need additional data from literature and experiments to support conclusions)
- If the model is a major factor in the conclusions to be drawn from the experiment or hypothesis being tested, then the end outcome of these conclusions should NOT have a noticeable impact on healthcare related decisions or high value assets (e.g. determining the direction of a clinical experiment, or a decision to allocate millions of dollars and essential assets to a new experimental study)
~~~~~~~~~
Tony Hunt in Reply to Lealem Mulugeta:
~~~~~~~~~
Pharma faces the same problems, in some respects amplified by the diversity of potential products. For the small subset of Pharma with whom I have most recently worked, the need for new methods is becoming somewhat clear. We addressed those issues in this recent paper.
http://onlinelibrary.wiley.com/doi/10.1 ... .1222/full
We envision actors and tools within the framework making it increasingly easy for anyone to follow best practices.
~~~~~~~~~
Ahmet Erdemir in Reply to Tony Hunt:
~~~~~~~~~
You definitely pinpoint the need for "making it increasingly easy for anyone to follow best practices."
If good practice cannot be followed easily, it's acceptance becomes difficult and the whole effort may become niche. This is at least what I had in mind with the effort for "10 Simple Rules". In a sense, same rules can be equivalently applicable for different paths the modeler picks. The intensity of how each of these rules are employed by the modeler in practice may render an activity credible enough for "scientific exploration" or for "clinical practice" or for both.
~~~~~~~~~
John Rice regarding Rules and Ethics:
~~~~~~~~~
I don't think we would have difficulty finding 10 rules that if simply stated would garner quick concurrence by really experienced old modelers. Trick is to leverage the process of capturing and simply wording them so that it pulled divers domain simulations into a venue that fostered cross disciplinary practices. A but harder would be the oath to vet the language that each domain's representatives use to translate/interpret the rules back to their domain in ways that amplified rather than diluted their spirit and intent in the letter as read and applied in their domain and more importantly to their personal professional behavior. The physicians grabbed "do no harm". What would the Simulationists put on their graduation rings?
That foes bring up the question of Ethic and a code of ethics for Simulationists. The one created in and adopted by SCS, which is now being adapted slightly (as it should be with no loss and maybe domain specific strengthening of its spirit and intent) seem appropriate. Its short and could've considered for personal adoption or even adoption (and framed posting) in your own labs signed by you colleagues who wish to be seen as simulations above or within their other professional credentials.
http://www.scs.org/upload/Simulationist ... Ethics.pdf
There is also an independent council admittedly arising from roots in the big defense M&S industry, that maintains a Certification for Simulation Professionals. CMSP requires a pier adjudicated review of a portfolio of experience and an examination. First was for practitioners and is excessively engineering or at least math weighted. There is also a cert for Simulation Managers which was intended to offset the tendency of program managers or acquirers of modeling/simulation contract thing to make decisions without knowing anything about simulation models. Test question bank is impressive and answers VERY well documented for both best and not best answers.
By the way the healthcare simulation educators also have a certification by the SSH = Society for Simulation in Healthcare (training). () mine addition to the name but true to what/where they are. (Another story).
I've already engaged the CMSP about their openness to a top level common to any "Simulationist" exam to be paired with a domain specific endorsement exam which would allow for a community that included say: CMSP - Bio . They are willing to share the test infrastructure so other domains would not have to start from scratch in turn for there being a shared top level minimum shared body of knowledge that would distinguish and simulationist as a sim specialist within their other disciplines/domains of application.
http://www.simprofessional.org/
I know most of the key players.
So. Some extensible simple non-controversial rules, accepting an existing code of ethics (which encompasses the rules (I suspect the SCS one already does.) And a recognizable certifying body which already exists and would embrace enfolding the uniqueness of the domains of M&S use. And we sort of have a sharing or shared or shareable community!!!
~~~~~~~~~
Lealem Mulugeta Replies to all:
~~~~~~~~~
Agreed, with John. Coming up with 10 simple rules which we can agree upon and which resonate with the greater community is quite doable.
It sounds a bit lofty at the outset of hearing, “Ten Simple Rules”. But if you look across all the different standards, guidelines and best practices, they can all really be boiled down to about 10 major points which are considered most important to the field of interest.
Case in point: Some people who are highly involved with NASA’s standard questioned the endeavor of establishing “Ten Simple Rules”... that is until I point out the fact that NASA-STD-7009 actually consists of NINE simple rules that are most valued by the NASA community. Namely, Verification, Validation, Input Pedigree (model data quality), Results Uncertainty (UQ), Results Robustness (sensitivity), Technical Review (peer review), Use History (M&S heritage), M&S Management (documentation, workflows etc) and Peoples Qualification (expertise of the M&S developers, users and analysts). Then a bunch of text and criteria were wrapped around each simple rule to describe how the M&S needs to be developed and applied depending on the intended use and type of decision to be made.
If you look at all of the other ASME, DoD, SAE, IEEE and other standards, you’ll find very similar trends. What is important to note is that each of these standards and guidelines were developed with a specific field in mind.
I am working on summarizing the results of the survey we completed among the three task teams, and I’m finding very similar trends with a keen focus on the healthcare/bio field. This is an excellent sign!
Moreover, through the various discussions the committee has engaged in thus far, I foresee the implementation of a credibility rigor “scale” or “ladder” based on the intended application of the model. In this particular case, the various discussions are pointing to four key areas of M&S application which have different levels clinical impact. So I propose we use these four areas as a means to guide the level of rigor to which the ten rules should be followed. More specifically, the four levels can be broken down into cases where M&S are being used as:
- Clinical decision making tools (very high clinical impact)
- Clinical trial and/or therapeutics discovery tools (high clinical impact)
- Research tools for hypothesis testing to drive/augment a clinical research (medium clinical impact)
- Basic research tools used to interrogate/tease out new areas of research (low clinical impact)
- It helps the M&S developer, user and analyst understand how tightly they need to adhere to the “Credible Practice Guidelines”
- If there is a need to elevate the M&S to a higher level of application (e.g. from a Basic research tool to a Clinical trial tool), the developer will have a good picture of what more they need to do with their M&S to elevate it to this higher level.
~~~~~~~~~
Tony Hunt in Reply to Ahmet Erdemir
~~~~~~~~~
So, what is actually needed to get on a path where one of multiple objectives is to make it increasingly easy for anyone to follow best practices and undertake useful simulation experiments for what ever purpose?
Having a set of "10 Simple Rules" for guidance in new situations is a great start, but more is needed.
Opinion: scientifically useful simulation models must be designed for experimentation. Call it in silico or virtual experimentation. Local students prefer the latter: VE.
Today it is relatively easy for a motivated high school student to design and successfully execute a complicated wet-lab experiment without being encumbered by long, complicated best practice documents, without having deep knowledge of optics, analytical chemistry, or even mathematics. In time, the same will be true for VEs.
That wet-lab scenario is feasible because such wet-lab experiments are enabled by an rich, deep, multi-faceted, credible infrastructure that adheres to hundreds of best practices. The experiment is "synthesized" from components (selected from many options), some living, all designed for (and validated for) use in different, even unanticipated experiment designs.
Most (all?) current multiscale biomedical simulation models are basically "handmade." It's analogous to organic chemistry or physiology circa 1860. Much of the stuff of experiments in the best labs in Berlin, London, and Boston at that time was handmade.
~~~~~~~~~
John Rice - Draft idea about Rules of best practice - this was a separate thread, yet it is relevant to the discussion above and therefore quoted here:
~~~~~~~~~
In formulating a set of ‘rules’ for Creditable Practices in M&S (in Healthcare) using the analogy of Golden Rules, a(maybe few Silver ones), and ‘ten commandments’ consider for each one::
1. The universe of ‘stakeholders’ in M&S. Example:
a. Developers of any kind of models from physical to computational,
b. Users who do not (necessarily) create models and who may not be capable (by training/experience) of creating models but rather who use models as tools.
c. Consumers of the output of specific models. May include people who use a model-based product without specifically using (consciously manipulating variables) the models. Flight simulator instructor or student. This group may not even be aware of the model behind what they are doing. But they should be and therefore have some responsibility for the results they get. (Buyer be ware)
d. Managers of teams or organizations that develop use or consume output from models.
2. The universe of all domains of modeling use.
3. The Universe of kinds of models
a. Clay sculpture to computational
b. DE, Deterministic, Stochastic, Agent-based , …….
4. Internationality
The way in which any universal rule applies may differ in detail depending on the stakeholder role, domain of use, type of model, or even culture but if the rule is universal there can be a description of those details germane to stakeholder and application of any model. Example. Thou shalt document assumptions in plain language. Developers actually have to do this. But users then are responsible for reading the assumptions and justifying them in the context of their application. Consumers need to be aware that all models make assumptions and that if they consume a product of a model, they need to consider evidence from the model user who provided the product that the assumptions for the model were consistent with its use, that the user accounted for the those assumptions and that the consumer can be shown the audit trail before the consumer ingests the model output. The Managers need to understand that models make assumptions and they are an essential part of the model. If they manage developers, managers must resource to let developers document. If they manager users, the manager must understand that models have assumptions with can limit their utility and manage to ensure that the model users they manage understand the assumptions of the models being used and have matched them to the intended use of the models output. Managers who manage the consumers of the output of models must endures that the outputs they are accepting are appropriate for their intended use.
For each rule/commandment, there is a cube where there is an axis for the Stakeholder, the Domain of use, and the kind of models. Within the cells of the cube there is a discussion written for each of the stakeholders, by each of the Domains, but each kind of model. That content will very greatly in detail and scope. In some cases the cube is full of technical papers, in others it may only have a warning and suggest ways to mitigate risk.
~~~~~~~~~
Hopefully this thread will continue on this page.