Ten Simple Rules for Credible Practice of Modeling and Simulation (CPMS) in Healthcare

Ahmet Erdemir, erdemira@ccf.org, September 28, 2013

This document is a response to a request for identification and ranking of ten simple rules from an individual's perspective and as part of the CPMS Task Team – End Users.

For details on the initiative, see http://wiki.simtk.org/cpms/Ten_Simple_Rules_of_Credible_Practice.

For details on CPMS task teams, see http://wiki.simtk.org/cpms/CPMS_Task_Teams.

Provided an initial set of rules, I attempted to consolidate various desirable actions in modeling and simulation (M&S) as a function of where they fit during the whole M&S activity. Three main categories of activities were identified during the process of M&S; first related to motivation, second to execution, and third to translation.

Motivation

At the heart of a M&S activity lies the motivation to use a computational model for a desired purpose, research or clinical. This motivation dictates the goals of the specific M&S procedures that can establish the requirements of credibility; to balance potential influence of the model and to identify undesirable consequences of its abuse.

Simple Rule #1: Define the context. The context, of which the model is intended to be used for, indicates the potential impact of the model and the results of simulations carried out with it. It also implies the risks associated with the use of the model. Therefore, a clear definition of the context for the M&S activity, including primary and secondary variables of interest, is necessary to establish the required bounds for credibility. This definition will also inform potential reuse cases of the model.

Initial set of rules relevant to Simple Rule #1

Define the context the model is intended to be used for

Execution

Execution of a M&S activity entails various decision-making steps during the development of the model, while carrying out simulations, and when interpreting simulation results. Soundness on the steps carried out during execution increases confidence in reproducibility and establishes accountability, in a sense defining technical credibility.

Simple Rule #2: Use version control. By using a version control system, the evolution of the model and its predictive capabilities can be tracked. This is important to establish a history of credibility in the predictions of primary and secondary outputs of a given model, starting from its initial development, its maturity within a given context of use, and its future updates aiming at its reuse. Versioning also prevents any confusion that may emerge during undocumented use of the models.

Initial set of rules relevant to Simple Rule #1

Use version control

Simple Rule #3: Use appropriate data. This rule simply relies on the principle of “garbage in-garbage out”. M&S activities require data i) to develop a model, ii) to inform simulation conditions relevant to lifelike situations, and iii) to evaluate how well M&S reflects actuality. Any deficiencies in the founding data will reflect on the credibility of a given case of M&S. The quality of data has significance along with the traceability of the source of data for quality assurance. Completeness of data for its desired purpose, e.g., for input or for validation, has the potential to increase the credibility of the M&S activity.

Initial set of rules relevant to Simple Rule #1

Use appropriate data (input, validation, verification)

Use traceable data that can be traced back to the origin

Simple Rule #4: Attempt verification. To ensure correct solution of the model, attempts for verification are necessary. At the fundamental level, the verification of the simulation software, to solve types of models indicated by the context of the M&S activity, is necessary. This may already be available as part of test problems delivered with the simulation software. Nonetheless, additional attempts of verification, which are in the control of the developer and/or the user, are useful to establish reproducibility, therefore credibility. These may range from the evaluation of discretization errors to numerical tolerances associated with convergence. Further along the line of verification, objectivity can be established by cross verification of M&S procedures by independent groups or by utilizing different implementations.

Initial set of rules relevant to Simple Rule #1

Attempt verification within context

Use credible solvers

Make sure your results are reproducible

Get it reviewed by independent users/developers/members

Use competition of multiple implementations to check and balance each other

Simple Rule #5: Attempt validation. The credibility of a given model and the outcome of simulated conditions are closely related to one's confidence in the predictive capacity of the model. This predictive capacity can be established through validation studies that can evaluate primary and secondary metrics of interest extracted from simulation results against experimental data. The extent of experimentation and access to variables of interest certainly depend on technological capabilities. Nonetheless, good faith attempts to objectively establish validity of a model, particularly within its targeted use, are important for credible practice. One should note that the role of the underlying motivation for and the context of the M&S activities may dictate the depth of criticism. For example, M&S activities targeted to generate hypothesis for scientific studies may have more relaxed requirements of credibility when compared to those targeted to support clinical decision making.

Initial set of rules relevant to Simple Rule #1

Attempt validation within context

Simple Rule #6: Attempt uncertainty estimation. Estimation of uncertainty in M&S establishes credibility if predictions based on the model are within acceptable bounds for the intended use of the model. Attempts to quantify the influence of modeling assumptions, incomplete data, or variability in data which the model is founded on or driven by, add on the potential value of the M&S activity. These attempts may delineate important properties of the model and their relationship to primary and secondary outputs; all useful for prospective intentions for reuse.

Initial set of rules relevant to Simple Rule #1

Attempt uncertainty (error) estimation

Perform appropriate level of sensitivity analysis within context of use

Simple Rule #7: Adopt and promote standard operating procedures. Adoption of existing discipline-specific or cross-disciplinary M&S approaches can strengthen the accountability associated with the M&S activity. There are many guidelines, standards, and trade know-hows in M&S, which may broadly inform multiple disciplines. In a sense, adoption and promotion of standard operating procedures facilitates the completeness and quality of the M&S workflow.

Initial set of rules relevant to Simple Rule #1

Learn from discipline-independent examples

Be a discipline-independent/specific example

Follow discipline-specific guidelines

Conform to discipline-specific standards

Practice what you preach

Translation

The perception of a model's utility and the guidance (for research or for clinical decision-making) acquired through simulation results depend on adequate translation of the outcome of M&S activities to the community and to the field of practice. This can be in the form of knowledge dissemination and preparation and dissemination of models as tools, e.g., virtual medical devices. While a M&S activity may have good intentions and sound execution; delivery of its results, understanding of its potential reuse, and delineation of its boundaries to support science or clinics also depends on the activities to establish translational credibility.

Simple Rule #8: Disseminate whenever and whatever possible. Dissemination of data upon which the M&S activity is based on, software that is utilized for M&S, the model, and the simulation results gives the opportunity for anyone to reproduce a given M&S activity. This provides the freedom for potential reviewers and users of the models to judge credibility level in a more hands-on fashion. This is beyond knowledge dissemination, which is simply carried out through scholarly publishing.

Initial set of rules relevant to Simple Rule #1

Disseminate whenever possible (source code, test suite, data, etc)

Simple Rule #9: Document appropriately. Technical details of an M&S activity is commonly implemented in a formal language. Yet, the motivation of the activity, the implementation of the model, and the implications of predictions are commonly conveyed in a natural language. The human factor during the M&S description process, emerging in the form of “lost in translation”, may portray a well-intended and technically adequate M&S activity as not so credible. For utmost credibility, appropriate documentation is a necessity; from description of code to markup of the model to scholarly publications. Various guidelines and terminology (discipline specific or across discipline) exist for modelers to rely on. When these are not available, definitions should be provided.

Initial set of rules relevant to Simple Rule #1

Report appropriately

Document your code

Use consistent terminology or define your terminology

Define your evaluation metrics in advance

Explicitly list your limitations

Simple Rule #10: Engage potential user base. For clear portrayal of credibility and to promote reusability, engaging the user base is desirable. Clear examples of use cases of a model and delivery of instructions to use and to further develop the model provide the possibility to establish a community around the M&S activity. Early adoption of the M&S activity by a larger number of people may expedite isolating and addressing errors that may be missed by original developers.

Initial set of rules relevant to Simple Rule #1

Provide examples of use

Develop with the end user in mind

Provide user instructions whenever possible and applicable

Ahmet Erdemir

2013-09-28T10:09:27