Notes:

This document is aimed to identify broadly applicable best practices for credible use of modeling and simulation in healtcare. The overall goal is to provide a summary of the discussions among the members of the Committee on Credible Practice of Modeling & Simulation in Healthcare.

The source file of this document is under version control in CoBi Core repository (in-house repository of Ahmet Erdemir, erdemira@ccf.org). The first version was copied from Revision 194 of https://simtk.org/svn/cpms/doc/articles/tsr-committee.doc

The document is targeted for submission to PLOS Computational Biology: Ten Simple Rules collection. It should be noted, however, this publication will be different from the final “Ten Simple Rules” article the Committee plans to publish in PLOS collection to lay the foundation for the guidelines the Committee has been charged to establish. Therefore, we may opt for another venue for this publication if potential conflicts arise with our final publication or with the objectives of the PLOS Ten Simple Rules Collection objectives (i.e. the Committee perspective actually resulted in 13 rules instead of ten).

The document, once published, is intended to be provided as open access, using Creative Commons Attribution 3.0 Unported License, see http://creativecommons.org/licenses/by/3.0/deed.en_US. Until then, any user of this work is encouraged to contact authors for copyright and licensing status.

Authorship criteria will be based on the considerations of International Committee of Medical Journal Editors (ICJME):

“The ICMJE recommends that authorship be based on the following 4 criteria:

Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND

Drafting the work or revising it critically for important intellectual content; AND

Final approval of the version to be published; AND

Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.”

For more details, see .http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html.

Those who want to claim authorship should briefly describe how they meet the conditions stated above. Use section on Author Contributions for this purpose.

This document is based on Committee activities summarized at:

http://wiki.simtk.org/cpms/Ten_Simple_Rules_of_Credible_Practice

http://wiki.simtk.org/cpms/CPMS_Task_Teams

http://wiki.simtk.org/cpms/Ten_Simple_Rules_of_Credible_Practice/Mathematical_and_Computational_Sciences_Team

http://wiki.simtk.org/cpms/Ten_Simple_Rules_of_Credible_Practice/End_Users_Perspective

http://wiki.simtk.org/cpms/Ten_Simple_Rules_of_Credible_Practice/Practice_Standards_and_Guidelines_Team

http://wiki.simtk.org/cpms/Ten_Simple_Rules_of_Credible_Practice/Summary_of_Results

Ten “Not So” Simple Rules for Credible Practice of Modeling and Simulation in Healthcare

Perspectives from a Multidisciplinary Committee

Perspective Article

Version 1.0

Ahmet Erdemir1, <AUTHORS>

1Department of Biomedical Engineering and Computational Biomodeling (CoBi) Core, Lerner Research Institute, Cleveland Clinic, Cleveland, OH 44195, USA

<AFFILIATIONS>

Proposed for submission to the PLOS Computational Biology: Ten Simple Rules Collection

September 9, 2016

Word Count: <#> (Abstract), <# > (Manuscript)

Corresponding Author:

<ADDRESS>

Abstract

<TEXT>

Keywords

<TEXT>

Table of Contents

Introduction	6

Methods	6

Overall Approach	6

Committee	7

Task Teams	7

Developers Team	8

Users Team	8

Regulators Team	9

Initial Broad Set of Practices	9

Task Team Perspectives: Approaches for Prioritization of Practices	10

Approach of Developers Team	11

Approach of Users Team	11

Approach of Regulators Team	12

Committee Perspective: Synthesis of Task Team Priorities	12

Results	14

Task Team Perspectives	14

Developers Team	14

Users Team	15

Regulators Team	18

Committee Perspective	19

Rule 1 – Define context clearly.	19

Rule 2 – Use appropriate data.	20

Rule 3 – Evaluate within context.	20

Rule 4 – List limitations explicitly.	21

Rule 5 – Use version control.	21

Rule 6 – Document adequately.	22

Rule 7 – Disseminate broadly.	22

Rule 8 – Get independent reviews.	23

Rule 9 – Test competing implementations.	23

Rule 10 – Conform to standards.	24

Discussion	24

Dissemination	26

Acknowledgments	26

Author Contributions	26

Conflict(s) of Interest	27

Appendices	27

Supplementary Material	27

References	27

Legends to Figures	28

Introduction

OUTLINE

Role of computational modeling and simulation (M&S) in healthcare

Necessity to establish credibility of M&S for safety and efficacy

Multidisciplinary nature of healthcare and biomedical research, multi-contextual use of M&S for development and delivery of healthcare; all raising cross-disciplinary issues for the use of M&S in research and clinical translation.

Broadly applicable practices may ease communication among various stakeholders (scientists, engineers, clinicians, regulators, policy makers; academia, industry and government), may allow a unified approach/workflow in establishing confidence in a specific M&S application; may increase the accuracy, efficiency, and safety of M&S when multidisciplinary groups are involved for its delivery.

Are there common rules among experts from varying disciplines, who use M&S in varying context, for establishing credibility in M&S in healthcare? The goal of this study is to synthesize discussions among various stakeholders from diverse disciplines and with differing contextual use of M&S (all assembled in the Committee on Credible Practice for Modeling & Simulation in Healthcare) to answer this question.

Methods

Overall Approach

The Committee on Credible Practice of Modeling and Simulation in Healthcare took upon the task of discussing the importance of various practices when establishing credibility of a given M&S work. The discussions were coordinated by the Co-Chairs of the Committee. The whole Committee included an Executive Committee and the Advisory Council. The Co-Chairs first assembled three task teams from the Committee roster. In following, they provided the task teams an initial set of broad practices to start discussions within the task teams. Task teams attempted to prioritize the initial set of rules with the specific instruction to identify the top ten. Task team leaders provided their feedback to Committee Co-Chairs, who later synthesized information from three task teams to develop an inclusive list of best practice rules that represents the whole Committee's perspective.

Committee

The Committee operated in two groups: an Executive Committee to carry out day to day operations and materialize the Committee's missions; and an Advisory Council to review and provide feedback on the activities. To provide a multifaceted interest and multidisciplinary nature relevant to the M&S applications in healthcare, membership roster attempted to include a balanced representation of different stakeholders. The Committee members were representatives from government, academia, industry, and healthcare organizations, coming from diverse disciplinary backgrounds with varying job functions, such as clinicians, engineers, scientists and technical managers. At the time of this study, Executive Committee had 9 members (including 2 Co-Chairs) and the Advisory Council consisted of 12 members. This study was part of the Committee's mission on the development of best practice guidelines.

Task Teams

For staged discussions on credible practices, the Committee was divided into three teams that would likely represent different levels of proximity to clinical applications of M&S in healthcare: those weighing towards mathematics and computation, those who have a vested interest in the end-use of models, and those inclining towards guidance, evaluation and regulation. In regard to scientific and/or clinical background, the groups were multidisciplinary. Each task team consisted of team coordinators from the Executive Committee and additional participants from the Advisory Council, while the Co-chairs provided general oversight to ensure cross-team interaction. The overall goal for these task teams were to represent a spectrum of interests of the key stakeholders and bridge synergistic activities in simulation-based medicine throughout the M&S communities.

Mathematics and Computation Team

This team was intended to represent the interests and perspectives of the stakeholders within the M&S community that comprise of the fundamental and applied researchers, as well as commercial developers of M&S related products for healthcare applications. Members of this group exhibited interest in development of models and simulation approaches with a proximity to mathematical and computational sciences. When established, this team included 7 participants – 3 from the Executive Committee and 4 from the Advisory Council; scientists and engineers with multidisciplinary backgrounds. Furthermore, the affiliation of the team members ranged from academia, industry, government agencies and independent.

Users Team

This team was intended to represent the interests and perspectives of the stakeholders within the M&S community that are focused on using M&S for clinical research and practice. When established, this team included 7 participants – 3 from the Executive Committee and 4 from the Advisory Council consisting of clinicians, scientists, and engineers from academia, industry, government agency and healthcare organizations with varying clinical focus and disciplinary backgrounds. The 7 members of this group had particular interests in end-use of M&S for clinical practice or research including design, development, and evaluation of interventions, as well as understanding various biological, physiologic and disease mechanisms.

Standards and Guidelines Team

Members of this group had an interest in the evaluation of M&S. This team was established with the intent to represent the interests and perspectives of the stakeholders within the M&S community that are mainly focused on establishing professional and/or industry practice guidelines, regulations and standards. When established, this team included 7 participants – 3 from the Executive Committee and 4 from the Advisory Council consisting of scientists, engineers, directors, and managers with different backgrounds. These 7 members were mainly affiliated with government agencies and industry.

Initial Broad Set of Practices

A non-prioritized list of good practice rules potentially applicable to the development and implementation of M&S, irrespective of the discipline and context of application, were prepared by the Co-Chairs of the Committee. These fundamental rules attempted to capture the essential activities associated with the conceptualization, development and implementation of M&S at various stages of its lifecycle. This includes before development of models, during development of models, while conducting simulations, for reporting models and simulation results, e.g., sharing knowledge, at dissemination of models and when applying the results for research purposes or for clinical decision-making. These rules were not necessarily definitive, they were simply provided to initiate discussions within task teams (see below).

The initial broad set of good practice rules, in no particular order, were:

Use version control.

Use credible solvers.

Explicitly list your limitations.

Define the context the model is intended to be used for.

Define your evaluation metrics in advance.

Use appropriate data (input, validation, verification).

Attempt validation within context.

Attempt verification within context.

Attempt uncertainty (error) estimation.

Perform appropriate level of sensitivity analysis within context of use.

Disseminate whenever possible (source code, test suite, data, etc).

Report appropriately.

Use consistent terminology or define your terminology.

Get it reviewed by independent users/developers/members.

Learn from discipline-independent examples.

Be a discipline-independent/specific example.

Follow discipline-specific guidelines.

Conform to discipline-specific standards.

Document your code.

Develop with the end-user in mind.

Provide user instructions whenever possible and applicable.

Practice what you preach.

Make sure your results are reproducible.

Provide examples of use.

Use traceable data that can be traced back to the origin

Use competition of multiple implementations to check and balance each other.

Task Team Perspectives: Approaches for Prioritization of Practices

The Co-Chairs provided the initial broad set of practices to each task team to start discussions. The task teams were instructed to prioritize these initial set of practices with the goal to identify the top ten for their group. In addition, the teams were asked to provide their description of how they perceive these generally applicable rules and why they prioritize in a certain way. The groups were given freedom to decide upon their own unique approaches to identify the top ten rules with or without ranking, and to interpret the rules in a way that was suitable to their background, and to combine rules or to add new ones. The unique process used by each team are described in the following sections. Task team coordinators organized and managed their respective team’s efforts to carry out discussions and summarize the task team specific results to the whole Committee.

Approach of Mathematics and Computation Team

This task team conducted a forum discussion, where each member was initially asked by the team coordinators to list their three most important rules. Nonetheless, some members provided more than three. Group ranking of best practices were conducted through a non-linearly weighted scoring: 1) Top three rules received a score of 12, 11, and 10, providing a higher preference on these. 2) When a member specified a fourth rule in their list without providing any additional rules afterward, that rule was scored as 5. 3) If more than one rule was specified following the first three, these rules were scored as 2 irrespective of their ranking in the individual's list. During the process, alternatives to the initial set or rules,based on the team's interpretation, were noted along with clustering of the rules and addition of new rules.

Approach of Users Team

This task team chose to cluster the initial set of practices in relation to each other. Team members avoided ranking, rather they indicated their own selection of the most important rules among the initial set and their grouping of the rules, including their justification. Group ranking relied on the frequency of the rules emerging as important, either by themselves or arranged within a rule set. When the number of a rule's appearance among responders was higher, so was its ranking. The team's approach did not consider resolving equal ranking of different rules in the group's list. The ranked group list was provided as a whole rather than focusing on top ten practices. There were also additional rules, predominantly for grouping of specific rules as a broad rule.

Approach of Standards and Guidelines Team

This task team used a combination of general discussions and a ranking scheme to identify the practices they felt were important to establish credibility of the M&S activity. Since not all of the members explicitly listed their ten important rules from the initial set, their feedback through discussions of the best practices were synthesized into ten general themes. These themes were then boiled down to ten rules relevant to initial list of broad practices. Group ranking was based on following principals: 1) The total number of people that explicitly stated the given rule or M&S practice was important. Higher weight was given to rules or practices that were ranked higher by the majority of the people and/or that were pervasive in the feedback from all contributors. 2) In case of a tie or close to a tie on the total number of people highlighting a particular rule or practice, the individual ranks given to the particular rule/practice by team members (if any) was taken into consideration. 3) The rules or practices that were indirectly viewed as being important through general consensus received lower weight. When necessary, practices were modified and/or consolidated into a broader rule, and new rules were devised.

Committee Perspective: Synthesis of Task Team Priorities

The Co-Chairs of the Committee curated prioritization and description of rules from all task teams to consolidate various desirable activities to establish credibility in M&S. In the process, commonalities of best practices among different task teams were first sought to establish broadly applicable ten simple rules. When a practice was noted to be important by all task teams, i.e., mentioned in the lists they have provided, the rule was introduced into the overall Committee list of ten best practices. In following, practices, which were mentioned to have importance in at least two task teams lists, were also included. For a practice that was mentioned only in one task team list, the rank of that practice within the individual task team list was considered (higher rank was better). Many of the practices that were mentioned only in a single task team were deemed as rules that may have less importance in a broad sense. Therefore they were not introduced in the Committee list, unless they became part of a more detailed consolidated practice. Consolidation accommodated feedback from task teams. The rules which may have similar outcomes in terms of their role in the assessment of credibility or those that may enhance each other were grouped into a single, more detailed, rule. The ranking of the Committee list favored commonality of a rule among individual task teams. Within the group of rules with the same commonalities, the ranking relied on a score calculated as the sum of inversely weighted individual task team ranks for the noted practice (score = 1/rank1 + 1/rank2 + 1/rank3). Nonetheless, the Co-Chairs acknowledged that this ranking was solely for reporting purposes as the priority of an individual practice within the holistic Committee list may change with the purpose of a given M&S activity; an issue that was emphasized by many task team members. Unique perspectives from teams and the individuals were also noted as they may indicate discipline specific or context specific needs to establish credibility of M&S. Individual components of the Committee's list of ten “not so” simple practices can be seen as a function of where they fit during the whole M&S process.

Results

Task Team Perspectives

Mathematics and Computation Team

Four task team members were responsive to draft the task team's perspective on the importance of various practices for credibility. The groups still remained diverse with scientists and engineers (independent or in academia) with multidisciplinary backgrounds. This team utilized a nonlinear scoring system based on task team member's individual ranking of existing or new rules that were introduced as alternatives to the initial set or consolidated or brand new. In following, the top ten practices in the order of decreasing importance level were identified as:

Use competition of multiple implementations to check and balance each other (score: 35) .

Document your code and make your code readable (score: 22) .

Explicitly identify experimental scenarios illustrating when, why, and how the model is false (alternative: Explicitly list your limitations) (score: 16) .

Make it easy for anyone to repeat and/or falsify your results (alternative: Make sure your results are reproducible) (score: 14) .

Use traceable data that can be traced back to the origin (score: 13) .

Use version control (score: 12) .

Define your evaluation metrics in advance (score: 12) .

Practice verification / validation / uncertainty quantification (alternative: attempt verification within context) (score: 11) .

Define the use context for which the model is intended (score: 10) .

Develop with the end user in mind (score: 5) .

Other practices mentioned in individual team members' priority list, which were later scored, include:

Report early and often, in the form of disseminated models and results (alternative: Report appropriately) (score: 4) .

Include credible/pedigreed implementations alongside other implementations (alternative: Use credible solvers) (score: 2) .

Attempt validation within context (score: 2) .

Disseminate whenever possible (source code, test suite, data, etc.) (alternative: Practice open science) (score: 2) .

Learn from discipline-independent examples (score: 2) .

Be very concrete and particular first, generalize second (alternative: Be a discipline-independent/specific example) (score: 2) .

Prespecify use cases and then choose standards and methods based on the context (alternative: Conform to discipline specific standards) (score: 2) .

Provide user instructions whenever possible and applicable (score: 2) .

Provide examples of use (score: 2).

Users Team

Four task team members were active in providing feedback on the relevance of various practices to establish credibility of a M&S activity. responsive to draft the task team's perspective on the importance of various practices for credibility. The group was able to sample insights from clinicians, scientists, and engineers from academia, and healthcare organizations with varying clinical focus and disciplinary backgrounds. Individuals of this task team provided their preferences by grouping initial set of practices based on their relatedness. The task team coordinators provided a list of rules listed based on the frequency of the rule's appearance in the individuals' responses (stand alone or within a group). Individual descriptions of the rules in the team's list mostly stayed faithful to those of the initial set of broad practices:

Define the context the model is intended to be used for (appeared 4 times; 4 stand alone, 0 in a group).

Disseminate whenever possible (source code, test suite, data, etc) (appeared 4 times; 4 stand alone, 0 in a group).

Use appropriate data (input, validation, verification) (appeared 4 times; 3 stand alone, 1 in a group).

Provide examples of use (appeared 4 times; 2 stand alone, 2 in a group).

Get it reviewed by independent users/developers/members (appeared 4 times; 2 stand alone, 2 in a group).

Use version control (appeared 3 times; 2 stand alone, 1 in a group).

Attempt uncertainty (error) estimation (appeared 3 times; 2 stand alone, 1 in a group).

Explicitly list your limitations (appeared 3 times; 1 stand alone, 2 in a group).

Perform appropriate level of sensitivity analysis within context of use (appeared 3 times; 1 stand alone, 2 in a group).

Attempt verification within context (appeared 3 times; 1 stand alone, 2 in a group).

Attempt validation within context (appeared 3 times; 1 stand alone, 2 in a group).

Develop with the end user in mind (appeared 3 times; 1 stand alone, 2 in a group).

Provide user instructions whenever possible and applicable (appeared 3 times; 1 stand alone, 2 in a group).

Use credible solvers (appeared 2 times; 0 stand alone, 2 in a group).

Use traceable data that can be traced back to the origin (appeared 2 times; 0 stand alone, 2 in a group).

Make sure your results are reproducible (DME*, AE*) (appeared 2 times; 0 stand alone, 2 in a group).

Document your code (appeared 2 times; 0 stand alone, 2 in a group).

Use consistent terminology or define your terminology (appeared 2 times; 0 stand alone, 2 in a group).

Define your evaluation metrics in advance (appeared 2 times; 1 stand alone, 1 in a group).

Conform to discipline-specific standards where possible (appeared 1 times; 0 stand alone, 1 in a group).

Use competition of multiple implementations to check and balance each other (appeared 1 times; 0 stand alone, 1 in a group).

Learn from discipline-independent examples (appeared 1 times; 0 stand alone, 1 in a group).

Be a discipline-independent/specific example (appeared 1 times; 0 stand alone, 1 in a group).

Follow discipline-specific guidelines (appeared 1 times; 0 stand alone, 1 in a group).

Conform to discipline-specific standards (appeared 1 times; 0 stand alone, 1 in a group).

Practice what you preach (appeared 1 times; 0 stand alone, 1 in a group).

Report appropriately (appeared 1 times; 0 stand alone, 1 in a group).

Standards and Guidelines Team

All task team members provided insight for discussions and construction of a task team list with ten practices deemed to be important for credibility. The outcome was a sampling of considerations mostly from participants of governing agencies with varying functions (scientists, engineers, directors, and managers). This team used the initial set of broad practices to identify ten general themes. These general themes seeded ten rules, which were related to the initial rule descriptions. In following, the members tried to prioritize their versions of the rules through discussions. The following list provides these practices in the order of decreased importance of perceived value by the task team members:

Plan and develop the modeling and simulation with the intended purpose/context, as well as the end-user in mind.

Use appropriate data (input, validation, verification).

Test the modeling and simulation appropriately within context (verification & validation, uncertainty quantification, sensitivity analysis, test cases).

Document important elements of the modeling and simulation (domain of validity/invalidity, intended use, users' guide, code documentation, etc.).

Explicitly list your limitations.

Have the modeling and simulation reviewed by independent users/developers/members.

Use version control.

Use appropriate discipline specific guidelines and standards.

Use consistent terminology or define terminologies.

Dissemination of the modeling and simulation.

Committee Perspective

The Co-Chairs of the Committee synthesized the feedback from the task teams and constructed a list of ten “not so” simple rules. Along the process, many of the items provided in the initial set of broad practices were consolidated. Some of the new rules proposed by the task teams, which were not necessarily consolidated versions, were incorporated into existing rules unless a task team emphasized the new rule by a higher ranking. Committee's list is provided below in sections. The order is provided mostly for reporting purposes.

Description of each rule includes the following:

Heading provides a representation of the abridged version of the rule.

Detailed Rule provides a description of the rule in an expanded fashion.

Included Practices list the seed and/or related practices from the initial set or as provided in different forms by the task teams.

Keywords list key vocabulary that may be used for categorization of the rule.

Lifecycle indicates where the rule may fit and will likely be utilized throughout the timeline of the M&S activity, i.e. where in the M&S workflow or at what stage in the M&S lifecycle. Relevant stages include but not limited to conceptualization, planning, development, utilization, evaluation, distribution, maintenance, modification, branching, reuse, etc.

Importance provides information to establish the significance of the rule.

Examples, when possible, cites and describes studies that demonstrate the appropriate application of the rule and/or consequences of not adhering to the practice.

Resources, when possible, summarizes tools and strategies for the implementation of the rule and directs the reader to guidelines and standards that may have specific relevance for the application of the rule in practice.

Rule 1 – Define context clearly.

Detailed Rule. Plan and develop the M&S activity with clear definition of the intended purpose or context accommodating end-users needs.

Included Practices. Define the context; define the use context for which the model is intended to be used for; plan and develop the M&S with the intended purpose/context, as well as end-user in mind; develop with the end user in mind; engage potential user base; provide examples of use.

Keywords. Context, purpose, intend, scope.

Lifecycle. Planning, development, utilization.

Importance.

Examples.

Resources.

Rule 2 – Use appropriate data.

Detailed Rule. Use data relevant to the M&S activity, which can ideally be traced back to the source.

Included Practices. Use appropriate data; use appropriate data (input, validation, verification); use traceable data that can be traced back to the origin.

Keywords. Data.

Lifecycle. Planning, development.

Importance.

Examples.

Resources.

Rule 3 – Evaluate within context.

Detailed Rule. Evaluate the M&S activity through verification & validation, uncertainty quantification, and sensitivity analysis faithful to the context/purpose/scope of the M&S efforts, with clear and a-priori definition of evaluation metrics and including test cases.

Included Practices. Define your evaluation metrics in advance; attempt verification; attempt validation; attempt uncertainty estimation; perform appropriate level of sensitivity analysis within context of use; practice verification, validation, uncertainty quantification (attempt verification within context); test the M&S activities appropriately within context (verification & validation, uncertainty quantification, sensitivity analysis, test cases) .

Keywords. Evaluation, repeatability, reproducibility.

Lifecycle. Utilization, evaluation.

Importance.

Examples.

Resources.

Rule 4 – List limitations explicitly.

Detailed Rule. Provide an explicit disclaimer on the limitations of the M&S to indicate under what conditions or applications the M&S may or may not be relied on.

Included Practices. Explicitly list your limitations; explicitly identify experimental scenarios illustrating when, why, and how the model is false (explicitly list your limitations).

Keywords. Disclaimer, scope.

Lifecycle. Utilization, evaluation, distribution.

Importance.

Examples.

Resources.

Rule 5 – Use version control.

Detailed Rule. Implement a version control system to trace the time history of the M&S activities, including delineation of contributors' efforts.

Included Practices. Use version control.

Keywords. Traceability, accountability.

Lifecycle. Development, utilization, distribution, maintenance, modification, branching.

Importance.

Examples.

Resources.

Rule 6 – Document adequately.

Detailed Rule. Document all M&S activities, including simulation code, model markup, scope and intended use of M&S activities, users' and developers' guides.

Included Practices. Document appropriately; document your code and make your code readable; document important elements of the M&S activity (domain of validity/invalidity, intended use, user guide, code documentation, etc.).

Keywords. Reporting, reproducibility.

Lifecycle. Utilization, evaluation, distribution.

Importance.

Examples.

Resources.

Rule 7 – Disseminate broadly.

Detailed Rule. Disseminate appropriate components of M&S activities, including simulation software, models, simulation scenarios and results.

Included Practices. Disseminate whenever possible; dissemination of the M&S; make it easy for anyone to repeat and/or falsify your results (make sure your results are reproducible).

Keywords. Dissemination, sharing, reusability, reproducibility.

Lifecycle. Distribution, modification, branching

Importance.

Examples.

Resources.

Rule 8 – Get independent reviews.

Detailed Rule. Have the M&S activity reviewed by independent third-party users and developers, essentially by any interested member of the community.

Included Practices. Get the M&S reviewed by independent users/developers/members; make it easy for anyone to repeat and/or falsify your results (make sure your results are reproducible).

Keywords. Review, evaluation, reproducibility.

Lifecycle. Utilization, evaluation, distribution.

Importance.

Examples.

Resources.

Rule 9 – Test competing implementations.

Detailed Rule. Use competition of multiple implementations to check the conclusions of different implementations of the M&S processes against each other.

Included Practices. Use competition of multiple implementations to check and balance each other.

Keywords. Evaluation, reproducibility.

Lifecycle. Development, utilization, evaluation.

Importance.

Examples.

Resources.

Rule 10 – Conform to standards.

Detailed Rule. Adopt and promote generally applicable and discipline specific operating procedures, guidelines, and standards accepted as best practices.

Included Practices. Adopt and promote standard operating procedures; use appropriate discipline specific guidelines and standards; use consistent terminology or defining terminologies.

Keywords. Standards, dissemination, sharing, reusability.

Lifecycle. Development, utilization, evaluation, distribution, modification, branching.

Importance.

Examples.

Resources.

Discussion

OUTLINE

Summary

Limitations (group specific issues, i.e., interpretation of common themes)

Limitations (broad issues)

Limitations (synthesis of information from initial set of practices to task-specific synthesis to Committee synthesis)

Role of context in which the M&S is used on the extend of establishing credibility (examples)

Implications of using these ten simple rules, will they help newcomers/experts to adapt M&S in clinics and clinical related research in an informed manner, safely, and effectively; in a multidisciplinary manner and when models can be used for varying purposes?

There are broadly applicable overlapping themes that allow a broadly applicable workflow. Nonetheless, potentially significant differences may arise in how intense these themes will be employed in a given discipline and context. Although the initial goal was to identify ten simple rules, the end outcome was first resulted in more than 10 rules that were partially ranked based on commonality across the team results and possible chronological sequence of the M&S process. At that point, further consolidation of the rules, based on similarities in their potential outcome (in regard to their role in the assessment of credibility) and relying on those enhancing each other, helped providing the ten “not so” simple rules. This process helped the Committee recognize the influence of disciplinary, organizational, and context of use (purpose) factors on the perception and relative importance of the rules for credible practice, which sometimes resulted in introduction of new rules by task teams. While we report this study with its breadth and its limitations, it became clear to the Committee that it was appropriate to reach out to the global community to identify the Ten Simple Rules of Credible Practice. Particularly given the fact that the guideline is to be developed for the global stakeholder community, it seems appropriate to gauge the values of the global community instead of relying on a small group of people to drive the ultimate direction.

Future work (public survey)

Dissemination

<TEXT>

Acknowledgments

Contributions by Ahmet Erdemir was supported in part by the National Institutes of Health under Grant R01EB009643 and Grant R01GM104139.

<TEXT>

Author Contributions

Authorship criteria will be based on the considerations of International Committee of Medical Journal Editors (ICJME):

“The ICMJE recommends that authorship be based on the following 4 criteria:

Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND

Drafting the work or revising it critically for important intellectual content; AND

Final approval of the version to be published; AND

Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.”

For more details, see .http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html.

Those who want to claim authorship should briefly describe how they meet the conditions stated above. Use section on Author Contributions for this purpose.

Ahmet Erdemir (as of September 9, 2016). 1) Conceived initial design of the study, coordinated a task team, and helped curate overall perspective. 2) Drafted article outline and contributed to methods and results sections. 3) Not applicable yet. 4) Agreed to be accountable.

Conflict(s) of Interest

Ahmet Erdemir owns and operates innodof, LLC, a consulting company for modeling and simulation.

Appendices

<TEXT>

Supplementary Material

<TEXT>

References

<TEXT>

Legends to Figures

Figure <#>. <TEXT>

<FIGURE>

Figure <#>

Table <#>.

<TABLE>