• Ten Simple Rules of Credible Practice
• End Users Perspective

This page is for providing feedback on Ten Simple Rules of Credible Practice from the perspective of End Users Task Team. The page complements the Forum discussion for those more inclined to utilize the wiki.

Please provide your insight on which of the candidate actions (provided in Ten Simple Rules of Credible Practice, add more if you find necessary) are important to establish credible practice. Ideally, you will rank these and provide justification on why one may have a precedence.

Summary of Initial Responses

-- aerdemir 2013-09-29 13:29:56

The common theme among responders to the 10 Simple Rules practice was the clustering of rules in relation to each other. For this reason many avoided ranking, rather indicated their own selection of the most important rules and their grouping of the rules. Nonetheless among the responders, there were multiple rules emerging as important and those are listed in here (based on total number of their appearance among responders)

• Define the context the model is intended to be used for (RNG, DME, GA, AE)
• Disseminate whenever possible (source code, test suite, data, etc) (RNG, DME, GA, AE)
• Use appropriate data (input, validation, verification) (RNG, DME, GA*, AE)
• Provide examples of use (RNG, DME*, GA, AE*)
• Get it reviewed by independent users/developers/members (RNG, DME*, GA, AE*)
• Use version control (RNG, DME*, AE)
• Attempt uncertainty (error) estimation (RNG, DME*, AE)
• Explicitly list your limitations (DME*, GA, AE*)
• Perform appropriate level of sensitivity analysis within context of use (RNG, DME*, AE*)
• Attempt verification within context (DME*, GA*, AE)
• Attempt validation within context (DME*, GA*, AE)
• Develop with the end user in mind (RNG, DME*, AE*)
• Provide user instructions whenever possible and applicable (RNG, DME*, AE*)
• Use credible solvers (DME*, AE*)
• Use traceable data that can be traced back to the origin (DME*, AE*)
• Make sure your results are reproducible (DME*, AE*)
• Document your code (DME*, AE*)
• Use consistent terminology or define your terminology (DME*, AE*)
• Define your evaluation metrics in advance (GA, AE*)
• Conform to discipline-specific standards where possible(DME*)
• Use competition of multiple implementations to check and balance each other (AE*)
• Learn from discipline-independent examples (AE*)
• Be a discipline-independent/specific example (AE*)
• Follow discipline-specific guidelines (AE*)
• Conform to discipline-specific standards (AE*)
• Practice what you preach (AE*)
• Report appropriately (AE*)

*Incorporated in a group of rules.

There were also additional rules, predominantly for grouping of specific rules as a broad rule:

• Choose an important problem to model (DME)
• Develop a rigorous, state-of-the-art, computationally sound model (DME)
• Define how the model will be assessed (verification, validation) (GA)
• Define expected output/metrics for assessment (GA)
• Provide basis/substantiating literature/data for assumptions of the model (GA)
• Link to associated work being done in the community, both modeling and in general biomedical sense (again, this is context but more explicit) (GA)
• Adopt and promote standard operating procedures (AE)
• Document appropriately (AE)
• Engage potential user base (AE)

For detailed individual perspectives in grouping of the rules, any additional rules, and justifications see Initial Individual Responses below.

Initial Individual Responses (In Chronological Order)

Ronald Germain - 09/23/2013 - RNG_10 rules.docx

I have attached my top 10. I believe many of the selected elements are closely linked and hence, simple ranking misses the point, so I have not given them numbers but grouped them according to relatedness.

David Eckmann - 09/25/2013 - DME_Top 5 rules.docx

I think that Ron's assessment points to the importance of the relatedness issues of these many aspects. His individual comments within his list stand tall. I see the list distilling down to only 5 truly key points with many credible and important subdivisions, which I have included without any particular ranking. Many of these I see clearly as good practice guidelines that should be followed without question (e.g., document your code!).

Gary An - 09/25/2013 - GA_10 Important things.docx

Here's my list. My emphasis is less on models as finished products but rather as rhetorical devices to augment the general scientific process, and thus focuses more on promoting transparency.

Ahmet Erdemir - 09/28/2013 - AE_CPMS-10_Simple_Rules.doc

Provided an initial set of rules, I attempted to consolidate various desirable actions in modeling and simulation (M&S) as a function of where they fit during the whole M&S activity. Three main categories of activities were identified during the process of M&S; first related to motivation, second to execution, and third to translation.