Ahmet Erdemir, erdemira@ccf.org created on December 12, 2014 updated on April 20, 2014 --- COMPLETE DATA - Complete data was downloaded from Survey results (195 responses) on April 20, 2015 (from Stanford University Qualtrics site) - Data were exported as - complete.csv (Comma Separated Values) - complete.xml (XML format) - other data export options are possible, e.g. SPSS (.sav) file - Export settings were - Questions: "All Questions" - Show Answers As: "Choice Text" (other option is "Coded Values") - Decimal Format Delimiter: "Period" (other option is "Comma") - Question Numbers: Yes (other option is "No") - Recode Values: unchecked (option is to "Recode Seen but Unanswered Questions as -99") - Randomization Reporting: checked (set to "Export Randomized Viewing Order Data") - This setting allows evaluating the order of Simple Rules as presented to the participant - There are two questions with the label Q13; please pay attention during analysis. - There are 35 simple rules not 36, Qs_8 does not exist; please pay attention during analysis. - It seems like last two simple rule Qs_35 and Qs_36 were not included in randomization; they are displayed as the last two simple rules to grade; please pay attention during analysis. - There are some rules that were purposefully repeated with different phrasing to understand consistency of the participant, e.g., - Qs_20 AND Qs_31 - Qs_24 AND Qs_35 - Qs_30 AND Qs_36 - etc. - Comment and text boxes were mostly empty. Nonetheless, the information is screened to ensure that no personal identifiers were provided by the participants in these boxes. If any, these were anonymized by replacing the keyword 'ANONYMIZED' with the potentially identifier text or with the full contents of the comment or text box. --- SAMPLE DATA - Sample data was extracted from Survey Results (154 responses) on December 12, 2014 (from Stanford University Qualtrics site) - Following export of all data, 20 random entries were selected to create the sample stat set. - Data were exported as - sample.csv (Comma Separated Values) - sample.xml (XML format) - other data export options are possible, e.g. SPSS (.sav) file - Export settings were - Questions: "All Questions" - Show Answers As: "Choice Text" (other option is "Coded Values") - Decimal Format Delimiter: "Period" (other option is "Comma") - Question Numbers: Yes (other option is "No") - Recode Values: unchecked (option is to "Recode Seen but Unanswered Questions as -99") - Randomization Reporting: checked (set to "Export Randomized Viewing Order Data") - This setting allows evaluating the order of Simple Rules as presented to the participant - There are two questions with the label Q13; please pay attention during analysis. - There are 35 simple rules not 36, Qs_8 does not exist; please pay attention during analysis. - It seems like last two simple rule Qs_35 and Qs_36 were not included in randomization; they are displayed as the last two simple rules to grade; please pay attention during analysis. - There are some rules that were purposefully repeated with different phrasing to understand consistency of the participant, e.g., - Qs_20 AND Qs_31 - Qs_24 AND Qs_35 - Qs_30 AND Qs_36 - etc. - Comment and text boxes were mostly empty. Nonetheless, the sample data provide some examples where participants utilized these. The information is screened to ensure that no personal identifiers were provided by the participants in these boxes. --- KEY FOR QUESTION NUMBERS: -------------------------------- <Qs> How important are the following "simple rules" for credible / practice of modeling and simulation (M&S) in / healthcare? / Â We recognize the importance of / all the rules presented below. However, our goal... <Qs_1> -Engage potential end-user base. <Qs_2> -Make the M&S results reproducible. <Qs_3> -Develop the M&S with the end-user in mind. <Qs_4> -Use appropriate data, e.g., for input, validation, verification. <Qs_5> -Explicitly identify experimental scenarios that illustrate when, why, and how the M&S is false or not applicable. <Qs_6> -Use competition of multiple M&S implementation methods to check and balance each other. <Qs_7> -Document the development and use of M&S <Qs_9> -Use version control, i.e., to track different revisions of the model. <Qs_10> -Be a discipline specific example of good practice. <Qs_11> -Use data that can be traced back to the origin <Qs_12> -Disseminate whenever and whatever is possible, e.g., source code, test suite, data. <Qs_13> -Validate the M&S activity within the context of use. <Qs_14> -Perform sensitivity analysis within the context of use. <Qs_15> -Define the M&S evaluation metrics in advance. <Qs_16> -Make your code readable. <Qs_17> -Provide user instructions whenever possible and applicable. <Qs_18> -Provide examples of use. <Qs_19> -Learn from specialized and broadly applicable guidelines for good practice. <Qs_20> -Follow discipline-specific guidelines and standards whenever possible. <Qs_21> -Perform uncertainty estimation/quantification within context of use. <Qs_22> -Perform numerical error estimation/quantification within context of use. <Qs_23> -Get the M&S reviewed by independent users, developers, and members of the intended stakeholder community. <Qs_24> -Explicitly list limitations of the M&S. <Qs_25> -Make it easy for anyone to repeat and/or falsify your results. <Qs_26> -Use consistent terminology or define your terminology. <Qs_27> -Verify the M&S processes within context of use. <Qs_28> -Report appropriately, i.e., to allow reproducibility, to assess reliability, and to establish accountability. <Qs_29> -Define the context in which the M&S is intended to be used. <Qs_30> -Use credible, e.g. verified, solvers (code, software, applications). <Qs_31> -Conform to discipline-specific standards. <Qs_32> -Disclose conflict of interests. <Qs_33> -Adopt and promote standard operating procedures. <Qs_34> -Document your code. <Qs_35> -Provide clear descriptions of limitations. <Qs_36> -Use simulation software with established reliability. <Q11> What is your geographical location? <Q12> What is the primary setting you work in? <Q13> What is your primary field of academic/professional training? <Q14> What is your highest level of education? <Q15> How familiar are you with computational modeling and simulation / (M&S)? <Q13> What is the primary reason for your use/interest in leveraging / modeling and simulation (M&S) for healthcare research and / practice? <DO-Q-Qs> Display Order: How important are the following "simple rules" for credible / practice of modeling and simulation (M&S) in / healthcare? / Â We recognize the importance of / all the rules presented below. However, our goal...