5 diagree 4 agree but didn't fill 1 missing but didn't fill 185 mostly completed survey ############################################### By region (most from North america) missing Asia Europe North America Oceania South America 3 8 39 129 5 1 ############################################# by job (most from academia) missing Academia Government Healthcare Facility Industry Other 4 99 25 11 19 4 Research Institution 23 ######################################## by field (most from engineering) Computer science Engineering Formal science Formal sciences Formal sciences Healthcare 8 90 1 1 4 13 Life sciences Life sciences Other Physical sciences Scientific computing Social sciences 4 22 10 21 7 4 ############################## by degree (most have Ph.D) missing Bachelor's degree 1 8 PhD MS 125 33 Medical and Doctorate degree or equivalent (e.g. MD/PhD) Medical degree or equivalent (e.g. MD) 14 4 ############################## by motivation (most support scientific research) Not using nor interested in M&S Other To support clinical decision making 1 14 27 To support clinical research To support medical training To support product development 15 5 19 To support regulatory approval To support scientific research 10 94 ########################### by experience missing Familiar without direct experience with M&S 1 15 Less than 5 years of experience in developing and/or using M&S More than 5 years of experience in developing and/or using M&S 28 140 Not familiar at all 1 ################################################################################################ Consistency is "good" but not excellent Qs_31 Qs_20 0 1 2 3 4 5 0 1 2 0 1 0 0 1 0 5 0 3 1 0 2 0 6 13 8 0 1 3 0 3 16 34 11 0 4 0 2 2 15 23 11 5 1 0 1 3 11 8 kappa for agreement is 0.587 (0.437, 0.693) -Follow discipline-specific guidelines and standards whenever possible. -Conform to discipline-specific standards. Qs_35 Qs_24 0 1 2 3 4 5 0 0 1 0 0 0 0 1 1 0 0 0 0 0 2 0 0 1 7 2 0 3 0 1 2 22 14 3 4 0 1 2 10 36 10 5 0 2 1 4 16 48 kappa for agreement is 0.567 (0.414, 0.697) -Explicitly list limitations of the M&S. -Provide clear descriptions of limitations. Qs_36 Qs_30 0 1 2 3 4 5 0 1 0 0 0 0 0 1 2 1 2 2 1 0 2 0 2 6 2 3 0 3 1 3 11 31 9 4 4 1 1 5 15 22 8 5 0 1 2 6 15 27 kappa for agreement is 0.552 (0.414, 0.658) -Use credible, e.g. verified, solvers (code, software, applications). -Use simulation software with established reliability. ############################################################# Answers to 35 questions (average score) mean 95%CI low 95%CI up Qs_4 4.332 4.201 4.451 Qs_2 4.231 4.104 4.330 Qs_13 4.115 3.962 4.231 Qs_24 4.011 3.853 4.147(rep 1) Qs_28 3.946 3.810 4.083 Qs_35 3.929 3.777 4.065(rep 1) Qs_29 3.874 3.721 4.016 Qs_5 3.709 3.533 3.874 Qs_27 3.705 3.536 3.847 Qs_26 3.694 3.536 3.836 Qs_30 3.670 3.492 3.816(rep 2) Qs_21 3.658 3.495 3.793 Qs_25 3.643 3.459 3.827 Qs_14 3.643 3.462 3.797 Qs_11 3.625 3.440 3.772 Qs_7 3.527 3.330 3.679 Qs_3 3.508 3.316 3.670 Qs_22 3.505 3.335 3.665 Qs_34 3.503 3.303 3.665 Qs_18 3.467 3.261 3.614 Qs_17 3.457 3.267 3.598 Qs_23 3.437 3.246 3.606 Qs_36 3.386 3.190 3.538(rep 2) Qs_9 3.378 3.195 3.551 Qs_12 3.261 3.054 3.448 Qs_32 3.250 3.038 3.440 Qs_20 3.246 3.077 3.404(rep 3) Qs_15 3.246 3.087 3.421 Qs_1 3.236 3.055 3.390 Qs_19 3.179 3.022 3.346 Qs_33 3.141 2.988 3.292 Qs_16 3.120 2.950 3.295 Qs_10 3.092 2.906 3.248 Qs_31 3.082 2.920 3.245(rep 3) Qs_6 2.929 2.761 3.092 Answers to 35 questions (proportions of selecting 5) mean 95%CI low 95%CI up Qs_4 0.543 0.462 0.609 Qs_2 0.440 0.363 0.505 Qs_13 0.401 0.324 0.467 Qs_24 0.386 0.313 0.451 Qs_29 0.333 0.257 0.399 Qs_35 0.332 0.255 0.397 Qs_28 0.310 0.234 0.375 Qs_25 0.308 0.232 0.368 Qs_5 0.286 0.220 0.357 Qs_30 0.281 0.216 0.341 Qs_11 0.272 0.203 0.326 Qs_26 0.268 0.197 0.322 Qs_14 0.258 0.198 0.313 Qs_34 0.249 0.189 0.308 Qs_27 0.246 0.186 0.306 Qs_32 0.245 0.185 0.310 Qs_3 0.243 0.184 0.308 Qs_18 0.234 0.174 0.288 Qs_21 0.228 0.158 0.283 Qs_36 0.212 0.152 0.266 Qs_9 0.205 0.142 0.259 Qs_22 0.203 0.148 0.264 Qs_17 0.196 0.130 0.250 Qs_23 0.191 0.131 0.240 Qs_7 0.179 0.125 0.234 Qs_15 0.169 0.115 0.230 Qs_12 0.168 0.120 0.228 Qs_1 0.148 0.099 0.198 Qs_10 0.141 0.092 0.189 Qs_20 0.137 0.093 0.191 Qs_16 0.136 0.087 0.185 Qs_31 0.114 0.065 0.158 Qs_19 0.112 0.061 0.156 Qs_6 0.098 0.060 0.141 Qs_33 0.086 0.049 0.124 The questions of the most importance are Qs_4, Qs_2, Qs_13 and Qs_24 The questions of the least importance are Qs_6, Qs_33, Qs_31 and Qs_16 -Make the M&S results reproducible. -Use appropriate data, e.g., for input, validation, verification. -Validate the M&S activity within the context of use. -Explicitly list limitations of the M&S. -Use competition of multiple M&S implementation methods to check and balance each other. -Make your code readable. -Conform to discipline-specific standards. -Adopt and promote standard operating procedures. ################################################################################# Subgroup analysis does not indicate that reposnes systematically vary accoriding to certain chacteristics of responders (see attached figures )