Vote on Proposed Publication on 10 Simple Rules

[Jacob Barhak]

This thread contains recording of information passed through email among committee members.

This information contains emails I received in my email address and was sent to all participants. Some edits were made and can be found below the transcript.

I hope you find this information correct and intact.

Below is the transcript:


#############################################################################
TRANSCRIPT START
#############################################################################

Co-Chairs

Feb 26


Dear Executive and Advisory Council Members,

As we work toward the release of the public survey that will help us establish the Ten Simple Rules of Credible Practice based on the perspectives of the global community, Ahmet proposed the possibility of also publishing the results of the internal survey the Committee completed last October. The results of this internal survey are summarized here: http://wiki.simtk.org/cpms/Ten_Simple_R ... of_Results

After we discussed this on several occasions, it seems to make a lot of sense for a number of reasons.

1. The results elucidate the Committee’s overall perspective on the Simple Rules of Credible Practice
2. The results also demonstrates how interests and disciplinary backgrounds and context of use can drastically influence opinions/perspectives on the “most important” the principles of credible practice even within a focused committee. This in turn further emphasized the need to solicit input from the global community in order to establish the “Ten Simple Rules” and the “Guide on Credible Practice of Modeling and Simulation in Healthcare”, which are likely to be a better reflection of the values of the global stakeholder community.

Obviously these findings will be published after the global survey period has ended (i.e. after June 2014). This will avoid the possibility of inadvertently confound the responses we receive from the global community.

With that said, we did not want to publish the results without first getting the agreement of the committee. Please let us know if you are in support of this idea or not. If you are in support, please let us know if you would like to contribute as a co-author. We will make the final decision based on majority feedback by March 15th.

Please note that the ultimate set of simple rules and guideline will be derived primarily from the global survey results. We may also be able to use the global survey results to contrast the Committee’s perspective and the public survey results in an attempt to elucidate the benefits, deficiencies and any overlaps there may exist between committee derived vs publicly derived guidelines. This kind of analysis can be useful for other emerging fields with similarly multifaceted influencing factors by providing insight/guidance on how to establish an appropriate framework for developing practice standards and guidelines for the field of interest.

Thanks,


#############################################################################


Jacob Barhak

Feb 26



Hi,

This thread should go on the SimTk CPMS forum and be public.

And note that the small internal work we did is already public on the SimTk CPMS web site. So in a sense publication already took place.

Correct me if I am wrong, yet there was an abstract sent that also included reference to this work so this is already one re-publication. I don't mind reiterating once more, yet it seems that instead of spending efforts on republication we should concentrate efforts on the big survey - and there is a lot of work to be done there and it has been delayed too long already.

To be clear - if there is a conference you wish to speak about this previous work - great do it - no need to ask. Yet writing a new paper only depending on this work is a waste - spend the time on the larger survey instead.

We do not want to stand in place reiterating what we already did, we want to move forward.

I hope this helps resolve the issue.

Jacob

#############################################################################


Gary An

Feb 26


Hi Lealem and Ahmet,

Thanks for pushing this forward. I think a publication is a good idea, even in cases where the content may be released and potentially available. To me an analogy would be the release of white papers or a series of presentations in anticipation of a formal paper. I think this is particularly true in terms of "guideline" oriented papers, where the internal content may be readily available, but people still look to a specific reference-able "unit" (i.e. a paper) when they want to communicate, defend, influence or substantiate a course of action.

I'm sorry I've been a pretty absent member of this group, so I leave it up to you as to whether I meet "author criteria," but I'd like to be included if you think that is appropriate.

Regards,
Gary



#############################################################################


Tina M. Morrison

Feb 27



I’d very interested in participating in writing a paper.

My suggestion would be to use this as an opportunity to describe the complexity of M&S in the healthcare enterprise and motivate our effort by that complexity and diversity. By sharing the results from our internal survey, we motivate (and create anticipation) for participation in the global survey. Hopefully people will want to react to article by participating in the survey.

I don’t think we should just write a paper about our activities; the paper should provide motivation and support for the larger survey.

My 2 cents.

Cheers,
Tina


#############################################################################


Vasilis Marmarelis

Feb 27



I share Tina's view and fully support it.
Vasilis Z. Marmarelis




#############################################################################


Co-chairs
Mar 15

Dear all,

Thank you to everyone that has already responded to the request below. Given that several people have not yet responded, we will extend the due date to Thursday March 20th.

We would really appreciate everyone’s input on whether or not they see value in the proposed publication below. If you do feel there is value in the publication, it is not mandatory that you participate in the writing effort, but we welcome participation from all interested.

Regards,

Co-Chairs


#############################################################################


Wing Kam Liu

Mar 15




Dear All:
I do not see the value/impact of such publication, but it is up to majority to decide.
I can be completely wrong, but wish to ask a question: who are the readers and how many will read the paper?



#############################################################################


Bill Lytton

Mar 17



as you know, marc garbey and i have had some concerns about the '10
rules' from early on
we've put togehter a couple ideas on this - here in .pdf and inline

bill
================================================================================================
We continue to have concerns about the relevance of the "10 Simple Rules" and remain uncertain about the value of publishing these rules, or of publishing a follow-up paper.
In his email, Lealem nicely underlines the problem: "interests and disciplinary backgrounds and context of use can drastically influence opinions/perspectives on the most important the principles of credible practice even within a focused committee."
One overarching problem is that the committee has not retained its focus. Rather than considering just "Credible Practice of Modeling & Simulation in Healthcare," the committee is now also concerning itself with broader biomedical research practices and other matters.
Beyond this, the reason that there are few simple rules is that the practice of software simulation (as opposed to physical simulation) in healthcare is 1. enormously complex; 2. enormously diverse; 3. still in its formative stages. Admittedly there are some simple software rules that are very general -- eg 'use version control' -- some of these basics cannot be repeated too often.
Our suggestions for a different approach to the committee's task includes the following.
1. We should focus our efforts on healthcare as much as possible while acknowledging that most of the models that are available remain only in the research sphere.
2. We should not determine policies on model credibility based on generalizations from second-hand literature review. The committee has members who are experts on models and can contribute direct experience, thereby embracing the diversity of approaches rather than trying to encapsulate all into a single set of mandates. Our own credibility as a committee depends on offering success stories and good examples that we can analyze, and that we and others can emulate.
3. Rather than a review of the interests of the many stakeholders -- many of whom know little about modeling -- it will be more productive to make use of the expertise available on the committee itself. Instead of publishing a description about people's opinions about models, we would suggest the need for a publication that is about actual models. For example, we could propose a special edition of some relatively high-impact journal that would solicit authors who have developed credible models. Admittedly, these models will be at various levels of credibility and one request would be to have the authors comment on the distance remaining to clinical applicability.
4. A related effort is the model index that Herb Sauro has established on the IMAG MSM site. Author of these models fill out a form that could be augmented to indicate the value of the model
for clinical practice. We might then invite clinicians to examine some of these ideas and provide further feedback.
================================================================================================


#############################################################################




Joy P. Ku

Mar 17


Hi all,

I would agree with Wing that the value of such a publication is not clear to me. Personally, I would want to see more extensive data or compelling examples backing up any publication we produce.

Joy


#############################################################################


Martin J. Steele

Mar 18


This is in response to the statements of Bill Lytton & Marc Garbey.
I’m a bit surprised this discussion has resorted to e-mail. I thought this discourse was to be on the SimTK site.
I firmly believe there are relevant rules for the Credible Practice of M&S, regardless of the domain of applicability. In publishing them, we share our perspectives with the greater community of M&S practitioners and customers, who will likely consider them, comment on them, and help make them better. They can provide a documented guide for an M&S effort and an established baseline for evaluating an M&S more consistently and completely.
The wide variety of interests, backgrounds, & contexts can give more power to the result, but that does not mean it will be easy. Divining such a thing as the Credible Practice of M&S requires an open mind for all involved. So, complexity and diverseness is a given, but just being in the formative stages does not mean it’s not worthwhile.
I’ve also found that relying of software standards to guide M&S efforts to be inadequate. While they do have good attributes, they:
•Lack aspects of the initial analysis of the ‘system to be modeled’ that are important to an M&S effort
•Typically focus on software development
•Usually do not address operational (use) aspects of M&S
•Do not address aspects of reporting an M&S-based analysis
Knowing the stakeholder’s perspective is valuable, even if they don’t know much about M&S and may not end up being used.
You mention authors who have developed credible models. How was that determined?
Is the distance to clinical applicability relevant to credible practice?
It is important to know who is determining the value of a model for clinical practice (developer, user, customer, independent authority/reviewer). Domain knowledge, M&S expertise and experience, and independence are all important.

Cheers !
Martin


#############################################################################


Jacob Barhak

Mar 18


Please everybody,

Please continue this on the forum/wiki.

If there are differences in opinion they should be public and subject to external scrutiny.

I hope no one objects.

#############################################################################


Pras Pathmanathan

Mar 18


Sorry, just before receiving this email, and since a few people had requested responses on the forum, I have just created a topic on the forum with your email and my response. Please delete that topic if you think appropriate.

My response in any case is:

My vote would be against this publication (in favour of proceeding with the global survey and the subsequent publication). I personally do not see the value in the committee results once the global survey results available, that would make it worthy of a separate publication. And I don't think there will be enough information to draw strong conclusions about committee-derived vs publicly-derived guidelines.
That being said, since the 10 Simple Rules paper will have to be quite short and with little room for discussion, one option could be a follow-up article discussing the rules, which could (amongst other things) contrast the survey based rules with the committee perspective.
Pras



#############################################################################

Pras Pathmanathan

Mar 18


(Actually, I was able to delete it myself)


#############################################################################
END OF TRANSCRIPT
#############################################################################

If anyone has any corrections, please reply to this forum or ask me to correct those - no spell checking or major editing was done. I announced about this post to all recipients and I got permission from one person - no objections were recorded.

Here is a list of edits made

1. Deletion of long signatures of authors
2. Formatting of mailer headers to match the same format
3. Deletion of repeated text due to inclusion in reply emails
4. File attachments are not included
5. New lines were compressed - so spacing may look different
6. Deletion of messages that contain private communication between me and the chairs. These posts are partially off topic and should be made public under a different forum discussion about openness that should be started according to previous meeting minutes. I would be glad to make those public in that venue.
7. Vote results are not in this thread and I expect the chairs to make those public as well as all other communications that was directed to them regarding this matter.

Again, please feel free to post corrections of continued discussion to this forum. And I do hope to see the vote results and missing communications on this thread.

Jacob