simtk.org

Hello everyone,

During the December 19th Committee Members meeting, it was determined the Committee needs to decide to what degree (if at all) we include “Technology” in the Committee’s objectives via a majority vote among the Committee Members and Advisory Council. This is important for finalizing the Committee Summary presentation, which we use as an informational package for the greater community. So we would appreciate everyone’s participation in this effort.

The inclusion of “technology” in the Committee’s objectives was first introduced by Jacob in an earlier effort to finalize the Committee Summary presentation. There was some discussion that followed his suggestion, but a conclusive decision was not reached: https://simtk.org/forums/viewtopic.php? ... 4&start=10

During the Committee Members meeting, it was clear that there is general confusion regarding the definition of “technology”. Ahmet and Jacob offered the following definitions/descriptions of “technology”:

Ahmet’s definition/description: Hardware, software, algorithms that may facilitate establishing credibility of modeling and simulation in healthcare.

Jacob’s definition/description: Innovative game changing technologies establishing model credibility.

In addition to the lack of clarity in the definition of "technology", there were differing opinions on whether technology influences M&S credibility or if credibility is solely dependent on process and metrics for assessing confidence in the M&S within the stated context of use. Therefore, in order for us to move forward productively in this task, the Committee as a whole should first address two key issues:

First: We need to establish a cohesive definition the Committee can agree on.

Second: There is varying opinion on whether or not “technology” plays a role in establishing M&S credibility.

Once we’ve addressed these two items, the Committee Members and Advisory Council will be given an opportunity to anonymously vote on the following three items to determine whether or not we should include “technology” in the Committee charge. Ahmet and I will coordinate these two activities.

Option 1: Do not add "technology" in Committee charges.
Option 2: Add "technology" in Committee charges blended with an existing charge balancing the emphasis.
Option 3: Add "technology" as a separate Committee charge

We look forward to your contributions.

Lealem
I hope you are all enjoying the holidays, and all the best in 2014

I vote for Option 1. My concern is that the committee is setting its mandate to be broader and broader to the point where it is becoming unfocused. This was also my concern about what seemed to be a broadening from healthcare to more general non-clinical simulation issues in prior meetings (eg simulation for biomedical research) -- I may have missed it but I do not think that was voted on?? -

Overall my suggestion is to stick with the original title of the committee as a mandate: Credible Practice of Modeling & Simulation in Healthcare

wwlytton wrote:the committee is setting its mandate to be broader and broader to the point where it is becoming unfocused.

I agree with you, Bill. I think the issue of technology (irrespective of definition) is not something we should address. In fact, I personally believe that it is divergent from the initial intent of the Committee. My personal position is that technology only plays a role in efficiency, cost, help minimize human error etc., which are not metrics which measure the credibility of a model. Credibility is a function of the activities one does to build confidence in the M&S. This can be done using a wide array of technology, ranging from more standard/rudimentary technology (e.g. desktop) to highly sophisticated systems (e.g. HPC). But at the end of the day, both methods can help you reach the same goal. One may just allows you to get there faster than another. Consequently, I personally believe that the technology platform one uses to get to this is goal is up to the model developer.

Therefore, attempting to establish a guideline that defines and/or promotes which technologies to use for establishing credibility is not reasonable. In fact, it may even be inappropriate since technology changes all the time, while the principles like Verification, Validation, Sensitivity Analysis etc. tend to be fairly universal. So I think we should stick to establishing metrics/principles on how one would go about to establish credibility, and let the modeler decide what technology they are going to apply to meet the metrics/principles.

With that said, this is a Committee, and everyone will be given an opportunity to weigh in on the matter so that we arrive at a well informed decision.

wwlytton wrote:This was also my concern about what seemed to be a broadening from healthcare to more general non-clinical simulation issues in prior meetings (eg simulation for biomedical research) -- I may have missed it but I do not think that was voted on??

The short answer to your question is, no, this was not voted on. There is a good reason for this, but this is a separate topic from the issue of technology. So if it needs to be discussed further, please let me know where I should respond to this question.

I also agree with Option 1, and the general arguments put forth by Lealem and Bill.

Gary

Technological innovation is an important engine of change.

Lealem is right. Technology does change, so mapping technologies for every type of modeling seems unreasonable. Never the less, I like and support Ahmet's suggestion of pointing out technologies that improve credibility. This is useful and important.

The committee should also recognize the need for constant innovation and include this element in its charge somehow, "innovative game changing technologies" seems a reasonable way to convey this, yet using the word "tools" or "methods" instead of "technology" seems reasonable as well. This is open to discussion.

There is a tension between the overly conservative stagnating approach and the overly risky innovation. There is also a point of good balance. I wish the committee address this issue since we do live in a constantly evolving technological environment and we are focusing on computational models.

Some technologies/methods/tools allow us to ask and answer new questions that were harder to address before. This does impact credibility, especially if we are focusing on computational models in this committee.

I agree with Lealem that concepts such as validation and sensitivity analysis are important and should be high up on priority. Yet technologies that support, scale to the next level, and reduce human error should not go unnoticed. New tools do influence our concepts.

A good simple example is version control. This concept can be practiced without special tools. Yet version control tools available with recent technology elevate this concept to a new level that includes sharing and communication. And there are more similar examples.

I hope the committee will recognize the importance of technological innovation this in this discussion.

Yes, “Technology” should be included in the Committee Charge. Wing Liu

jbarhak wrote:I wish the committee address this issue since we do live in a constantly evolving technological environment...

This was not the principle under which this committee was proposed for. In my opinion, this is a charge for a different committee all together, because the challenges of technological change and projecting need and applicability is an arduous task on its own. We have not even addressed the issue of establishing credible practice (THE reason why the committee was established) let alone even attempt to address the challenges of technological change.

jbarhak wrote: Some technologies/methods/tools allow us to ask and answer new questions that were harder to address before. This does impact credibility, especially if we are focusing on computational models in this committee.

Just because one can build a model with the most advanced system with the intent of asking or answering new questions more easily does not mean that model is more credible than a model that is built on a basic computer. It is the VV&UQ and other activities aimed at building "believability" or "confidence" in the model that establish the credibility of the model. The reality is that a unvalidated model built on a sophisticated system is not more credible than a validated model that has been developed to run on a basic desktop, even if the second model cannot easily answer all of the questions of interest. On the same line, if the end-user/developer sees limitations in using a desktop, then this is a question of application domain/context of use. In which case, it is up to the end-user/developer to expand to meet their needs. Even then, credibility is not established unless the model goes through verification, validation, uncertainty quantification, sensitivity analysis, etc.

More technological capability does not translate to higher credibility.

jbarhak wrote: A good simple example is version control. This concept can be practiced without special tools. Yet version control tools available with recent technology elevate this concept to a new level that includes sharing and communication. And there are more similar examples.

In my view, this drives at the point I made earlier about technology making things more efficient/easier, but not more credible. A manual version control system that reliably tracks versions has just as much credibility as an automated version control system. The difference is that one is more laborious than the other. So in my view, this example does not support the argument that technology improves credibility. Technology presents the opportunity to make things more efficient, cost effective and minimize human error. These are not areas that the committee was intended to focus on.

Lealem represents the conservative approach considering the tension between "good old popular" and "too new to be true". I myself associate his words with the idea of "the worst enemy of the best is the bestest". And choice of words is intentional. I understand his position and even support it from some aspects.

Never the less I do believe that efficiency associated with technology is relevant for credibility. Especially since we focus on computational models and computation was driven for years by exponentially increasing computing power - see Moore's Law: http://en.m.wikipedia.org/wiki/Moore's_law
Think about models running on an old 70s super computer compared to a model running on a modern desktop and models running on a cluster on the cloud. There is a difference in the strength of the model and the ability to address a more complex issue.

I do, however, share Lealem's concern of overwhelming the committee with tasks - we are already behind schedule. Therefore I suggest that technology/innovation will not carry a separate deliverable, instead we can just incorporate it in our other findings. This discussion combined with the wiki page Ahmet started, and reference to innovation/technology whenever seems appropriate in our text should be sufficient to support the innovation/technology aspect.

And I also suggest that instead of using the term "best" associated with technology we use the term "fitness" or "best fitting". In fact, what about adding the following rule to our list:
"Use the most innovative technology available that fits the task" ?

I hope this direction will find wider support.

jbarhak wrote:"Use the most innovative technology available that fits the task" ?

One rarely uses the most innovative technology available to establish a model. In fact, one generally would use the most well established platform since the most innovative typically tend to be less proven, and therefore lack credibility within the greater community. Also the concept of "innovative technology" is wide varying.

From my own experience, and I'm sure many would agree, one would apply the technology platform that is sufficiently appropriate and most affordable to accomplish the task. This is not always the most "innovative" platform. In this light, it is always up to the developer to decide what technology to use. In fact, the developer can decide to develop their own platform if they saw fit (I can think of several cases where this has been done). But, regardless of what the developer uses, it is the process and methods they use that ultimately builds confidence in their model, and not what platform they built it on.

I think I will stop posting on this matter now. My position is clear in that I think including "technology" in the committee charge is inconsistent with the original intent of the Committee.

It would be interesting to hear what the rest of the Committee has to say about this. But I recommend we make a decision by the end of January.

I think option 1 because it seems premature to expand the scope.

That said, "Technology" (no matter how defined) could be indirectly addressed through the original Charge:

Adopt a consistent terminology
Propose guidelines and procedures for credible practice
Demonstrate workflows for credible practice
Promote good practice

(e.g., Guidelines we identify may not call out a particular technology, but could lead you towards a particular technology based on the recommendations).