We encourage anyone interested to participate in this discussion as it will play a role in the development of the Guidelines for Credible Practice of Modeling and Simulation in Healthcare.
Lealem
****Start of email discussion*****
From T. Hunt on Wednesday, September 16, 2015 6:56 PM
For me, there are several workflow protocols and practices that come before—and are actually more important (in terms of work product) than—a downstream choice (or avoidance) of some standard.
Ahmet, from you presentation on Thursday, I infer that the same may be true for you.
Might it be worthwhile for the CPMS to try to initiate discussions within the Consortium on good M&S good workflow practices?
There may (or not), for example, be differences between a good practice to which Ahmet tries to adhere (during development of a new FE model) and the corresponding good practice to which I try to adhere (when developing an agent-based model).
The survey suggests that there should be method agnostic good practices.
Activities exhibiting noteworthy differences should also be of interest to others.
I could say more, but I'll stop here to get your reactions.
-Tony-
---
From A. Erdemir on Mon, Sep 21, 2015 at 8:02 AM
Tony, for me as well workflow protocols and practices come before choice of standards. For my work standards (their existence and our compliance to them) become an issue when we need to exchange data/models/results and when there are no appropriate format conversion tools. With this mindset, I view standards as the means to facilitate a collaborative workflow, evaluation of its outcome, and reuse of its components. Will I need one if I can convert from different file types easily? May be not.
Manufacturing industry built various standards for geometry exchange, i.e. in computer aided design. There is STEP (an ISO standard), there is IGES (an ANSI standard). Exchange of geometry in manufacturing was a significant need as different parts may need to be build by different providers and need to match at high tolerances. Nonetheless, standards for models of finite element analysis (even for meshes) has not gained much traction. I suspect that this is simply due to companies not necessarily sharing models with other companies, and they unify the use of a specific finite element analysis software within the company (allowing exchange of models between different groups within the company using a proprietary and a non-standard form).
I will be interested in someone (may be you) provide the anatomy of an agent-based modeling & simulation study from start to end; similar to what I tried to convey with my presentation for FEA. In following, we can go through various steps to identify overlapping themes and some that may differ.
The way I see it; a broad workflow in M&S goes
i) find data to build and evaluate the model
(from a credibility perspective convince yourself and others that the data is useful)
ii) process data to bring it in a usable form to incorporate in to a model - derivative data
(from a credibility perspective convince yourself and others that your analysis is correct and useful)
iii) assemble the model using various components of the derivative data
(from a credibility perspective convince yourself and other that your assembly is correct, i.e. you defined interactions between components right)
iv) conduct simulations under desired cases
(from a credibility perspective convince yourself that your simulation software operates as expected and the cases you simulate are represented appropriately and are relevant)
v) evaluate your results
(from a credibility perspective convince yourself that your results are believable, i.e. can have a desired realism and can be utilized for interpretation within context)
vi) report your work
(from a credibility perspective convince yourself that your report reflects your previous actions and can be followed to reproduce the outcome of your work)
vii) share your data/models
(from a credibility perspective convince yourself that the way you share your data/models to not result in potential errors in their interpretation)
How much is this modeling & simulation workflow different based on varying simulation strategies, disciplines, etc.?
Thanks for your input.
Best,
ahm.
---
From T. Hunt on Mon, Sep 21, 2015 at 2:51 PM
Great response. Thanks for taking the time to respond. My feedback is inserted
concurTony, for me as well workflow protocols and practices come before choice of standards. For my work standards (their existence and our compliance to them) become an issue when we need to exchange data/models/results and when there are no appropriate format conversion tools. With this mindset, I view standards as the means to facilitate a collaborative workflow, evaluation of its outcome, and reuse of its components.
Will I need one if I can convert from different file types easily? May be not.
Manufacturing industry built various standards for geometry exchange, i.e. in computer aided design. There is STEP (an ISO standard), there is IGES (an ANSI standard). Exchange of geometry in manufacturing was a significant need as different parts may need to be build by different providers and need to match at high tolerances. Nonetheless, standards for models of finite element analysis (even for meshes) has not gained much traction. I suspect that this is simply due to companies not necessarily sharing models with other companies, and they unify the use of a specific finite element analysis software within the company (allowing exchange of models between different groups within the company using a proprietary and a non-standard form).
I will be interested in someone (may be you) provide the anatomy of an agent-based modeling & simulation study from start to end; similar to what I tried to convey with my presentation for FEA.
OK. I have in mind the following.
* I create a workflow diagram that provides "the anatomy of an agent-based modeling & simulation study from start to end.”
* You use the outline below and content from your FEA presentation to create a similar FEA workflow diagram (without trying to force any matches to mine).
* We email the Working Groups and ask leads if one or more Group members will volunteer do the following:
– offer constructive/critical feedback on our workflows (with a view toward making both a good practice example)
– generate and contribute a similar workflow diagram for their particular area of M&S
If important differences emerge, we will have learned something useful. Similarities across 3 or 4 very different M&S domains may suggest an overarching good practice. Whatever the outcome, it should make a publishable communication.
Your thoughts?
-T-
---
From T. Hunt on Tue, Sep 29, 2015 at 4:13 PM
Ahmet, please attachments. I attached a PDF of the .docx in case that is more convenient.
It turned out that we had separate parts of complete workflow document, but not a whole. So creating a complete workflow was worthwhile. I looks like your steps below are a subset of a more complete workflow.
Can you take an hour or so this week and expand what you have below so that somewhat parallels the attachment and add examples as I did?
If we can get two or three others to do the same (I plan to ask Linderman first), my expectation is that we will begin to see workflow steps/stages that can be generalized across diverse MSM research projects. It may also become clear that several of the generalized workflow steps/stages (alone or in combination) can be restated as Best Practices.
Comments/additional thoughts?
Regards
-Tony- ---
From A. Erdemir on Thu, Oct 1, 2015 at 5:53 PM
Thanks Tony for these documents. I am on the road at this moment and coming back to the country on Saturday; so please bear with me as I try to assemble a more complete workflow for FEA. In the meanwhile, here are some thoughts:
As I quickly glance through your document it reminded me a manuscript that we wrote for reporting finite element analysis studies in fea. Many reporting parameters that we have listed there (under the categories Model Identification, Model Structure, Simulation Structure, Verification, Validation, Availability) are in parallel with the components of a complete workflow. Except, they need to be reiterated focusing on actions of M&S rather than their reporting. See manuscript at
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3278509/
The workflow that I provided in my previous e-mail (and in my presentation) essentially deals with various technicalities of implementation.
---
From T. Hunt on Fri, Oct 9, 2015 at 2:40 PM
Ahmet, can you please send me your enhanced workflow within the next few days. I’d like to send out the pair of workflows next week before my academic workload ramps-up.
Thanks
-T-
---
From A. Erdemir on Mon, Oct 12, 2015 at 10:03 AM
Tony, I apologize for my tardy response. It took me longer than I anticipated to return back to this and detail the workflow. Attached are a doc and pdf versions of the workflow document that I prepared. In future, I will most likely detail this document further by giving more concrete examples. Nonetheless, it portrays a commony strategy we employ to conduct studies using finite element analysis. It will hopefully encourage others to come up with workflows for their domain and tool of choice.
Thank you for following up on this.
Best,
Ahmet ---
From T. Hunt on Mon, Oct 12, 2015 at 10:59 AM
This week, I’ll format my document similar to yours. Because FE M&S is more mature than AB M&S, my document will be relatively sparse. That’s fine.
Examples will help, but I think that we have enough to get the ball rolling.
-T-
---
From T. Hunt on Mon, Oct 12, 2015 at 2:31 PM
As written, it seems that the decision was made prior to your step [1] that a FE analysis (M&S) approach is best/preferred/essential. That’s not an issue. I just want to confirm that is the case.
When we have several workflows, there may be different branch points, with each branch leveraging a specialized workflow and methods. Your FE analysis would be one branch.
Do you concur? If yes, I’ll suggest a couple of steps prior to your [0]. If you like what I suggest, then you could add one (sufficient for now) example question (whatever) such that the FE analysis branch is the obvious best approach.
It’s too early for me to guess if most branches emanate from a common node (step [0] followed by your [1]).
Additional commentary welcomed.
-T-
---
From A. Erdemir on Mon, Oct 12, 2015 at 3:54 PM
It is true that, when preparing this workflow document, I took the FE analysis as the preferred approach as essential. I concur that there needs to be branching in relation to specialized methods.
It will be good to have a preliminary workflow that can help the user/developer decide upon which branch to follow. I suspect that is what you mean by steps prior to my [0]. Nonetheless, without defining the problem and metrics of interest, this may be hard to do. Therefore, it may be wiser to include my [1] and my [2] into the initial workflow to decide upon the specialized modeling strategy. Also, it may be useful that (as a reminder) to add a step in my workflow after defining problem and metrics as Confirm FEA as tool of choice. In such a case, I will pose a question to the user - "Are you looking to solve for field problems?", give a biological example , e.g. mechanical risk of cartilage degeneration after anterior cruciate ligament deficiency - therefore exploring stresses and deformations of the cartilage. Does this make sense?
Cheers,
ahm.
---
From T. Hunt on Tue, Oct 13, 2015 at 3:17 PM
Ahmet, let me set the stage for the comments that follow. At the start of most wet-lab/clinical research workflows, it is understood that, for example, the huge variety of available/accessible analytical chemistry methods are “on the table,” should they be needed. They may not be mentioned in early draft protocols.
At some future time, it will become “a given” that a huge variety of computational M&S methods are on the table when new projects are first discussed. Rather than looking backwards at what we’ve done, for this workflow discussion, let’s look forward and imagine that we are getting close to that future vision.
When a new project is first discusses, no M&S approach/method is off the table. As tasks and issues come into focus along with things like budgets, some M&S methods are conceptually “pushed aside” until it is clear that one or a small set of methods will be needed to achieve objectives and goals.
So, I envision a (future) research workflow starting broad, with many M&S methods on the table, and then getting progressively narrow. We are focused on two workflows (A & B) where both FE & AB methods are on the table (with other methods) early on. However, after some stage in workflow A, AB methods are off the table. Only FE methods remain. Whereas, after some stage in workflow B, FE methods are off the table. Only AB methods remain.
Is that scenario reasonably clear?
See the attachment. My example at step 1.1 is bone fracture union/nonunion in adults. At that stage, until more is specified, FE & AB methods would be on the table along with many others (systems M&S, pattern recognition [in proteomics big data], etc.). However, as we answer the questions, the set of remaining methods shrink.
I think that I can envision answers (specific example answers) such that by the time we get to 6.1, only FE methods remain on the table. Do you agree? Depending on the answers, I can imagine that all but FE methods get pushed off the table earlier.
I’m sure that I can envision answers (different specific example answers) such that by the time we get to 6.1, only AB methods remain on the table.
Am I still making sense? If yes, then can we do that? We can pick different 1.1 responses and provide different examples in response to the questions. Long term, my expectation is that following best practices will lead the research team to the best M&S methods rather than having a biased modeler like me pushing them prematurely in my direction.
-T- ---
From A. Erdemir on Wednesday, October 14, 2015 at 7:56 AM
Tony, my comments are below.
Ahmet, let me set the stage for the comments that follow. At the start of most wet-lab/clinical research workflows, it is understood that, for example, the huge variety of available/accessible analytical chemistry methods are “on the table,” should they be needed. They may not be mentioned in early draft protocols.
AE: Agreed and understood.
At some future time, it will become “a given” that a huge variety of computational M&S methods are on the table when new projects are first discussed. Rather than looking backwards at what we’ve done, for this workflow discussion, let’s look forward and imagine that we are getting close to that future vision.
AE: I see. Essentially when drafting an broad workflow, we will assume that we have a toolbox full of variety of M&S methods, where each method may have their own nuance of conducting the work targeted to solve a specific problem (as described in M&S strategy specific workflow).
When a new project is first discusses, no M&S approach/method is off the table. As tasks and issues come into focus along with things like budgets, some M&S methods are conceptually “pushed aside” until it is clear that one or a small set of methods will be needed to achieve objectives and goals.
AE: Agreed. Logistics of the environment, expertise of developers, and capabilities of a M&S method may cause them to be pushed aside but until than the broad workflow should be immune to bias in choice of M&S.
So, I envision a (future) research workflow starting broad, with many M&S methods on the table, and then getting progressively narrow. We are focused on two workflows (A & B) where both FE & AB methods are on the table (with other methods) early on. However, after some stage in workflow A, AB methods are off the table. Only FE methods remain. Whereas, after some stage in workflow B, FE methods are off the table. Only AB methods remain.
Is that scenario reasonably clear?
AE: It is. As of now, some items in your broad workflow (provided in your e-mail) overlaps some of the M&S strategy specific workflows. Yet, when we end up using a specific M&S strategy, we (or the user) may want to assume that a broad workflow has been worked through to identify a relevant M&S strategy potentially suitable to address the specific problem of interest. Nonetheless, we can reframe overlapping steps in M&S specific workflows to relate to the broader workflow better; i.e. rather than saying "Define problem/context" in the specific workflow, we can say "Confirm this specific strategy fits to problem/context". I believe I do that in my mind anyways; for example, if I am interested in cartilage contact forces during walking, I think of the mechanics of the tissue (domains are defined) and I scan through different modeling strategies in my toolbox (rigid body dynamics based musculoskeletal movement simulations or finite element analysis). I may choose one over the other through a gross selection criteria. Let's say I decided to use FEA. Before I start working on the model, I think a bit harder to ensure that I can get what I want from FEA and then move on.
AE: Agreed.See the attachment. My example at step 1.1 is bone fracture union/nonunion in adults. At that stage, until more is specified, FE & AB methods would be on the table along with many others (systems M&S, pattern recognition [in proteomics big data], etc.). However, as we answer the questions, the set of remaining methods shrink.
I think that I can envision answers (specific example answers) such that by the time we get to 6.1, only FE methods remain on the table. Do you agree? Depending on the answers, I can imagine that all but FE methods get pushed off the table earlier.
I’m sure that I can envision answers (different specific example answers) such that by the time we get to 6.1, only AB methods remain on the table.
AE: Agreed, ideally by 6.1 someone should decide which specific M&S workflow need to be considered. Nonetheless, economics and expertise maay be influential, particularly when multiple methods can do the work, i.e. alternatives exist. Also, one interesting concept is that the problem may dictate the use of multiple modeling and simulation strategies, e.g. use FEA to get the mechanical environment, feed the mechanical environment to an agent based model to calculate bone adaptation and return back to FEA to rerun with change material properties, i.e. curse of domain/scale coupling.
Am I still making sense? If yes, then can we do that? We can pick different 1.1 responses and provide different examples in response to the questions. Long term, my expectation is that following best practices will lead the research team to the best M&S methods rather than having a biased modeler like me pushing them prematurely in my direction.
AE: Agreed. I am indeed a biased modeler as well simply based on what I know and what I have access to but for someone who doesn't know and who may have access to all; an objective workflow, immune to ourselves, will be greatly helpful.
---
From T. Hunt on Wed, Oct 14, 2015 at 11:06 AM
An observation/comment followed by a request:
You stated:
AE: Agreed, ideally by 6.1 someone should decide which specific M&S workflow need to be considered. Nonetheless, economics and expertise may be influential, particularly when multiple methods can do the work, i.e. alternatives exist. Also, one interesting concept is that the problem may dictate the use of multiple modeling and simulation strategies, e.g. use FEA to get the mechanical environment, feed the mechanical environment to an agent based model to calculate bone adaptation and return back to FEA to rerun with change material properties, i.e. curse of domain/scale coupling.
Agree. That’s how it should work. We’re not there yet.
If we can generate a set of diverse workflows that start agnostic about M&S approach but end with creative use of one (or two) particular method, I think that they will prove useful in explaining to the larger community how M&S can be integrated with wet-lab methods to achieve better solutions to R&D problems.
Request (referring to [GeneralWorkflow 13Oct15.pdf]):
Please select a step 1.1 context that will lead to FEA at step 6.1. Below you stated, “e.g. mechanical risk of cartilage degeneration after anterior cruciate ligament deficiency - therefore exploring stresses and deformations of the cartilage.”
Maybe the step 1.1 context statement could be a question related to [cartilage stress and deformation] <=> [cartilage degeneration after anterior cruciate ligament deficiency]
I’ll do the same for AB methods & will provide example answers to all 1-6 questions. I’ll send that to you this afternoon, and the you do something similar for your context.
The next step would be to complete our specialized workflows beyond 6.1.
Given those two workflow examples, it will be easier for other MSM consortium members to contribute.
-T-
****End of email discussion and start of transition to Public Forum***