Workflows & Standards & good practices

[C. Anthony Hunt]

Lealem,

Re: uploading to the CPMS Glossary wiki page. Yes. I have added that task to my to-do list.

Regards
-Tony-

[Pras Pathmanathan]

Hi all,

This is a very interesting discussion. Here are a few comments

I don't really feel like I have a 'workflow', but rather than just not reply I thought it might be interesting to really think about why I don't have, or don't feel like I have, a workflow. Some reasons are:
* I don't tend to ever work on the entire process - most of my work is related to one small aspect of an overall workflow, usually specific research questions related to models that have already been developed (assessment approaches, numerics, etc)
* I don't think in terms of a formal consistent workflow, and I don't know if there will be enough people who do for you to get a good response to your email
* Applications research that I do is usually in the hypothesis-generation (low risk) arena; which often involves with 'playing around' with current models until there are enough results for a paper. For such low-risk application I wouldn't be surprised if workflows as implied by the presentation of the work in a paper is very different to what happened chronologically in real life.

Some specific comments on the workflows

* Lealem's is more natural to me, and does seem very complete. The other workflow feels more like a series of questions that could be asked as you go through the process (especially the early stages of the process - the workflow feels weighted towards the early stages). But the two workflows together is a useful resource - maybe they can be merged in some way
* Ahmet asked if workflow is likely to depend on: context (e.g. research vs clinics); strategy (modelling approach); risk; discipline; environment (e.g. time/money constraints). It may well be that one of the biggest determinant of workflows is personal background (for example, engineer vs comp scientist/mathematician vs biologist).
* A 'Grand Unified Workflow' is unlikely (and probably not your aim) - but explicitly describing all these things is useful to read and will make for an interesting publications - I look forward to seeing more (and I'm planning on looking more closely at the publication (Peterson et al, BMC Systems Bio) linked earlier)

Pras

[James Sluka]

I received this groups email via the IMAG/MSM email list. Looks like an interesting, though challenging effort.

Looking through the documents attached to the email I don't see a workflow that recognizes that a computational model is typically a model with multiple conceptual layers. As a computational modeling project progresses the "model" evolves from (minimally) a biological description (conceptual model) to a mathematical model to a computational model. The model's output is from the computational model but what is often lost at that level is the original biological model's concepts (see attached PPT slide).

I think this poses a significant challenge to verification and utilization of computational models. To ensure that the computation actually represents the biological model (which may have undergo a number of revisions during the modeling workflow) it is necessary that the final biological model embodied in the code is recoverable from the code. This should include the recovery of all of the biological descriptors and formalisms.

The current state of the art in MSM is that the mapping of biological concepts to parts of the code is laborious, if not out right impossible. Therefore, verifying that the code implements the biological model is generally not possible and reuse of a model (or one of its components) is difficult.

A successful computational model embodies a significant amount of biological knowledge. Currently, there is often no way to leverage that biological knowledge without painstakingly recreating the modeling effort starting with a publication.

This challenge is an opportunity that needs to be considered. Can a workflow (and the needed tools) be developed that addresses the goals of the CPMS by simultaneously handling the various conceptual layers of a model? I think this would go a long way towards increasing the acceptance of MSM by enabling "wet-lab and clinical scientist to 'see' and begin to understand how and why M&S should be 'on the table' at the start of (almost) any research project." If the wet-lab and clinical researchers see a tool that allows them to describe a systems in terms they are familiar with and then have the ability to "drill-down" into the mathematical and computational details, then that tools provides a lingua franca linking the experimentalists to the computationalists.

jps

[C. Anthony Hunt]

Thank you, James. We appreciate you joining the discussion. Before responding, let me summarize observations made in “CPMS's Interest in Workflows 19Oct15”. At this stage, we simply seek a collection of real, yet somewhat polished, idealized workflows that are sufficiently fine grain to bring workflow differences between various consortium teams into focus and enable discussion of possible ways to use and enhance workflows to improve credibility of models.

Given that, please contribute a particular workflow example drawn from one of your team projects.

Regarding your comment: a research team should be able to explicitly choose a workflow depending on the purpose of their research project and the roles of the M&S components. The vision of designing an example workflow that can handle the various conceptual layers of a model is fine. However, we have to identify that, in order to "zoom out" and design a single workflow that a given team (or a collection of teams) should use, or is always expected to use, regardless of the purposes of a project, is a different vision. The former risks being a trap because it may degenerate into the latter.

Some workflows might (again appropriately) be designed to address different sets of model layers. Perhaps some projects should NOT embody the biological model within the computational model. Perhaps even some should not embody the math model in the computational model. Whether these models should or should not connect such layers depends on the project's objectives.

An additional note: the three, example workflow documents that we provided are obviously idealized, "linearized" workflows, rather than the actual, non-linear, often unique (sometimes messy) workflows most consortium members really experience. We envisioned idealized workflows being easier to discuss.

What we are looking for, here, is not merely a workflow whose artifacts are documents and models. Rather, we seek workflows whose artifacts are finer grained workflows.

[Martin Steele]

This is an interesting discussion and is similar to those we’ve had within NASA in defining the M&S Life Cycle. In the NASA Standard for Models and Simulations (NASA-STD-7009) Revision A, a general M&S Life Cycle is defined and parallels commonly known program management life cycles. We have found that attending to the needs of each life cycle phase helps avoid problems usually experienced downstream (e.g., empirical validation is often difficult because of problems created earlier in the M&S life cycle).

I also have included some comments & questions to Ahmet’s workflow (From A. Erdemir on Mon, Sep 21, 2015 at 8:02 AM).

Model Workflow or Life Cycle.docx

(60.45 KiB) Downloaded 78 times

[Martin Steele]

This reply is to James Sluka.
James, I have a couple different responses to your entry.

First, I like the PPT chart you provided, and it’s similar to other historical depictions (flows) of the process of modeling. The answer to your question at the bottom of the chart depends on how well the process of model development was accomplished. Linking model output back to, not just the original model concepts, but even more necessarily to the reality the model represents, is absolutely crucial. The “patient” should not be forgotten when making prescriptions from a diagnosis.

Ensuring the biological model is recoverable from the code is the exact purpose of first verification, and then validation. Tracking and managing revisions of the model and the code is crucial.

Here is my other response to your entry, and it’s a hard line response. Yes, “mapping of biological concepts to parts of the code is laborious.” If it is impossible or incomplete, a potentially great amount of risk is incurred in every use of the model. Results from an incompletely verified model, or a model used outside the domain of validation, should be so labeled. Would you buy the results of a model that is not fully verified, or used in a manner that is outside the scope of validation?