Credibility assessment: Is it a well posed problem?

[Marc Garbey]

To understand what we mean by credibility of a model, it will be nice to:

(i) put a software repository we share about two dozen of publication we believe are:
a. representative of the state of the art.

b. representative of the broad use of modeling

(ii) starting from this specific material: establish a step by step questionnaire that give elements on the "credibility of the model", very much like questions we have when we review,very simple things such as:

a. is the model completely explicited?

b. is the field of application of the model completely defined?

c. the data used to set up the model construction are available?

d. what is the input of the model?

e. what is the output of the model: a decision 0/1, a number ? a multidimensional field? a probability ? etc....

f. is an error assessment provided? what form: a priori error estimate?, a posteriori error estimate?, interval of confidence? etc...

g. what are the parameter of the models?

h. is there sensitivity analysis provided? linear? nonlinear?

i. what is the cost of the clinical data required to exercise the model?

j. what is the uncertainty on these data to be acquired to use the model ?

I am sure that we can come up with many questions that follows a rigorous logic...
A bit of graphic / classification on this questionnaire outcome, may show the complexity of the problem, and help to focus on what is feasible?

Marc Garbey

[Jacob Barhak]

Hi Marc,

You are aiming to create a model publication repository.

It seems a good idea.

It is possible to use our bibliography repository for that and add a note to each publication we add. And place all models into a specific folder.

Since you raised the idea, perhaps you can start with a few examples that others can follow?

Alternatively we can have a single spreadsheet with the questionnaire and we can upload the papers links to the Zoreto repository and update the spreadsheet in the SVN repository. This way we will have all the comparative information in a single file.

This can also help to determine the model scope.

[Ahmet Erdemir]

(ii) starting from this specific material: establish a step by step questionnaire that give elements on the "credibility of the model", very much like questions we have when we review,very simple things such as:

We have previously provided considerations for reporting finite element analysis studies in biomechanics, with a similar goal. See

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3278509/

I wonder if we can port some of those ideas to a broader use.

[Martin Steele]

To understand what we mean by credibility of a model

We need to be specific about what we are assessing the Credibility of:

• The Practice of Modeling & Simulation
• The Model
• The Simulation
• The Analysis
• The Results

Perhaps one place to start is looking at NASA’s Standard for Models & Simulations, which includes a method of credibility assessment for M&S-based analysis results.

is the model completely explicited?

What is "explicited"?

what is the cost of the clinical data required to exercise the model?

Does cost influence credibility?

[C. Anthony Hunt]

To understand what we mean by credibility of a model, it will be nice to:

(i) put a software repository we share about two dozen of publication we believe are:
a. representative of the state of the art.
b. representative of the broad use of modeling

The space of model types and model uses within healthcare is too large to make such an exercise (by this Committee) productive.

We recently looked at the variety of computational model types and uses within Pharma R&D:
http://onlinelibrary.wiley.com/doi/10.1 ... .1222/full
The variety is impressive, even intimidating. The the variety of model types and uses within healthcare is larger. My impression is that subsets of specific model use cases can be clustered under broader use case statements and that the credibility of models within a cluster can be established following similar good practices. However, my impression is also that practices that establish credibility for two quite different clusters (e.g. one that includes FEMs of bone fracture repair and another that includes clinical trial simulation for a candidate biologic) may--likely will--be different.

Given the above impressions,

[Martin Steele]

The space of model types and model uses within healthcare is too large to make such an exercise (by this Committee) productive.

The ‘space of model uses’ will likely be quite large, but the ‘space of model types’ may not be if we keep it at a high enough level. Understanding model type will immediately give context to a given discussion on a particular M&S, which gives clarity in any discussion. This is high level structure of model types that I have compiled:

Narrative
Verbal or Written Speech, Prose, Poetry, Requirements Specification

Visual
Pictures, 3D Virtual Objects, Graphs

Physical
Tangible Objects that are abstract or scaled versions of the real thing

Relational
Databases, Object Oriented Hierarchies, Organizational or Conceptual Structures

Mathematical, Physical, or Chemical Formula
Algebraic Equations, Differential Equations, Physics Equations, Chemical Reaction Equations

Behavioral
Simulations (Continuous, Discrete, Real-Time, Geometry, Agent-based)