<div dir="ltr"><div>Hi Melanie, Hi Jeljer, Hi Stefan, Hi Olaf,</div><div><br></div><div>Since the 14-March deadline for paper submission is coming closer, and since we were discussing a panel, I would appreciate it if you can decide on a common prefix for the panel.</div><div><br></div><div>So far we had a discussion revolving around estimation, validation, calibration. Please choose a common denominator title to fit all your work that will form a base for the panel and submit it to the list.</div><div><br></div><div>So far, here are the titles you suggested as I extracted them from your posts:</div><div>Melanie: 'simulation vs. estimation’ </div><div>Olaf: "data sources, constraints, validation issues"</div><div>Stefan & Jeljer: "model calibration" - </div><div><br></div><div>The last topic of "Model Calibration" seems to be a common denominator so far, yet still possible for debate - after all you had some nice discussions and may have a better idea.</div><div><br></div><div>Yet if the last topic prefix is comfortable to you all, then I ask that you will submit your paper title to the mailing list to set the expectations from the panel and leave you all sufficient time to write the short paper.</div><div><br></div><div>I look forward to see your paper titles.</div><div><br></div><div> Jacob</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 8, 2016 at 9:29 AM, Jeljer Hoekstra <span dir="ltr"><<a href="mailto:jeljer.hoekstra@rivm.nl" target="_blank">jeljer.hoekstra@rivm.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><font face="sans-serif" size="2">Dear Jacob, Stefan and others,</font>
<br>
<br><font face="sans-serif" size="2">I agree with Jacob that estimation and
calibration generate parameter values based on observed data. I never explicitly
thought about in those terms but that is what it is.</font>
<br><font face="sans-serif" size="2">I have always used the terms in combination;
calibrate a model and estimate a parameter. I don't consider the one an
automated search and the other a manual human search per se . Usually if
you estimate a parameter the focus is on one parameter and a lot of statistical
theory and software exist to do that automatically. Whereas if you calibrate
a model the focus is often on all or a group of parameters in the model,
which is perhaps somewhat messier and needs more human interference. I
am not sure though.</font>
<br>
<br><font face="sans-serif" size="2">I am interessted in the topic Stefan
raised. What if you find parameters in the literature or you have estimated
them yourself with some dataset and then you test and adjust those variables
(calibrate?) so your model replicates some other dataset better.
How much change in those parameters do you accept, keeping in mind that
the parameter may be interpreted slightly different in the calibrated model.</font>
<br><font face="sans-serif" size="2">Furthermore, if you calibrate your model
you need some goodness of fit criterium. I wonder if people have experience
with weighing output variables including a mixture of categorical variables
(e.g. dead/alive, smoking) and continous variables( e.g. BMI, cholesterol
levels).</font>
<br>
<br><font face="sans-serif" size="2">Validation is a related subject. I consider
a model validated if it can mimic, to some degree a dataset that was not
used to calibrate/estimate it. Obviously also here you will need some goodness
of fit criterium to see if the model is validated or not. In my experience
we do not often have the luxury of a complete extra dataset for validation.
So we end up in the discussion Stefan mentioned, if the model is calibrated
how far do we accept parameters to be different from those estimated elsewhere.</font>
<br>
<br><font face="sans-serif" size="2">@stefan, thanks for poining out GAMLSS
we will have a look.</font>
<br>
<br><font face="sans-serif" size="2">best wishes</font>
<br><font face="sans-serif" size="2">Jeljer</font>
<br>
<br>
<br>
<br><font color="#5f5f5f" face="sans-serif" size="1">From:
</font><font face="sans-serif" size="1">Jacob Barhak <<a href="mailto:jacob.barhak@gmail.com" target="_blank">jacob.barhak@gmail.com</a>></font>
<br><font color="#5f5f5f" face="sans-serif" size="1">To:
</font><font face="sans-serif" size="1">Stefan Scholz <<a href="mailto:stefan.scholz@uni-bielefeld.de" target="_blank">stefan.scholz@uni-bielefeld.de</a>>,
</font>
<br><font color="#5f5f5f" face="sans-serif" size="1">Cc:
</font><font face="sans-serif" size="1">"<a href="mailto:popmodwkgrpimag-news@simtk.org" target="_blank">popmodwkgrpimag-news@simtk.org</a>"
<<a href="mailto:popmodwkgrpimag-news@simtk.org" target="_blank">popmodwkgrpimag-news@simtk.org</a>>, Mélanie Prague <<a href="mailto:melanie.prague@isped.u-bordeaux2.fr" target="_blank">melanie.prague@isped.u-bordeaux2.fr</a>>,
Jeljer Hoekstra <<a href="mailto:jeljer.hoekstra@rivm.nl" target="_blank">jeljer.hoekstra@rivm.nl</a>>, "Dammann, Olaf"
<<a href="mailto:Olaf.Dammann@tufts.edu" target="_blank">Olaf.Dammann@tufts.edu</a>></font>
<br><font color="#5f5f5f" face="sans-serif" size="1">Date:
</font><font face="sans-serif" size="1">05/02/2016 00:50</font>
<br><span><font color="#5f5f5f" face="sans-serif" size="1">Subject:
</font><font face="sans-serif" size="1">Re: [Population
Modeling] Discussing the Population Modeling panel in SummerSim</font>
<br>
</span><hr noshade><div><div class="h5">
<br>
<br>
<br><font size="3">So Stefan,</font>
<br>
<br><font size="3">Since this is part of the discussion on your panel topic,
you will have to choose the topic and I will try not to intervene much.
Yet do allow me to add a note regarding calibration and estimation.</font>
<br>
<br><font size="3">Both calibration and estimation generate model parameters
as outputs from observed known data.</font>
<br>
<br><font size="3">Will you agree with me that the term calibration would
be more appropriate to human manipulated parameters while estimation
is perhaps more general term that includes automated machine algorithm
methods? </font>
<br>
<br><font size="3">I have seen the term estimation used for parameter estimation
using Delphi style human voting, so I think "estimation" would
include "calibration" as a sub category. </font>
<br>
<br><font size="3">Even though there is always some sort of human input
to the modeling process, it seems things are becoming more automated these
days. What term would you use for heavily machine dependent estimation
algorithms as opposed to human tightly controlled calibration? Is
there a term for those methods anyone in the list prefers using?</font>
<br>
<br><font size="3">Hopefully this will contribute to the panel discussion.</font>
<br>
<br><font size="3">
Jacob</font>
<br>
<br>
<br><font size="3">On Thu, Feb 4, 2016 at 4:28 AM, Stefan Scholz <</font><a href="mailto:stefan.scholz@uni-bielefeld.de" target="_blank"><font color="blue" size="3"><u>stefan.scholz@uni-bielefeld.de</u></font></a><font size="3">>
wrote:</font>
<br><font size="3">Dear all, <br>
<br>
I just want to add the issue of "model calibration" which seems
to me as being related to estimation and validation. Just to give a short
example what I mean by calibration: We use some information of a data set
to estimate our model parameters, run the model based on those parameters
and see that the results are not externally valid. We the iteratively change
some of the input parameters until the model results are externally valid.
I would find it very interesting to discuss if calibration should be performed
and if so, how far from the originally estimated parameters you would accept
your calibrated values to be. [I hope I am not stating the obvious or missed
some guidance on this topic already available ;-) ]<br>
<br>
@ Jelster: If you use R, maybe the GAMLSS-package developed by Prof. Mikis
Stasinopoulos is helpful to you. As far as I understand it, the package
was developed from the need of parameter estimation in agent-based modeling.
You can estimate all parameters of large list of probability distributions.
So let's say you want to estimate the probability of getting diabetes conditional
on age, sex, education, etc. You can estimate a general linear model using
a beta-distribution and include the resulting coefficients to include them
in your model. So, every person in your model can calculate the parameters
of the beta-distribution based on their age, sex, education, etc. and you
can draw random numbers from that distribution to determine whether a person
gets diabetes or not.<br>
<br>
Best,<br>
Stefan</font>
<br><font size="3"><br>
</font>
<br><font size="3">Am 03.02.2016 um 01:45 schrieb Jacob Barhak:</font>
</div></div><p><div><div class="h5"><font size="3">Hi Melanie, Hi Jeljer, Hi Olaf, Hi Stefan,</font>
</div></div><p><div><div class="h5"><font size="3">It seems the panel is forming nicely. I will try
to summarize what we had so far and help figure the rest.</font>
</div></div><p><div><div class="h5"><font size="3">1. The topic seems to revolve around "estimation and
validation in population modeling" with some variations. If you are
all ok with this general topic, I suggest we stick with it as a base.</font>
</div></div><p><div><div class="h5"><font size="3">2. It seems there is agreement on separate papers with
the same title prefix. So please allocate time on writing a short 3 page
paper. Since panelists are not closely affiliated, each will review the
papers of another panelist which will contribute to panel cohesion since
the panelists will influence each others final paper. Note that the review
process is public and non-blind.</font>
</div></div><p><div><div class="h5"><font size="3">3. Presentations followed by a period of questions to all
panelists seems to be the choice. I assume there will be 20-30 minutes
per panelist, yet we will have to set timing once we know number of presentations.
</font>
</div></div><p><div><div class="h5"><font size="3">Olaf asked about other presenters. Yes, there will be other
presentations by non panelists. In fact any one of you can choose to detach
from the panel and submit a paper on their own. I will send a CFP to the
list following this message. </font>
</div></div><p><div><div class="h5"><font size="3">The difference for panelists would be: <br>
1. Panelists will have some discount that SCS promised - I have no exact
details yet. This makes sense since they will have more involvement.
<br>
2. Panelists will gain extra exposure which you are already getting with
these communications. <br>
3. If time is available, panelists will get more time for discussion beyond
other presenters. I will communicate with organizers to see what is possible
beyond that. Yet for now, assume the panel is part of the BMPM track.
</font>
</div></div><p><div><div class="h5"><font size="3">Note that SummerSim is a Multi-conference, so having a
panel may attract more people. From the past, you should expect about
10-20 in the room for the presentation if last years are indicative. I
suspect a panel can attract more. </font>
</div></div><p><div><div class="h5"><font size="3">So for panelists still interested, please: </font>
</div></div><p><div><div class="h5"><font size="3">1. Confirm that you are ok with the topic and format by
sending an email to this list. Or continue discussing the topic until consensus
is reached. And you can split into two panels with separate topics, or
announce you are interested in a paper outside the panel. </font>
</div></div><p><div><div class="h5"><font size="3">2. Start writing a short 3 page paper to submit to the
SCS web site. Recall that the title prefix should be the same for all papers
if you are in the panel. </font>
</div></div><p><div><div class="h5"><font size="3">3. Allocate time to review a paper or two by another panelist.
This review will be public. </font>
</div></div><p><div><div class="h5"><font size="3">Hopefully this explains the next steps and I hope more
panelists would express interest in the topic forming. </font>
</div></div><p><div><div class="h5"><font size="3">
Jacob<br>
<br>
<br>
</font>
<br><font size="3">Dear All, </font>
<br><font size="3">I agree with Stefan re 1 and 2. </font>
<br>
<br><font size="3">1. I like the idea of talking about input-output stuff
- data sources, constraints, validation issues.</font>
<br>
<br><font size="3">2. Fully agree with Stefan. </font>
<br>
<br><font size="3">3. If we have slides, this should be flash talks, not
longish formal presentations. I am still unclear whether we have presentations
from conference participants who are NOT panelists?</font>
<br>
<br><font size="3">My 2 cents</font>
<br><font size="3">Olaf</font>
<br><font size="3"><br>
</font>
<br><font size="3">--</font>
<br><font size="3">Olaf Dammann, MD</font>
<br><font size="3">Professor of Public Health & Community Medicine</font>
<br><font size="3">Tufts University School of Medicine </font>
<br><font size="3">Boston, MA 02111</font>
</div></div><p><div><div class="h5"><font size="3"><br>
On Jan 21, 2016, at 4:00 AM, Stefan Scholz <</font><a href="mailto:stefan.scholz@uni-bielefeld.de" target="_blank"></a><a href="mailto:stefan.scholz@uni-bielefeld.de" target="_blank"><font color="blue" size="3"><u>stefan.scholz@uni-bielefeld.de</u></font></a><font size="3">>
wrote:<br>
</font>
<br><font size="3">Hi all,<br>
<br>
thanks for all your efforts! Here are my thoughts on the three points:<br>
<br>
1. Topic: I am not quite sure whether the topics really do mean the same
thing. I would understand Melanies suggestion as "what are the differences
in the results/<b>outputs</b> from estimation vs. simulation" whereas
I would understand the topic "estimation in population modeling"
more as estimation of model <b>input </b>parameters. (Please tell me, if
I got that wrong!) I think both are interesting topics and maybe we could
bring them both together under the general topic of validity (external
and internal). (i.e. how do we estimate model input to get externally valid
results and how do we assess the latter)<br>
<br>
2. I would vote for separate papers under the same topic, sharing the same
prefix.<br>
<br>
3. I would go for option B). Option A) is fine as well, but we should make
clear if there are some contrary opinions on this topic. If panelists agree
on almost every topic, this might get boring. Also, I would see the number
of people in the audience a critical factor for a panel. If we are a small
group, sharing the methods used for estimation and discussing it in the
group might be more beneficial to all attendees.<br>
<br>
Best,<br>
Stefan<br>
</font>
<br><font size="3">Am 19.01.2016 um 19:42 schrieb Jacob Barhak:</font>
<br><font size="3">Hi Melanie, Hi Olaf, Hi Stefan, Hi Carl,</font>
<br>
<br><font size="3">You all expressed interest in appearing in a population
modeling panel in SummerSim.</font>
<br>
<br><font size="3">Melanie also suggested a topic:</font>
<br><font size="3">"differences and extrapolation concerns around 'simulation
vs. estimation’ in bio-medical area"</font>
<br>
<br><font size="3">At this point, I wish interested parties to discuss the
following:</font>
<br>
<br><font size="3">1. The topic - feel free to suggest alternative topics/titles
and we can see how having the panel will contribute to the topic. Note
that if we end up with different topics, it is also ok since others may
join to support the topic you suggested. Hopefully there will be synergy,
yet complementary topics or even different opinions are possible. This
discussion itself is valuable.</font>
<br>
<br><font size="3">2. Writing Format. The conference includes a paper.
Part of the discussion should be how do you prefer to be published. Do
you want a joint paper? Or would you like each to submit a short paper
with similar topics? This would probably be tied to the topic you suggest.
Yet note that whatever paper format chosen, it will undergo public non-blind
review.</font>
<br>
<br><font size="3">3. Presentation format: How would you like the talk to
be? Possibilities include: A) Totally informal discussion where panelists
converge amongst themselves, possibly with moderation and questions from
he audience. B) Presentations with a projector of each panelist and then
a period of questions. C) A combination of both, for example very
short introductions with a projector and then a discussion. Assume half
an hour per panelist, yet this may change.</font>
<br>
<br><font size="3">As a default starting point for discussion, allow me to
suggest the following:</font>
<br>
<br><font size="3">1. Topic Estimation in population modeling - its
generalization for what Melanie suggested - feel free to reshape it any
way comfortable to you.</font>
<br>
<br><font size="3">2. Writing format: Very short separate papers using the
topic as a title prefix. to have a common prefix fro all panelists. Here
is an example: Estimation in population Modeling - application in
Disease Models. </font>
<br>
<br><font size="3">3. Presentation format: Short digital introductions of
about 15 minutes each - with only a few slides and a discussion that will
start with expanding prepared topics encountered during discussions
and review and then answering questions from the audience.</font>
<br>
<br><font size="3">This default can be changed during discussion. </font>
<br>
<br><font size="3">Please feel free to join this discussion if you are
interested in appearing in a population modeling panel in SummerSim - even
if you are not personally addressed. This post is initially directed to
those who expressed interest on this list, yet we can certainly expand
the scope to include more panelists, and I know of interest by others at
this point.</font>
<br>
<br><font size="3">I look forward to your opinions.</font>
<br>
<br><font size="3">
Jacob</font>
<br>
<br><font size="3"> </font>
<br>
<br>
<br>
<br>
<br>
<br>
<br><font size="3"><br>
</font>
<br><tt><font size="3">_______________________________________________<br>
PopModWkGrpIMAG-news mailing list<br>
</font></tt><a href="mailto:PopModWkGrpIMAG-news@simtk.org" target="_blank"><tt><font color="blue" size="3"><u>PopModWkGrpIMAG-news@simtk.org</u></font></tt></a><tt><font size="3"><br>
</font></tt><a href="https://simtk.org/mailman/listinfo/popmodwkgrpimag-news" target="_blank"><tt><font color="blue" size="3"><u>https://simtk.org/mailman/listinfo/popmodwkgrpimag-news</u></font></tt></a><tt><font size="3"><br>
</font></tt>
<br>
<br><tt><font size="3">-- <br>
Stefan Scholz<br>
<br>
University of Bielefeld<br>
Faculty of Public Health<br>
Department of Health Economics and Health Management<br>
P.O. Box 10 01 31<br>
D-33501 Bielefeld, Germany<br>
<br>
Phone: +49 0521 | 106-2648<br>
Mail: </font></tt><a href="mailto:stefan.scholz@uni-bielefeld.de" target="_blank"><tt><font color="blue" size="3"><u>stefan.scholz@uni-bielefeld.de</u></font></tt></a>
<br><font size="3">_______________________________________________<br>
PopModWkGrpIMAG-news mailing list</font><font color="blue" size="3"><u><br>
</u></font><a href="mailto:PopModWkGrpIMAG-news@simtk.org" target="_blank"><font color="blue" size="3"><u>PopModWkGrpIMAG-news@simtk.org</u></font></a><font color="blue" size="3"><u><br>
</u></font><a href="https://simtk.org/mailman/listinfo/popmodwkgrpimag-news" target="_blank"><font color="blue" size="3"><u>https://simtk.org/mailman/listinfo/popmodwkgrpimag-news</u></font></a>
<br><font size="3"><br>
_______________________________________________<br>
PopModWkGrpIMAG-news mailing list</font><font color="blue" size="3"><u><br>
</u></font><a href="mailto:PopModWkGrpIMAG-news@simtk.org" target="_blank"><font color="blue" size="3"><u>PopModWkGrpIMAG-news@simtk.org</u></font></a><font color="blue" size="3"><u><br>
</u></font><a href="https://simtk.org/mailman/listinfo/popmodwkgrpimag-news" target="_blank"><font color="blue" size="3"><u>https://simtk.org/mailman/listinfo/popmodwkgrpimag-news</u></font></a><font size="3"><br>
</font>
<br>
<br><tt><font size="3">-- <br>
Stefan Scholz<br>
<br>
University of Bielefeld<br>
Faculty of Public Health<br>
Department of Health Economics and Health Management<br>
P.O. Box 10 01 31<br>
D-33501 Bielefeld, Germany<br>
<br>
Phone: +49 0521 | 106-2648<br>
Mail: </font></tt><a href="mailto:stefan.scholz@uni-bielefeld.de" target="_blank"><tt><font color="blue" size="3"><u>stefan.scholz@uni-bielefeld.de</u></font></tt></a>
<br>
<br><font face="sans-serif" size="2"><br>
</font><font color="blue" face="Verdana" size="1"><u><br>
</u></font></div></div><a href="http://www.rivm.nl/Proclaimer" target="_blank"><font color="blue" face="Verdana" size="1"><u><br>
Proclaimer RIVM http://www.rivm.nl/Proclaimer</u></font></a>
<p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p><p></p></p></p></p></p></p></p></p></p></p></p></p></p></p></p></p></blockquote></div><br></div>