Mathtimidation by analytic solution vs curse of computing by simulation

Recently, I was chatting with Patrick Ellsworth about the merits of simulation vs analytic solutions in evolutionary game theory. As you might expect from my old posts on the curse of computing, and my enjoyment of classifying games into dynamic regimes, I started with my typical argument against simulations. However, as I searched for a positive argument for analytic solutions of games, I realized that I didn’t have a good one. Instead, I arrived at another negative argument — this time against analytic solutions of heuristic models.

Hopefully this curmudgeoning comes as no surprise by now.

But it did leave me in a rather confused state.

Given that TheEGG is meant as a place to share such confusions, I want to use this post to set the stage for the simulation vs analytic debate in EGT and then rehearse my arguments. I hope that, dear reader, you will then help resolve the confusion.

First, for context, I’ll share my own journey from simulations to analytic approaches. You can see a visual sketch of it above. Second, I’ll present an argument against simulations — at least as I framed that argument around the time I arrived at Moffitt. Third, I’ll present the new argument against analytic approaches. At the end — as is often the case — there will be no resolution.

My journey from simulation to analytic solutions

A decade ago, when I first started using evolutionary game theory, I relied on simulations. At first, this was for completely practical reasons. I encountered EGT (or what I would later learn is EGT) in Tom Shultz’s computational psychology course (PSYC315 at McGill), and so I made computational models. You can see this in Shlutz et al (2009) — where my contribution to the paper was directly from ‘world saturation’ observations of simulations I made in Tom’s class; a few years later, I converted some of my other observations on probabilistic strategies from the class into easy early posts for TheEGG — and in Kaznatcheev (2010a) — where I assigned a ‘cognitive cost’ to some of the minimal complexity of tag-based models. But I was coming from a more pure math background so I wanted to be able to make more general statements than a few simulation runs. So I moved from presenting the average results of 20 or 30 simulations of a few parameter settings to doing broad parameter sweeps. This pushed me towards game space: a representation where I could see how my simulations behaved under a wide range of parameter values. By Kaznatcheev (2010b), you can see me sweeping over broad ranges of parameters.

As I learned more, I started to rationalize my simulation process to myself by saying that even though the models I was looking at were simple, they were still spatial models that were not analytically tractable. And this space, stochasticity, and discreteness is important even in minimal models (for examples, see: Shnerb et al., 2000). But as I read the literature more, I realized that people weren’t presenting simulations because they had already solved the ‘easy and tractable’ analytic cases. They were using simulations just because it was easy to code them and modify them. More importantly, in a lot of cases there was no reason to make an explicitly spatial model. Or if a spatial model was needed, to prefer one type of spatial model over another. This was especially relevant when the results that people claimed to follow from a game, actually followed from a specific but arbitrary choice of space instead.

This left me disenchanted with simulations and pushed me towards the minimal models that I could solve analytically. This focus on the analytic was important in helping me move to mathematical oncology and get a job at the Moffitt Cancer Center. In particular, I got David Basanta’s attention — in part — by showing how to analytically solve the INV-AG-GLY game that Basanta et al. (2008b) had instead studied through simulation of broad parameter sweeps. Given that I was still attached to my background with spatial models, this eventually resulted in Kaznatcheev et al. (2015) — where we spatialized in an analytically-solvable way David’s old INV-AG (or Go vs. Grow) game (see: Basanta et al., 2008a).

Simulation and the curse of computing

By the time I arrived at Moffitt, I was firmly in the ‘do it analytically’ camp. And always pushed for minimal models. My lingering question became: why would you even do simulations for simple evolutionary games?

A possible objection to the purely analytic focus is Patrick’s: “the main reason people prefer simulations is because it’s easier to present in a biology paper rather than explaining the regular payoff matrix and everything”. This is an important point on the rhetorical utility of computational models.

But a lot of papers that do present games, present the whole payoff matrix with symbolic parameter numbers. For an example, see Basanta et al. (2008a,b). Thus if we’re simulation games, we already have to go through the hard effort of justifying the payoff matrix and our parameterization of it. However, with simulation, we then go on to just stick a range of particular values — sometimes a very small range, sometimes a good wide range — and show simulation results. So doing an analytic treatment does not require more explanations at the start.

A purely analytic treatment might require a couple of sentences on the idea of a dynamic regime. But that is usually on topic. More importantly, it seems more honest since people don’t usually draw conclusions from the particular numbers the simulation of particular parameters outputs but rather its qualitative features. An example might be: “and the cancer took over the tissue if drug was low, but if it was high then the healthy tissue won”. These qualitative features are just dynamic regimes.

In a simulation, we have an extra step of inference that an analytic treatment does not require. In a simulation we observe a number of qualitatively similar results over a parameter range and make a qualitative conclusion. In an analytic treatment, we directly prove that a certain specific parameter range produces a certain specific kind of qualitative dynamic.

The most concrete advantage of eliminating this extra step of inference is that sometimes the qualitative observations of a number of simulations don’t actually capture an underlying regularity of a dynamic regime. This can be especially dangerous when one is tempted to cut corners by not carefully exploring their simulation or the way simulation results are aggregated (as is often required for stochastic models). For example, in Kaznatchev & Shultz (2011) I made the mistake of faulty inference from a simulation average to a dynamic regime. I saw an average proportion of cooperation trending towards zero and so innocently concluded — as would be consistent with theory in that particular case — that it was going to zero. In reality, there was a high heterogeneity in simulation runs that tended to bifurcate between going to all cooperate or all defect. I was able to notice this by 2012 when I eventually looked more closely at the individual simulations. But I should have done that right away. Trying to understand this unexpected behavior then taught me some useful analytic theory.

Another example I’ve seen with people starting to play with EGT for the first time — and part of what prompted my original discussion with Patrick — is when someone does a discrete-time simulation of replicator dynamics with a relatively large step size. In this case, one can get what looks like oscillations around a fixed point. But these oscillations are not due to some inherent property of the game being simulated. Rather they are a feature of the discrete step size. This can, of course, be of interest if there was a genuine reasons for discrete steps: like studying a seasonal population. But in most case, the discrete steps were made for simplicity of writing simulation code. In these cases, concluding an oscillatory dynamic regime is a mistake.

But these mistakes of inference are not that common, and are not central to my concerns. In most case, the inference from simulations to dynamic regime is correct. But why take the extra step? Here, my overall objection is not that the simulation is misleading but that it will draw a conclusion in broad symbolic and qualitative terms. But if we will draw a conclusion in broad symbolic and qualitative terms then why not use an analysis technique that is also symbolic and outputs qualitative statements (like dynamic regimes) instead of having a middle step of simulation? The step through simulation feels in some way like a non-sequitur. At least when the analytic alternative is right there.

So the cynic in me, would agree with Patrick on “easier” but disagree on “to present”. Rather, I would argue that the simulation is “easier to do” or “easier to delegate” at the expense of an unnecessary step of inference. A step we should save our readers from, even if we first noticed the effect through simulation.

Analytic solutions as mathtimidation

The cynic’s cynic might say though: doing an analytic treatment of dynamic regimes doesn’t actually add all that much beyond simulations. I often argue that simulations are preferred because they require marginally less background than analytic treatment. But that doesn’t mean analytic treatment of minimal models is hard to do. In fact, the whole point of minimal models is to pick ones that are straightforward to analyze. So from the perspective of just doing mathematics, few or no points are earned. Solving a well picked model should not be more difficult that an exercise.

What is more important is than lack of ‘math points’ is that we don’t trust minimal models enough to make sense of sharp analytic boundaries. I don’t think anybody thinks their minimal mathematical models captures all the relevant aspects of a phenomenon — I certainly don’t with my models. Rather, it is usually meant to draw our attention to a specific salient feature of the phenomenon. This is what I call heuristics, and what I focus on when I build analytically-solvable minimal models.

So, heuristic models in general are just cartoons or illustrations. Solving them analytically instead of just doing some simulations doesn’t make them less of an illustration. The analytic solution creates an extra certainty where no extra certainty is needed. In itself, this isn’t a problem. But often this extra precision and the ‘analytic’-brand can be used to make the whole model seem more certain. This is especially transparent with decorative mathematical models that accompany experiments without contributing any extra insights. But it also happens in purely theoretical work. In this case, the analytic solution of dynamic regimes acts as a bit of mathtimidation. Since the biggest uncertainty comes from the model itself rather than how its solved, using a more rigorous solution process can be a way to use mathematics to intimidate the reader into overlooking the shaky standing of the model itself. This is a criticism I often hear against economic models, but I think it applies just as well in biology.

This leaves me at a bit of an impasse. On the one hand, proceeding through simulation when an analytic option is available feels like a non-sequitur. On the other hand, the analytic option rarely provides something that can’t be gathered from just simulation and is too tempting to use to lend undue certainty to models.

It is tempting to side-step this impasse. And there are several sides. One direction around this problem is to embrace model pluralism. We can just pick and choose our methods based on what is most convenient in a given setting. And I certainly recommend model pluralism when it comes to heuristic models. Another direction is to turn from heuristic to abductions. We can invert simple analytically-solvable models as a way to directly measure abstract objects like games — as we do in Kaznatcheev et al. (2017) for measuring effective games in lung cancer. Here, inverse simulation can become a more difficult option that inverse analysis.

But both of these approaches leave this knot of tension intact. And I don’t know how to resolve it. Or if it needs to be resolved. Hopefully you, dear reader, will have some advice.

References

Basanta, D., Hatzikirou, H., & Deutsch, A. (2008a). Studying the emergence of invasiveness in tumours using game theory. The European Physical Journal B, 63 (3), 393-397

Basanta, D., Simon, M., Hatzikirou, H., & Deutsch, A. (2008b). Evolutionary game theory elucidates the role of glycolysis in glioma progression and invasion. Cell Proliferation, 41(6): 980-7.

Kaznatcheev, A. (2010a). The cognitive cost of ethnocentrism. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd annual Conference of the Cognitive Science Society.

Kaznatcheev, A. (2010b). Robustness of ethnocentrism to changes in inter-personal interactions. Complex Adaptive Systems – AAAI Fall Symposium.

Kaznatcheev, A., Scott, J. G., & Basanta, D. (2015). Edge effects in game-theoretic dynamics of spatially structured tumours. Journal of The Royal Society Interface, 12(108): 20150154.

Kaznatcheev, A., Peacock, J., Basanta, D., Marusyk, A., & Scott, J. G. (2017). Fibroblasts and alectinib switch the evolutionary games that non-small cell lung cancer plays. bioRxiv, 179259.

Shnerb NM, Louzoun Y, Bettelheim E, & Solomon S (2000). The importance of being discrete: Life always wins on the surface. Proceedings of the National Academy of Sciences of the United States of America, 97 (19): 10322-4

Shultz, T. R., Hartshorn, M., & Kaznatcheev, A. (2009). Why is ethnocentrism more common than humanitarianism? In N. Taatgen & H. van Rijn (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society.