The blog A Fine Theorem is an excellent source of thoughtful discussions of interesting, mostly very recent, papers in economic theory and related fields. A link to it appears topmost in Sites I Like on the right side of this site. After an altogether too-long hiatus, I returned today to A Fine Theorem to find several items of interest that are new to me.
One of these items is the discussion of why economists (economic theorists, mostly) want to see proofs, not simulation results, to be convinced. As I understand the post, economists recognize that it is totally impossible to nail down economic theories on a few parameters that will be conceivably measured better and better over time. We economic theorists just have such a much harder time dealing with the complexity of a social system than a physicist who studies gravity that, as a profession, we have given up on coming up with a true model. Instead we build models upon a small, unified set of principles, a Foundation for all economics, that can give us ways of thinking consistently in many domains of application within economics. A paper that presents nice theorems and their proofs contributes to this enterprise, especially if the assumptions of the theorems do not depart much from the commonly-agreed Foundation. A paper that reports the results of a few thousand simulations hides the reasoning within a black box. Hence papers with proofs are more usable in building the economic theory pyramid and therefore more popular. Theorists stay away from trying to use the opaque (to them) reasoning that drives simulation results.
(I tried very hard not to make here the obvious analogy of economic theorizing as a pyramid scheme, and as you can see, I failed. If economic theory only aspires to indoctrinate researchers in THE ONE WAY to build models—and we know that econometric testing of the predictions of such models has not been a shining beacon of perfectibility in avoiding narrowminded, blid even, adherence to the Foundation—then we may be building totally useless beautifully constructed mathematical models, in fact harmful models, as they teach policy makers the wrong things. See below for another stab at this subject, and my total capitulation to have to devote a new post to it.)
I am not convinced by the black-box statement. (Confession: I still need to read seriously the paper that the post refers to, which can be found here (PDF).) If the authors of simulations publish their code, the only reason to say that the logic of the result is hidden is the inexperience of the reader with algorithms. Without going so far as to side with Wolfram, who said in his massive A New Kind of Science that all nature (all of it, including the living bits of nature and their behavior) is a bunch of algorithms, I would propose that we economic theorists need a better education in understanding algorithms.
As Leigh Tesfatsion says in one of her surveys of agent-based simulation (unfortunately I have no exact reference right now), every single run of a simulation is a mathematical proof, just with assumptions that appear narrower than we are used to (for instance, because they specify parameters numerically and exactly). A few thousand simulation runs can cover a decently-sized portion of the parameter space and then they are arguably as general, if not more, than a formal theorem on the same conceptual domain.
And things can be even worse for formal theorizing. It feels like we have been working with assumptions and techniques we are familiar with and which deliver consistently publishable papers based on assumptions from the Foundation. We are uncomfortable thinking about the possibility that this foundation is a quicksand. Computer simulations have a clear advantage here: flexibility. You can experiment with a simulation without feeling as restricted as by the straightjacket of mainstream economic theorizing. Simulations are more light-footed than Foundation-based theorizing, so they have a better chance to avoid sinking in the quicksand. (This is veering too far from the topic of the post, however, so I will have to write more in this vein in a subsequent post of mine.)
We could also imagine a source of unease about simulations in general, based on the complexity of the computer’s innards and even of the computer’s output. When the four-color theorem was proved by computer (not by simulation, but by exhaustive enumeration of the possibilities, as I understand it) a debate raged for a while among mathematicians. How could we accept a proof so gigantic that we cannot inspect it ourselves? I may be wrong, but mathematicians as a whole have resolved this debate by accepting that computers can give us acceptable proofs. We simply must accept the black-box problem; it comes with too many demonstrably solid and useful contributions of electronic computation to our lives. This is true even in cases where the box is kept black for commercial reasons, such as the exact Google search algorithm. A Google search these days may give us a few too many content-farm results that are not useful, but it still is an amazing tool whose operation is secret to most people.
As someone who has dabbled in agent-based computer simulations and intends to continue doing so, I appreciate the clarity this discussion gave me on the unpopularity of this kind of work among my colleagues, as well as the stimulation to think more about this issue. The unpopularity should not be overemphasized, however, as simulations are studied enough and published enough in economics to have their own volumes in the Handbook of Economics Series (Handbook of Computational Economics, in two volumes as of now, the second one on agent based computational economics, the kind of simulation I consider most useful for economists).
The curious incident of DD on the Web 
Dimitrios,
Enjoyed the comments, but I want to take issue with one of the claims above which I think is really at the heart of misunderstandings between ABM types and more traditional theorists. "Econometric testing of theoretical models" is, in and of itself, completely incompatible with what I think the role of theory in social science ought be. As you noted, physicists and chemists attempt to write models of better and better match with reality, hoping to actually pin down parameters in nature. For complexity reasons, such a goal is hopeless in economics. Rather, economic theorems can only, using math for standardization and formality, state logical arguments about how interagent interactions aggregate. They help provide intuition to policymakers about what effects might be important and how important they might be. They do not, in the strict sense, make predictions.
So then, why ambivalence about ABM? Since theory provides arguments only, and since the mathematical structure itself *is* the argument, only an argument written in a consistent way which follows a foundational paradigm and allows simple comparison with earlier work, the proof *itself* is the critical component of theory. Although simulation is, as you note, nothing but a series of proofs, the exact nature of how those proofs work is what is hidden in the black box. I happen to like a lot of the simulation work, so (as I'm sure you agree) the final argument, the analogy, the intuition about what might be going on in society can be more easily and more comprehensively read from the output of a simulation than from a traditional theorem. But the black box hides the proof itself from future researchers, which in a real sense limits the ability of social science to progress.
LikeLike
I fear theory unconstrained by empirical testing, any valid sort of empirical testing. I realize that ABM can be seen as such a check, and conceivably (although as you say it is very hard, verging on the impossible) even econometrics, although its track record is not great. Using unfettered theory to "help provide intuition to policymakers about what effects might be important and how important they might be" carries the potential of an ideology, the one behind the logical proofs, hidden in the assumptions of the Foundation, pulling policy strings while potentially leading the policymakers to ruinous directions. No matter how great and transparent the proofs in the theory are, they may still be based on wildly wrong assumptions.
LikeLike
I completely agree with everything except the conclusion, hah! Much theory uses wildly wrong assumptions, much advice is guided by ideology, etc. – but testing the theory (and by testing, I suppose I mean everything from testing the assumptions to testing the "predictions"; I have a note I'm working on that discusses five ways in which theory can be "wrong", so I'm leaving out the other three) does not avoid ideological problems. From a philosophical standpoint, this is closer to someone like Quine ("Word & Object", for instance) than the pure positivists which seem to have strong influence on the philosophy of science thinking among economists…
In any case, most theorists probably don't worry about these issues at all, so I'm all for more discussion of them!
LikeLike