The blog A Fine Theorem is an excellent source of thoughtful discussions of interesting, mostly very recent, papers in economic theory and related fields. A link to it appears topmost in Sites I Like on the right side of this site. After an altogether too-long hiatus, I returned today to A Fine Theorem to find several items of interest that are new to me.
One of these items is the discussion of why economists (economic theorists, mostly) want to see proofs, not simulation results, to be convinced. As I understand the post, economists recognize that it is totally impossible to nail down economic theories on a few parameters that will be conceivably measured better and better over time. We economic theorists just have such a much harder time dealing with the complexity of a social system than a physicist who studies gravity that, as a profession, we have given up on coming up with a true model. Instead we build models upon a small, unified set of principles, a Foundation for all economics, that can give us ways of thinking consistently in many domains of application within economics. A paper that presents nice theorems and their proofs contributes to this enterprise, especially if the assumptions of the theorems do not depart much from the commonly-agreed Foundation. A paper that reports the results of a few thousand simulations hides the reasoning within a black box. Hence papers with proofs are more usable in building the economic theory pyramid and therefore more popular. Theorists stay away from trying to use the opaque (to them) reasoning that drives simulation results.
(I tried very hard not to make here the obvious analogy of economic theorizing as a pyramid scheme, and as you can see, I failed. If economic theory only aspires to indoctrinate researchers in THE ONE WAY to build models—and we know that econometric testing of the predictions of such models has not been a shining beacon of perfectibility in avoiding narrowminded, blid even, adherence to the Foundation—then we may be building totally useless beautifully constructed mathematical models, in fact harmful models, as they teach policy makers the wrong things. See below for another stab at this subject, and my total capitulation to have to devote a new post to it.)
I am not convinced by the black-box statement. (Confession: I still need to read seriously the paper that the post refers to, which can be found here (PDF).) If the authors of simulations publish their code, the only reason to say that the logic of the result is hidden is the inexperience of the reader with algorithms. Without going so far as to side with Wolfram, who said in his massive A New Kind of Science that all nature (all of it, including the living bits of nature and their behavior) is a bunch of algorithms, I would propose that we economic theorists need a better education in understanding algorithms.
As Leigh Tesfatsion says in one of her surveys of agent-based simulation (unfortunately I have no exact reference right now), every single run of a simulation is a mathematical proof, just with assumptions that appear narrower than we are used to (for instance, because they specify parameters numerically and exactly). A few thousand simulation runs can cover a decently-sized portion of the parameter space and then they are arguably as general, if not more, than a formal theorem on the same conceptual domain.
And things can be even worse for formal theorizing. It feels like we have been working with assumptions and techniques we are familiar with and which deliver consistently publishable papers based on assumptions from the Foundation. We are uncomfortable thinking about the possibility that this foundation is a quicksand. Computer simulations have a clear advantage here: flexibility. You can experiment with a simulation without feeling as restricted as by the straightjacket of mainstream economic theorizing. Simulations are more light-footed than Foundation-based theorizing, so they have a better chance to avoid sinking in the quicksand. (This is veering too far from the topic of the post, however, so I will have to write more in this vein in a subsequent post of mine.)
We could also imagine a source of unease about simulations in general, based on the complexity of the computer’s innards and even of the computer’s output. When the four-color theorem was proved by computer (not by simulation, but by exhaustive enumeration of the possibilities, as I understand it) a debate raged for a while among mathematicians. How could we accept a proof so gigantic that we cannot inspect it ourselves? I may be wrong, but mathematicians as a whole have resolved this debate by accepting that computers can give us acceptable proofs. We simply must accept the black-box problem; it comes with too many demonstrably solid and useful contributions of electronic computation to our lives. This is true even in cases where the box is kept black for commercial reasons, such as the exact Google search algorithm. A Google search these days may give us a few too many content-farm results that are not useful, but it still is an amazing tool whose operation is secret to most people.
As someone who has dabbled in agent-based computer simulations and intends to continue doing so, I appreciate the clarity this discussion gave me on the unpopularity of this kind of work among my colleagues, as well as the stimulation to think more about this issue. The unpopularity should not be overemphasized, however, as simulations are studied enough and published enough in economics to have their own volumes in the Handbook of Economics Series (Handbook of Computational Economics, in two volumes as of now, the second one on agent based computational economics, the kind of simulation I consider most useful for economists).