How to revise research papers after receiving harsh reviews

I just learned of Daniel Lemire’s blog from a post by Noam Nisan on Google+. The following post from Lemire’s blog is so good I had to fight the temptation to quote its entirety here. I heartily recommend it, and the whole blog, to you, gentle reader. Do not waste time; go visit it now.

How to revise research papers after receiving harsh reviews:

Whether you submit your work scientific journal or just post it on a blog, you can expect to receive harsh criticism from time to time. Sometimes you are facing arrogant or ignorant readers. Other times, your work is genuinely flawed. My own work is frequently flawed, as you know if you read this blog.

Over time, I have learned that even if the reviewer is wrong, spending time to careful respond can be tremendously useful. If you are 100% correct, then you get to build up your confidence and can later answer similar criticism hastily. Very often, however, you did not do everything perfectly. Maybe your arguments and data are correct, but you might have presented them better.

There are specific strategies to deal with harsh reviews:

(Snipped. Excellent practical advice is here, but I really think you should go read the original post!)

(Via Daniel Lemire’s blog)

Should economic theorists shun simulations?

The blog A Fine Theorem is an excellent source of thoughtful discussions of interesting, mostly very recent, papers in economic theory and related fields. A link to it appears topmost in Sites I Like on the right side of this site. After an altogether too-long hiatus, I returned today to A Fine Theorem to find several items of interest that are new to me.

One of these items is the discussion of why economists (economic theorists, mostly) want to see proofs, not simulation results, to be convinced. As I understand the post, economists recognize that it is totally impossible to nail down economic theories on a few parameters that will be conceivably measured better and better over time. We economic theorists just have such a much harder time dealing with the complexity of a social system than a physicist who studies gravity that, as a profession, we have given up on coming up with a true model. Instead we build models upon a small, unified set of principles, a Foundation for all economics, that can give us ways of thinking consistently in many domains of application within economics. A paper that presents nice theorems and their proofs contributes to this enterprise, especially if the assumptions of the theorems do not depart much from the commonly-agreed Foundation. A paper that reports the results of a few thousand simulations hides the reasoning within a black box. Hence papers with proofs are more usable in building the economic theory pyramid and therefore more popular. Theorists stay away from trying to use the opaque (to them) reasoning that drives simulation results.

(I tried very hard not to make here the obvious analogy of economic theorizing as a pyramid scheme, and as you can see, I failed. If economic theory only aspires to indoctrinate researchers in THE ONE WAY to build models—and we know that econometric testing of the predictions of such models has not been a shining beacon of perfectibility in avoiding narrowminded, blid even, adherence to the Foundation—then we may be building totally useless beautifully constructed mathematical models, in fact harmful models, as they teach policy makers the wrong things. See below for another stab at this subject, and my total capitulation to have to devote a new post to it.)

I am not convinced by the black-box statement. (Confession: I still need to read seriously the paper that the post refers to, which can be found here (PDF).) If the authors of simulations publish their code, the only reason to say that the logic of the result is hidden is the inexperience of the reader with algorithms. Without going so far as to side with Wolfram, who said in his massive A New Kind of Science that all nature (all of it, including the living bits of nature and their behavior) is a bunch of algorithms, I would propose that we economic theorists need a better education in understanding algorithms.

As Leigh Tesfatsion says in one of her surveys of agent-based simulation (unfortunately I have no exact reference right now), every single run of a simulation is a mathematical proof, just with assumptions that appear narrower than we are used to (for instance, because they specify parameters numerically and exactly). A few thousand simulation runs can cover a decently-sized portion of the parameter space and then they are arguably as general, if not more, than a formal theorem on the same conceptual domain.

And things can be even worse for formal theorizing. It feels like we have been working with assumptions and techniques we are familiar with and which deliver consistently publishable papers based on assumptions from the Foundation. We are uncomfortable thinking about the possibility that this foundation is a quicksand. Computer simulations have a clear advantage here: flexibility. You can experiment with a simulation without feeling as restricted as by the straightjacket of mainstream economic theorizing. Simulations are more light-footed than Foundation-based theorizing, so they have a better chance to avoid sinking in the quicksand. (This is veering too far from the topic of the post, however, so I will have to write more in this vein in a subsequent post of mine.)

We could also imagine a source of unease about simulations in general, based on the complexity of the computer’s innards and even of the computer’s output. When the four-color theorem was proved by computer (not by simulation, but by exhaustive enumeration of the possibilities, as I understand it) a debate raged for a while among mathematicians. How could we accept a proof so gigantic that we cannot inspect it ourselves? I may be wrong, but mathematicians as a whole have resolved this debate by accepting that computers can give us acceptable proofs. We simply must accept the black-box problem; it comes with too many demonstrably solid and useful contributions of electronic computation to our lives. This is true even in cases where the box is kept black for commercial reasons, such as the exact Google search algorithm. A Google search these days may give us a few too many content-farm results that are not useful, but it still is an amazing tool whose operation is secret to most people.

As someone who has dabbled in agent-based computer simulations and intends to continue doing so, I appreciate the clarity this discussion gave me on the unpopularity of this kind of work among my colleagues, as well as the stimulation to think more about this issue. The unpopularity should not be overemphasized, however, as simulations are studied enough and published enough in economics to have their own volumes in the Handbook of Economics Series (Handbook of Computational Economics, in two volumes as of now, the second one on agent based computational economics, the kind of simulation I consider most useful for economists).

Brad DeLong Says Economic Theory Does Not Exist

In a column that ran today in the Project Syndicate, Brad DeLong said this:

One of the dirty secrets of economics is that there is no such thing as “economic theory.” There is simply no set of bedrock principles on which one can base calculations that illuminate real-world economic outcomes. We should bear in mind this constraint on economic knowledge as the global drive for fiscal austerity shifts into top gear.

Unlike economists, biologists, for example, know that every cell functions according to instructions for protein synthesis encoded in its DNA. Chemists begin with what the Heisenberg and Pauli principles, plus the three-dimensionality of space, tell us about stable electron configurations. Physicists start with the four fundamental forces of nature.

Economists have none of that. The “economic principles” underpinning their theories are a fraud – not fundamental truths but mere knobs that are twiddled and tuned so that the “right” conclusions come out of the analysis.

I am of two minds about this. I certainly feel that the beautiful economic theories that have been created with the help of some serious mathematics in the last few decades have yielded valuable insights. Yet on the other hand, these insights are far from telling us unambiguously important things about economic reality and from giving us good recipes for economic policy. I don’t even feel we understand, as economists, how such a basic thing such as economic trade can emerge, based on trust among people. So we have ended up with “theory” as a plaything of political interests. For such reasons, I share DeLong’s frustration. Yes, Paul Seabright has written the wonderful book The Company of Strangers, but still we don’t have a good grasp of the fundamentals of economic trade at the level of really basic theory! Naturally, I am trying to do something about this in ongoing research with my long-time collaborator, Rob Gilles, or I would not be justified in airing my complaints on this theoretical lacuna.

But is it really beauty and some insights of doubtful empirical relevance versus abandoning all hope of having an economic theory? I sincerely hope not. We have, in game theory, the mathematical theory of networks, and in the technology of simulation, some tools that should allow us to build a better theory. One that, although it will always be subject to criticism and will always create the longing for something better, will not be so easily dismissed as nonexistent by a leading economist.

I understand that DeLong is concerned about macroeconomics and one can read his nonexistence claim in terms of macroeconomic theory. But that is a cop-out. If we had a good theory of economic fundamentals, we would be able to build an, at least existent, theory of macroeconomics, by DeLong’s standards. So I addressed my remark here to basic economic theory, not its macroeconomic special case.

Bacteria that channel Elinor Ostrom

This blog post, from Not Exactly Rocket Science, caught my attention. I blogged about it in my more general-audience blog. Here I want to elaborate a little bit on the connection with Elinor Ostrom’s work. Ostrom studies how various human societies have evolved mechanisms to manage common property resources. She shows how in many cases these mechanisms lead to much better outcomes for the users of the commons than what the plain old game theoretic foundation of the “tragedy of the commons” that we teach undergraduate students (and graduate students, too) leads us to believe. I like how the study of bacteria I started this post with shows the same idea operating via chemical signals and evolutionary pressures in populations of bacteria. Surely humans can continue to evolve useful mechanisms to manage their own common property resource problems better, if bacteria can. Note that the bacteria in this study did not have a uniformly good solution: only if the population of the colony gets large enough does the evolutionary advantage of cheaters evaporate. But it does evaporate, eventually.