‘Twixt Peer Reviewed and Bits on the Web

What to cite when the research frontier is moving fast…

One of the things I do in my economics classes is assign term paper assignments, which is not terribly common in undergraduate economics courses (problem sets and exams are more typical). For many students, this will constitute the first economics paper they’ve ever written. Yet this type of assignment requires that students become more discerning regarding the material they take seriously when doing their own research. It seems that commentators on this blog might also benefit from some thoughts on what constitutes useful data sources and plausible economics research, especially given the lags in peer review publication that seemingly give some people license to cite any ol’ thing on the web. Here are my “guidelines” for my students:

  1. Don’t willy-nilly go researching on the web. With low costs of dissemination, anybody can write anything online.
  2. Peer reviewed publications in journals on average have had at least one level of screening; they are not “correct”, or necessarily “unbiased”, but at least they have had the benefit of some critique.
  3. If there are no published/forthcoming works, due to the rapidity of developments, then working papers or papers under submission (like a preprint) are a source. Not all working papers are created equally, though.
  4. Organized working paper series often have an internal review process, such as those from central bank and international organization research departments.
  5. Research associations (NBER, CEPR) don’t typically review the working papers, but there is an implicit screening in that only members can circulate papers. (By the way, this highlights the distinction between “publish” versus “circulate” – publishing in a peer reviewed journal requires – well – peer review).
  6. Relying on limited-membership publication could restrict one to “group-think” bias; for the association series I mentioned, I don’t think that that is a big worry – unless you think “group-think” encompasses the economic mainstream (i.e., if you’re looking for Marxian, Sraffian, or Austrian analysis, I admit they’re unlikely to show up).
  7. None of the foregoing says ignore an unaffiliated working paper. If one is unable to independently critique, it’s useful to know if the paper has been subjected to comment at conferences/seminars. There’s some in-group bias there since invitations depend on networks.
  8. BUT, the development of the internet means one can access many more critical analyses than in earlier times. For those with access to university libraries, the successor to the Social Sciences Citation Index, the Web of Science, is key. For those without, Google Scholar is a good substitute.

None of the foregoing guarantees one identifies the “right” research. But it reduces the likelihood of getting the random piece of research that just happens to validate one’s prior views – you know, like 22% is the relevant herd immunity threshold – or (more durable), income tax cuts typically pay for themselves.

Meta-analyses – quantitative analyses of estimates – can also signal when you’re taking a particularly iconoclastic view. If for example you’re citing an elasticity that is the most extreme out of 300 estimates, well, you might have reason to wonder. (See example re: minimum wage here.)

By the way, unbiased (once again, not necessarily correct) economic analysis and projections for the US are available at CBO, and unbiased recounting of developments and anaysis is now available at the (Library of Congress’s) Congressional Research Service — the Congress’s think tank.