Recently, I’ve been spending a lot of time with the peer review process.\u00a0 I’ve been revising a couple of papers in response to comments from the referees, and at roughly the same time I’ve been acting as referee for a couple of articles.<\/p>\n
Some journals send each article just to a single referee, while others use multiple referees per article.\u00a0 In the latter case the author and the editor can get some idea of how well the process works by how consistent the different referee reports are with each other.\u00a0 In particular, if one referee says that a paper has a fundamental flaw that absolutely must be fixed, while the other notices nothing wrong, it’s natural to wonder what’s going on.<\/p>\n
In fact, by keeping track of how often this sort of thing occurs, you can estimate how good referees are at spotting problems and hence how well the peer review process works. Let’s consider just\u00a0 problems that are so severe as to prevent acceptance of the paper (ignoring minor corrections and suggestions, which are often a big part of referees’ reports).\u00a0 Suppose that the typical referee has a probability p of finding each such problem.\u00a0 If the journal sends the paper to two referees, then the following things might happen:<\/p>\n
A journal editor, or an author who’s written a bunch of papers, can easily estimate the ratio of the last two options: when one referee finds a problem, how often does the other referee find the same problem?\u00a0 From that ratio you can solve for p and know how well the referees are doing.<\/p>\n
In my experience, I’d say that it’s at least as common for just one referee to find a problem as it is for both referees to find it. That means that the typical referee has at best a 2\/3 chance of finding any given problem.\u00a0 And that means that the probability of both referees missing a problem (if there is one) is at least (1\/3)2<\/sup>, or 1\/9.\u00a0 I’ll leave it up to you to decide whether you think that’s a good or bad success rate.<\/p>\n The main thing that made me think of this is that in one referee report I sent off recently, I pointed to what I think is a major, must-fix error, and I’m willing to bet the other referee won’t mention it.\u00a0 That’s not because the other referee is a slacker — I don’t know who it is, or even whether this particular journal uses multiple referees.\u00a0 It’s because the paper happens to have a problem in an area that I’m unusually picky about. It’s bad luck for the authors that they drew me as a referee for this paper.\u00a0 (But, assuming that I’m right to be picky about this issue — which naturally I think I am — it’s good for the community as a whole.)<\/p>\n This sort of calculation shows up in other places, by the way.\u00a0 I first heard of something like this when my friend Max Tegmark<\/a> was finishing his Ph.D. dissertation.\u00a0 He got two of us to read the same chapter and find typos.\u00a0 By counting the number of typos we each found, and the number that both of us found, he worked out how many undiscovered typos there must be.\u00a0 I don’t remember the number, but it wasn’t small.\u00a0 On a less trivial level, I think that wildlife biologists use essentially this technique to assess how efficient their population counting methods are: If you know how many bears you found, and how many times you found the same bear twice, you can work out how many total bears there are.<\/p>\n","protected":false},"excerpt":{"rendered":" Recently, I’ve been spending a lot of time with the peer review process.\u00a0 I’ve been revising a couple of papers in response to comments from the referees, and at roughly the same time I’ve been acting as referee for a couple of articles. Some journals send each article just to a single referee, while others … Continue reading Does peer review work?<\/span><\/a><\/p>\n","protected":false},"author":12,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-100","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts\/100","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/comments?post=100"}],"version-history":[{"count":0,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/posts\/100\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/media?parent=100"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/categories?post=100"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.richmond.edu\/physicsbunn\/wp-json\/wp\/v2\/tags?post=100"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}