For some time now, I've been complaining to anyone who will listen that double blind peer review of academic papers is a broken game. It’s broken, because reviewers are anonymous and unaccountable for their reviews, and anyone who has used the internet knows how dangerous it is to create anonymous and unaccountable people with the power to hurt others. Supposedly, we shouldn't worry about the potential abuse because academics are classy human beings. Except they're not, they’re all-too-human and everyone knows it – especially the academics! Indeed, if academics were virtuous people and not petty, narrow-minded, power-trip nerds, it wouldn’t be necessary for double blind peer review because we could trust virtuous academics to do their job well. But they don’t. And blind peer review helps them abuse their position.
In principle, a double blind review prevents bias by making the reviewers unaware of the people being reviewed, and protects the reviewers from potential backlash from the people they have reviewed. The result is that nobody is supposed to know whom anyone is, which is not a situation likely to bring out the best of humanity! In fact, as many have pointed out, it is usually easy to identify the authors of an anonymous paper, and certainly simple to separate research done at well-funded campuses from those at poorer (i.e. ‘foreign’ universities). This rather undermines the supposed benefits of blind peer review. Indeed, rather than eliminating bias, what double blind peer review does is allow reviewers full reign to exercise their personal biases by dismissing papers based solely upon their own prejudice or ignorance, without ever having to be held accountable for it. It is well documented that blind peer review blocks the publication of new research that runs against established dogmas, even when the new research is absolutely correct. (I wrote about this in The Mythology of Evolution).
I’d like to use a concrete example. I had a paper in review for two years at a fairly prestigious game studies periodical (if that isn’t an oxymoron...). When I eventually got the rejection, one reviewer had been dutiful in assessing the paper I had submitted. The other dismissed it out of hand based on his misreading of the paper, with a series of falsely construed judgements about its content. For instance, it was objected that Grand Theft Auto should not have its roots traced to Dungeons & Dragons, as I suggested. In such a situation where a peer review disagrees with an assertion, their obligation is to ask the author to back up the unsupported claim in revision. (The argument for this particular point, incidentally, appears in tomorrow’s post at ihobo, and only takes one sentence). You do not simply reject papers that you disagree with if you are a virtuous scholar. The trouble is – and this is the root of all evil here – there are too few virtuous scholars for blind peer review to be anything other than a nest of vipers. Anyone publishing outside of a small handful of close-knit disciplines will be able to share stories as bad as (or even worse than) what I am mentioning here. Just take a look at Rebecca Schuman’s Slate article from last year.
The fix is easy: make blind peer reviewers accountable. It’s very simple to do so. Each paper receives at least two peer reviews, so just make the other peer reviewer provide a rating for the quality of the second review. Then make aggregated reviewer scores publicly available every year via a central repository, like the ones already being used for reviews (e.g. easyChair for conferences). This means peer reviewers have an obligation to provide quality peer review, which at the moment they mostly do not do. I say this as a peer reviewer who has been frequently praised for the quality of my reviews. But then, the papers I am assigned are reviewed to a high standard because I review every paper in the expectation that I might have to face the author. Indeed, I attach to my peer reviews a statement that I consider blind peer review unethical, and will waive my anonymity if permitted. It never actually happens, but knowing that I am reviewing in a situation that even might put me face-to-face with those I am reviewing encourages me to be a virtuous scholar. Anonymous unaccountable reviewers, as anyone with even a passing experience of the internet can attest, will always be problematic.
In the absence of a change to the rules of blind peer review, it will remain fundamentally unethical and a broken game. We all deserve more than this. But who has the courage to turn against the status quo and help bring about the revolutionary changes to established academic practice that are desperately needed?