Pinky Elephant (Kickstarter)
The Essence of RPGs (1): Children of TSR

The Broken Game of Peer Review

RejectedFor some time now, I've been complaining to anyone who will listen that double blind peer review of academic papers is a broken game. It’s broken, because reviewers are anonymous and unaccountable for their reviews, and anyone who has used the internet knows how dangerous it is to create anonymous and unaccountable people with the power to hurt others. Supposedly, we shouldn't worry about the potential abuse because academics are classy human beings. Except they're not, they’re all-too-human and everyone knows it – especially the academics! Indeed, if academics were virtuous people and not petty, narrow-minded, power-trip nerds, it wouldn’t be necessary for double blind peer review because we could trust virtuous academics to do their job well. But they don’t. And blind peer review helps them abuse their position.

In principle, a double blind review prevents bias by making the reviewers unaware of the people being reviewed, and protects the reviewers from potential backlash from the people they have reviewed. The result is that nobody is supposed to know whom anyone is, which is not a situation likely to bring out the best of humanity! In fact, as many have pointed out, it is usually easy to identify the authors of an anonymous paper, and certainly simple to separate research done at well-funded campuses from those at poorer (i.e. ‘foreign’ universities). This rather undermines the supposed benefits of blind peer review. Indeed, rather than eliminating bias, what double blind peer review does is allow reviewers full reign to exercise their personal biases by dismissing papers based solely upon their own prejudice or ignorance, without ever having to be held accountable for it. It is well documented that blind peer review blocks the publication of new research that runs against established dogmas, even when the new research is absolutely correct. (I wrote about this in The Mythology of Evolution).

I’d like to use a concrete example. I had a paper in review for two years at a fairly prestigious game studies periodical (if that isn’t an oxymoron...). When I eventually got the rejection, one reviewer had been dutiful in assessing the paper I had submitted. The other dismissed it out of hand based on his misreading of the paper, with a series of falsely construed judgements about its content. For instance, it was objected that Grand Theft Auto should not have its roots traced to Dungeons & Dragons, as I suggested. In such a situation where a peer review disagrees with an assertion, their obligation is to ask the author to back up the unsupported claim in revision. (The argument for this particular point, incidentally, appears in tomorrow’s post at ihobo, and only takes one sentence). You do not simply reject papers that you disagree with if you are a virtuous scholar. The trouble is – and this is the root of all evil here – there are too few virtuous scholars for blind peer review to be anything other than a nest of vipers. Anyone publishing outside of a small handful of close-knit disciplines will be able to share stories as bad as (or even worse than) what I am mentioning here. Just take a look at Rebecca Schuman’s Slate article from last year.

The fix is easy: make blind peer reviewers accountable. It’s very simple to do so. Each paper receives at least two peer reviews, so just make the other peer reviewer provide a rating for the quality of the second review. Then make aggregated reviewer scores publicly available every year via a central repository, like the ones already being used for reviews (e.g. easyChair for conferences). This means peer reviewers have an obligation to provide quality peer review, which at the moment they mostly do not do. I say this as a peer reviewer who has been frequently praised for the quality of my reviews. But then, the papers I am assigned are reviewed to a high standard because I review every paper in the expectation that I might have to face the author. Indeed, I attach to my peer reviews a statement that I consider blind peer review unethical, and will waive my anonymity if permitted. It never actually happens, but knowing that I am reviewing in a situation that even might put me face-to-face with those I am reviewing encourages me to be a virtuous scholar. Anonymous unaccountable reviewers, as anyone with even a passing experience of the internet can attest, will always be problematic.

In the absence of a change to the rules of blind peer review, it will remain fundamentally unethical and a broken game. We all deserve more than this. But who has the courage to turn against the status quo and help bring about the revolutionary changes to established academic practice that are desperately needed?

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

I'm going through my feedly starred surplus of things I haven't read yet. I totally agree. I found the title misleading -- You do in fact approve of double blind peer review but with accountability. There is an odd thing of automatically thinking double blind is good because it's used in medicine for drug trials etc. Very good idea to have an aggregate scoring system, things seem to be going in that direction with academia anyway.

Hi Michael,
Thanks for your comment - and apologies for the slow reply (May was my Spring-Summer Social Media Break).

The title isn't *entirely* misleading, as I think peer review that led to relationships between academics would be both more virtuous and more effective than blind peer review. Accountability is a minimum fix to blind review. But I see this as a blight on academic discourse that does not do what people claim it does outside of very narrow situations *that would equally be stopped in non-blind review*.

I also agree with your allusion to drug trials etc. - double blind is being predicated because of the prevalence of a postivistic mythos that has become uncoupled to the actuality of peer review. I find the situation intolerable, but others seem content to just accept the status quo.

I find myself in this situation a lot! :)

All the best,

Chris.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Your Information

(Name is required. Email address will not be displayed with the comment.)