Reflections on open peer review

Last week I put up a manuscript for open peer review (It's academic publishing Jim, but not as we know it). In that post, I explained my reasons for doing this rather than going down the conventional (journal) academic publishing route. The review process, which I arbitrarily set at 14 days, is still running, but in this post I want to discuss my reflections on the process to date.

As I expected, reviews started to come in rapidly, 7 within the first 48 hours, then stopped equally rapidly. Internet attention is transitory, but in part this is a reflection of the fact that I drew the blog post to the attention of a number of people by email, inviting reviews. However, this pattern is typical for Internet content - a fast decay phase followed by a longer, slower tail (The Spread of Scientific Information: Insights from the Web Usage Statistics in PLoS Article-Level Metrics. (2011) PLoS ONE 6(5): e19917). There were no "spam" comments, which I had anticipated, and even though I had attempted to make clear in the post that anonymous reviews were entirely acceptable, all reviewers chose to identify themselves. Ironically, this is a concern, as while I suspect that reviewers consider named reviews to be somehow more "valid", I am worried that potential negative reviews are simply not posted, rather than being contributed anonymously. Interestingly, relatively few colleagues from my own institution, who I had alerted by email, contributed a review. In part this may be because they were wary of possible conflict of interests. When I repeat this process in future, I will simply post the article and reviewing guidelines online, without individual email notifications. Another concern for the future is the possibility that familiarity may breed indifference, limiting the number of reviews received.

I am grateful to Martin Weller for his additional comment on the review process:

"I tried to put my official reviewer hat on and review it as if I was doing a standard (blind) peer review. It may be that this is an inappropriate transfer of process, and instead I should adopt a different style for open, informal review. But we fall back on what we know. My review may be a bit harsh, but I was conscious that 'asking your mates to review' isn't really comparable to anonymous peer review at all. I might be far less likely to criticise a friend. My colleague Gill Kirkup maintains that anonymity in the peer review process is essential because it protects the reviewer, particularly a young reviewer who is reviewing a paper by someone eminent in the field. Of course, it also allows people to be ruder than they would be otherwise, and often to say incorrect judgements because there is no debate or come back.
So this may be a good way to get feedback on a paper, but would it equate to peer review? I don't think so, but then maybe it's a sufficient filter to allow publication and then post-review. It's also quite a brave thing to do and I suspect many colleagues might be reluctant to go this route. If you write a crap paper that gets rejected by a journal, only a handful of people have seen it - if you do it this way, potentially hundreds will."

A number of people commented on various forums that I was "brave" to expose my work in this way. It doesn't feel brave to me, it feels liberating, although possibly foolish. Specifically, it feels far less brave than exposing my work to non-transparent peer review. Maybe I've just had a run of bad luck, with editors taking capricious cost-based decisions to refuse to even send my work out for review. Entering that lottery - now that's brave (or foolish). Accepting that my peers may tell me that my work is of little or no value (and I have no doubts about the honesty of people who responded, so I feel confident they would), the whole process feels right to me. If some papers are slammed, then I either work on them further or abandon the concepts they contain.

So will I repeat this exercise in future? Most definitely - I already have a manuscript in mind, although this one is perhaps more of a technical report than an investigation. Will this become my sole future publication channel? No, not because I do not believe in it, but there are circumstances (collaboration with junior colleagues for example) where the alleged kudos attaching to publication in conventional journals is important for their careers. Should you repeat my experiment. That's up to you, but if you feel your circumstances permit, I would encourage you to try it for yourself. As I commented on Frances Bell's blog, "... I am not suggesting the approach I have taken is the “best” solution, nor necessarily appropriate for everyone – I have already identified a number of flaws. I do suggest that it is an improvement on the current model of closed, and frequently capricious, peer review. Open is good. If we support open access, why not open peer review?".


Update: Storify capture of Twitter discussion:

Tweet