Poor reasons for preprints & post-publication peer-review

Last week, I revived the blog with some reflections on open science. In particular, I went into the case for pre-prints and the problem with the academic publishing system. This week, I want to continue this thread by examining three common arguments for preprints: speed, feedback, and public access. I think that these arguments are often motivated in the wrong way. In their standard presentation, they are bad arguments for a good idea. By pointing out these perceived shortcoming, I hope that we can develop more convincing arguments for preprints. Or maybe methods of publication that are even better than the current approach to preprints.

These thoughts are not completely formed, and I am eager to refine them in follow up posts. As it stand, this is more of a hastily written rant.

Speed and precedent

But let us look at some concrete arguments for preprints. One common argument that is actually used for preprints: it gets your work out there faster. Peer review just takes too long.

On the face of it: how is this relevant? No scientist is sitting idle twiddling their thumbs as they wait for my latest result, they have other things to do. And while my results are in the slow publication pipeline, I have plenty of other projects to work on myself.[1] This blog post is one of them.

But yet those statements still feel true, so we need to tune in on the key word: ‘faster’. It is tempting — and probably intended — to take this as ‘faster than a counter-factual world where you didn’t post the preprint’. But what gives the statement the feeling of truth is the real world: ‘faster than those academics that don’t post their work as preprints.’ A way to get precedent in a needless race. Suddenly, it looks much less like a clear benefit and potentially more like trees wasting energy on building long trunks to get slightly above other trees for the limited sunlight. A great benefit to those who chop down trees for lumber, less useful to the trees themselves or to efficient use of resources.

To leave the colourful analogy: making scientists work faster across the board does not benefit scientists or science. It benefits funding agencies and publishing houses.

That being said, the focus on precedent in this poor point for preprints suggests a better argument. As I sketched above, it isn’t delay that is counting against journals but inequality in processing times. If all papers came out exactly 12 months after being submitted to a journal, regardless of the paper and journal then this would not be too much of a problem. The issue is that some papers and some journals delay you more than others. Or, worse yet, people actively seek to delay the publication of your paper so that they can claim precedent with their own work. Rushing your own papers through and delaying others is easier for more senior and famous academics, and this allows the inequality to be perpetuated. If we argue for preprints based on their ability to eliminate this inequality then I think that is a good argument.

However, plenty of other approaches could also achieve this: blogs, working papers, and conference. It is just a matter of deciding as a community how precedent should be counted. Better yet, we can try to question why we make such a big commotion about precedent and “ownership” of an idea. Is this a useful convention, or should we seek to eliminate it?

Fundamentally, much of these concerns come from a production line view of science. We view papers as the final “products” and everything done along the way is just a means to an end. But why? The process of comments and responses to comments is a useful one, it is through dialog that science is made. There is no reason to have a paper as the minimal unit of such dialog.

More importantly, it isn’t saying a lot (i.e. publishing many papers) that should matter, but saying something relevant that contributes to the conversation. We don’t want to be in a room with everyone shouting at the top of their lungs but not listening to anyone else. Always starting conversations and never continuing them, making disconnected proclamations without an overall narrative to cohere them.

Feedback and post-publication peer review

Let’s move on to second argument for preprints: feedback. In principal, anybody could read a preprint and provide feedback, either by email, blog post, a response paper, or many other methods. But then again, the same can be done with papers. What, in particular, makes preprints better than other approaches at gathering feedback? When most people aren’t using preprints then there is some novelty in their format that might slightly increase readership and urge for feedback.

But imagine that every paper first came out as a preprint: who would be reading them and giving feedback? This is actually one of the few upsides of the peer-review system. The editor serves as a motivated agent who is determined to convince two to three people to take the time to read and provide feedback on your paper. Since the editor is often more senior, she can harness her wider network (plus the name and institutional status of the journal) to promote your paper for feedback. As part of this deal, comes the implicit agreement that after you receive feedback from these reviewers, you will act on it. And the journal uses the only carrot it has — publication — to entice you. Unfortunately, our own competitiveness often gets the better of us and we co-opt this system and instead of providing useful feedback to improve the drafts we read, we take it on ourselves to be judges of if others should bother reading these papers or not. At times, it feels like this gate-keeper role is the carrot that editors have — beyond our general sense of duty to carry out review — in encouraging reviewers to participate in this system.

Does post-publication peer review change any of these dynamics? It certainly removes a big part of the gate-keeping function that has been built into reviewing. Although not all of it: if the reviews become anything like book reviews are now then major reviewing venues will still be able to set the reading agenda to some extent.

Public access

Now for a final argument for preprint: public access. This is a tricky one. Publicly posted preprints definitely eliminate (at least partially) one barrier to public access: paywalls. But are paywalls the things that are stopping people from reading scientific papers? I think that is a sentiment that only someone who has never written for the public would have. It takes a lot of work to write something that is compelling for the public, even for a niche market.

My blog, for example, is always just one url away for most people with internet access. Yet, to have a post be read (or I guess clicked into, since that is the actual statistics that I have access to) by more than 200 people requires a lot of effort. Effort that has nothing to do with paywalls. It requires finding and understanding my audience, it requires writing something they are interested in reading. This is something most academic work does not achieve, or even aspire to. The people we are writing for are usually other scientists or funding agencies. The problems we select from — although we like to imagine that we are the sources of our research programs — are those that are of interest to funding agencies. And although the interests of funding agencies and the interests of people more broadly sometimes align, they usually don’t.

The final requirement for being read, sadly, is already being popular. Preprints only amplify this. If you are a famous researcher at a famous institution (or you just add one of them to your paper) then it will suddenly be seen by many more people. Especially if you engage your self-promotion network.

None of this means that we shouldn’t post preprints. But we shouldn’t get on a high horse when we do so. And we should be aware of the problems that preprints can and cannot address, or ones that they might worsen. There are many ways forward, but given the length so far, I’ll have to save those discussions for another post.

Notes

  1. The exception to always having something to work on might be at the start of the production line. It can matter a lot for students to get their work out quickly, since they often cannot move on to their post-graduation jobs without the papers. As a personal example, I might have benefited greatly from preprints early in my career. Since I publicly posted the original 2013 draft of my complexity of evolutionary equilibria work, Dan Nichol was able to share the paper with Peter Jeavons and that has been part of the reason for why I am currently working with Pete.

    Unfortunately, the “might have” before “benefited” suggests that the story is not as clear cut of a case for preprints as this. I had actually sent Dan a draft of my paper before it was a preprint and he could share that as easily as the original preprint (since I would certainly approve of sharing it with Pete). This also highlights the point that most feedback comes not from people reaching out after having randomly encountered your preprint, but via direct requests through your social network. In the case of this evolutionary equilibria work, I got a lot of useful feedback from reaching out to various scientists with the paper via email. In fact, I think that the only feedback I got without first sending an email was from Marc Harper, but even that relied either on Jacob Scott reaching out to him first or my posts on the evolutionary game theory G+ community.

Advertisements

About Artem Kaznatcheev
From the Department of Computer Science at Oxford University and Department of Translational Hematology & Oncology Research at Cleveland Clinic, I marvel at the world through algorithmic lenses. My mind is drawn to evolutionary dynamics, theoretical computer science, mathematical oncology, computational learning theory, and philosophy of science. Previously I was at the Department of Integrated Mathematical Oncology at Moffitt Cancer Center, and the School of Computer Science and Department of Psychology at McGill University. In a past life, I worried about quantum queries at the Institute for Quantum Computing and Department of Combinatorics & Optimization at University of Waterloo and as a visitor to the Centre for Quantum Technologies at National University of Singapore. Meander with me on Google+ and Twitter.

One Response to Poor reasons for preprints & post-publication peer-review

  1. I do suspect that people use these arguments partly because they don’t feel comfortable saying “I want to drive a stake through the heart of the academic publishing industry”. Online preprint services are interesting because they could possibly develop into an alternative to the current journal system. As it is, preprints take over the overt function of publishing (communication), leaving the most important function (certifying academic output) to the traditional journals. I can’t help feeling there ought to be another way to certify output.

    I don’t think you can dismiss the speed/convenience argument quite so easily. Physics has a long tradition of preprint distribution going back even before the net. It can’t be entirely true that a 1 year delay on communication has no effect on the system. What if it was 10 years?

    r.e. the tree argument, this would be more convincing if there was a high cost to authors for uploading a preprint.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s