On Peer Review

Responding to Terry Anderson.

Interesting commentary (I would webcam comment but we have the TV on playing the Olympics in the background).

I want to focus on this: "these annotations and references are all from a look in the rear view mirror and very rarely result in improvements to the work."

There are two stances you can take with respect to commentary (not necessary mutually exclusive, but which creates the divide between peer review and my own style of formal aggregation postings):

- you can intend the comment to create an improvement in the current work, or

- you can intend the comment to create an improvement in a future work

This is a difference in attitude toward knowledge and writing, and it has wider ramifications in scholarship as a whole.

First, the attitude. When you are focused on improving a single work, you are approaching knowledge and scholarship as something that can be pinned down and definitively recorded.

Now by that I don't mean you can achieve perfection, or any such thing. But this approach can be contrasted with one that takes knowledge and scholarship to be more like a stream of (social) consciousness, with each article an artifact of the moment.

I don't think of knowledge and scholarship as static; I think of them as fluid, and therefore to me it seems counterintuitive to attempt to capture a paper and fix it as a definitive statement of fact (or knowledge, or findings, or however you want to represent it).

Second, the wider ramifications. "The author also gains the lesson learned and experience (hard though it may feel) of how NOT to write a scholarly article." So there is an educational value to reviewer comments, and this forms a significant part of the value of the review.

But it is very inefficient to address these comments to a single recipient. One of the comments I make in one of my posts about an article can reach thousands of people. If I offer suggestions about how to write a scholarly article (as I sometimes do) then this lesson is learned my many more people at a time.

But there is another, even more important, factor at work here. These reviewers do not merely educate people in what counts as scholarly work, they *define* what counts as scholarly. This definition is part and parcel of the science or discipline in question. And yet, if the reviews are never seen publicly, this most important part of a science - defining what counts as knowledge - takes place in secret.

And because it can be, it is abused. Rivals reject papers for no good reasons. Reviewers suggest that certain citations are needed - namely, of their own work (or if they are more clever, work that in turn cites their own work). Standards of evidence may be looser if it supports a certain perspective (usually, the need for more research) and an entire discipline (like, say, education) may see its standards slip.

The difference between me, and an academic reviewer, is that I am held accountable for every harsh word, every appeal to an academic standard, every suggestion of a missing reference, every appeal to theoretical support. I can't secretly lobby for a certain theory, undercut an opponent or competitor, bias the evidential basis for a proposition, or any of the many other things that can and do happen in peer review.

For me, these three major factors are a conclusive argument against the practice of peer review. The post-publication review represents a more appropriate epistemological stance, is pedagogically more efficient, and is academically more honest.