More Thoughts on Peer Review

Updated 9:22 p.m. August 20, 2006

It seems that academic peer review is my favorite rant on this Weblog.

• On June 10, 2006, I said that Numbers Chasing Sullies Science in reference to many insecure scholars in my discipline who prefer to publish several heaps of garbage rather than a single original idea.
• On May 7, 2006, I said that the Academic Publishing Model Is Broken in reference to how libraries were being pillaged by journals.
• On December 3, 2005, I said that the System Is Fundamentally Broken after I had a good paper rejected because too many good papers were submitted.
• And on November 24, 2005, I said that Some Days You Just Want to Know because I was spending more time anticipating reviewer complaints than concentrating on good science.

My favorite pet peeve made it back to the forefront Saturday when a review came back negatively. I can handle negative reviews; they're part of the process. But this one was extremely wrong minded.

In any given field, there are only a handful of journals that matter. And when a poorly trained editor rotates into the top spot, it is a painful few years.

Poor scientific training shows itself in an editor. You can mask poor training as a researcher. But as an editor, you are exposed. If you miss the point of being a scientist, you cannot be an editor. You will make bad decisions. You will let in finely polished bad ideas and fail to recognize good ideas when they fail to conform to some standard.

My career is far from perfect, but the best decision I ever made was to head to Bloomington, Indiana. Thanks to Annie Lang, I feel that I was well trained as a scientist. I care passionately about what it means to do science. I care about the process. And I respect the process.

And I hate ... HATE ... when the trappings of science get confused with science. Are the data good? Is the idea meaningful? This is the key. Yet too many poorly trained scientists -- and editor X -- seem to not understand. They appear to have gotten their Ph.D.s as the "walks like a duck" university.

Well science that walks like a duck can still be -- and often is --bad science. Polish is confused for quality, and good ideas can be lost if they do not fit into the square hole, as the square peg should. It takes good training to know the difference ... or to care enough to look for the difference.

When I received my first journal article to review, I asked Annie for advice. As always, she came through in the clutch. She said to separate the data from the idea. Are they good? Is it good science no matter how poorly written? Can the science be saved? And ever since, I have reviewed every article with this metric. Science deserves no less.

I wish that my coauthor and I had received the same treatment. Instead I spent today using the handful of good ideas in the review to improve the paper (there's some merit to the system), which will go out to a better journal tomorrow. And rest assured, this piece will find a home. The idea is good, even if the polish was a bit smudged.

In the meantime, we wait for another right-minded editor to rotate onto another journal.