In my short blogging career I have been challenged, on more than one occasion, to submit my writing to peer-reviewed academic journals. My response has been to point out that my work undergoes peer review the second I post it online and that interdisciplinary and multi-disciplinary science is not well served by the traditional academic peer review process. The sad reality is that academic peer-reviewed journals are not terribly relevant in the non-academic world. As I will discuss in this blog post, peer-reviewed academic journals serve a useful function in highly-specialized fields but they quickly lose their relevance for broader topics. As a consequence, in some fields peer-reviewed academic journals serve simply as a means by which academics can keep score (for their C.Vs) and to get, or block, promotions. That being said there is a constituency, mostly academic, that remains firmly convinced that peer-reviewed academic journals should form the primary basis for the development of policy in the environmental and climate fields. I hope this post will help correct that misguided impression.

In my opening paragraph I make a pretty bold statement: that the academic peer-reviewed press serves a marginal role in real-world policy decisions. To substantiate my claim let’s start with a few self-evident truths of the peer-reviewed academic press.

• The peer-reviewed academic press is slow to present results. The publication process often stretches on for months and can go for years. This can leave it well behind the decision curve.
• The peer-reviewed academic press can be subject to institutional and professional gate-keeping that can alienate stakeholders.
• Accessing peer-reviewed articles is typically so expensive that most non-acdemics only read the freely-accessible abstracts and don’t delve into the actual details of the articles that are hidden behind paywalls.

Now I am pretty certain that no one will challenge my first two points but some might be surprised by my third. Those people are likely academics who have never really had to consider how much it can cost to keep current with academic publications. Most academics are fairly unaware of the cost of subscription or access fees to academic articles. Consider that access to an article from “Science” may cost you $30 FOR A SINGLE DAY. Given those costs most non-academics simply do the inexpensive thing and skip reading the full papers altogether safe in the knowledge that if a paper is sufficiently important someone else will summarize its contents or a pre-print will be made available.

My major complaint about the peer-reviewed academic press is that it only really works well when the journal covers a very distinct body of knowledge. This is because only then will the peer-reviewers be capable of carrying out an effective peer review. The biggest problem with the peer-review system is that the peer reviewers are limited in number, time and experience and as such normally a paper only sees two or three reviewers. These reviewers are picked not only for their expertise but also on their willingness to serve. Typically what this means is that the job is dominated by the very young academics (who have the enthusiasm and want to build up experience) and the very old academics (who have the time). Mid-career academics often lack the time to do much peer-review thanks to their teaching and administrative responsibilities. Most try and make time but they cannot address every manuscript that comes across their desk.

Let’s be clear here, peer reviewers serve an important, but often misunderstood multi-faceted role in the scientific endeavour. Firstly they are there to confirm that the science is done in an appropriate manner. In this they are conducting a quality assurance/quality control (QA/QC) role within the system. They look at how the work was done to identify errors in methodology: were appropriate statistical tests applied? Was the sample cooled sufficiently…that sort of thing. Peer reviewers will often suggest additional/different analyses to confirm the validity of results. This task is only possible if the peer reviewer is extremely familiar with the field under study.

Peer reviewers also serve an editorial role. They are responsible for suggesting changes to make the work more comprehensible. That is the sort of task that can be accomplished by any good reviewer whether they understand a topic or not.

The final role of the peer reviewer is that of gate-keeper, to identify if the science presented in the paper is novel and interesting enough to warrant space in the journal. A reviewer might say that an article is beautifully written, including all the right tests, but that it is not a good fit for the journal in question. This final role is a legitimate one within the system, but as members of the climate science community can tell you, it also risks problems if one part of the community chooses to block the publication of research from those with whom they differ.

You are probably wondering when I am going to get to my point? Well here it is: once you leave a peer reviewer’s particular area of expertise they cease to provide a reasonable value for money. At that point they simply become a highly-credentialed copy editor. In a field like climate change or renewable energy policy, that is very broad in content/context, it is simply not possible to find reviewers who can do a reasonable job of effectively peer reviewing papers. You don’t believe me? Well let me give you an example.

One of my recent bugbears at this blog has been Dr. Mark Jacobson’s 100% Wind, Water and Sunlight series with the latest article being 100% Clean and Renewable Wind, Water, and Sunlight (WWS) All-Sector Energy Roadmaps for 139 Countries of the World (called 100% WWS World hereafter). This paper discusses almost a dozen renewable energy concepts and provides plans that quite literally span the globe. As a consequence no two or three peer reviewers could conceivably do a comprehensive job of reviewing this paper and a lot of things were simply missed by the reviewers. As an example, most North Americans know little about Scandinavia, imagining it is all mountains and fjords. So when Dr. Jacobson suggested that Finland could store huge amounts of energy in the form of pumped hydro storage that probably sounded pretty reasonable to the reviewers (and frankly to me when I read the paper). It took a Finn to point out that Finland, unlike the rest of Scandinavia, is an exceedingly flat country. Pumped hydro needs abrupt changes in topography to be efficient and is not practical in flatlands. So the 100% WWS World plan fails spectacularly in that respect for its analysis of Finland. Similarly, a look at Wikipedia would confirm to Dr. Jacobson’s reviewers that Canada’s coastline is vast (over 200,00 km) but what the reviewers clearly didn’t recognize is that the majority of that coastline is in the north where weather and ice preclude the construction of off-shore wind turbines. Thus, when the paper calculated how much offshore wind potential was available it missed the fact that much of the area described is not practical for offshore wind energy generation. Similarly, it is clear that the peer reviewers didn’t know that the Pacific Continental Shelf pretty much abuts much of the West Coast of Canada. The sea floor drops precipitously very close to the coast and as such over much of the coast it would not be affordable to install an offshore wind platform as the foundation of such installations would need to be 100s of meters deep.

What this all means is that when someone assures me that Dr. Jacobson’s work is correct and relevant “because it underwent peer review” I take that with a heavy dose of salt. I have read almost a dozen critiques of the work (and authored 4 myself) which challenge topics ranging from how it handles transmission lines to how it deals with individual countries. In this case the peer-review process failed because the topic was simply too broad for any small group of peer reviewers to complete effectively. It is an object lesson in the limitations of the peer review process.

Going back to our main topic, in our modern era a blogger doesn’t need to publish in peer-reviewed academic journals to have an effect. Bloggers can build a following and influence policy simply through the act of consistently producing good, well-referenced and well-argued work. A well-respected blogger, like Steve McIntyre (at Climate Audit), can instantly reach and influence a targeted audience of thousands. Meanwhile a typical, peer-reviewed article can take hundreds of hours wending its way through the publishing process, cost thousands of dollars to produce and is likely to be read completely by no more than 10 people.

Once a blog post is presented online it has the ability to be reviewed by the hundreds of readers who encounter it. Moreover, unlike a published article in a journal, corrections and addendums in blogs can be made pretty much right away. I am quick to fix errors in my blog because that allows me to maintain my credibility. A faulty journal article, on the other hand, is exceedingly hard to fix. Sometimes it seems like journals actually discourage corrections as they don’t “constitute original content” so only the most egregious errors get corrected or papers retracted.

In a similar vein, the peer review process is not a magical or proprietary process limited to the academic press. Think tanks and policy foundations can conduct research which they can subsequently submit for peer review from knowledgeable experts before publication. Thus a study on ocean acidification by Patrick Moore can be published by the Frontier Centre for Public Policy while rightfully claiming that the study was peer-reviewed. Now to be clear I know nothing about the FCPP but given the prominence of this paper the absence of any legitimate complaints/corrections since this report was published leads me to believe that I can take what I read as factual. Happily, given the scope and breadth of the world-wide net I know that if this statement is wrong I will receive many corrections (which I will share if sufficiently relevant) that may change my view of that report.

Going back to the premise of this post. I don’t plan to submit my writing to a peer-reviewed academic journal anytime soon as I do not see any particular value in doing so.  I know that I will get feed-back once I post my thoughts on my blog and my professional status does not hinge on such publications. Most importantly, my blog stats and link hit statistics re-assure me that my thoughts are being widely read and shared which is really all I am looking for with this blog.

Author’s note: a number of readers have questioned the lack of peer-reviewed output from my graduate work. My graduate work was interdisciplinary in nature and involved developing systems to allow data collected by governmental scientists to be evaluated for its reliability and quality; stored in information systems; and made available for subsequent re-use by other researchers. Upon graduation I went immediately to the private sector and saw no real value in writing up my research. The reason for my reluctance was that the output of my research was already being used on a daily basis as a component of the information systems used by our provincial and federal governments and was being widely shared by those governments (including through workshops that I was asked to moderate).

As for the practical applications of my research, well it was almost immediately applied at my current employer and has become a standard in our industry for the last decade. I cannot imagine a better demonstration of the value of my research than the fact that our clients insist that our competitors conduct data analysis in the manner suggested by my research and in doing so our clients have done more to publicize my research than any academic publication might ever have done.