Would you want your best ideas openly scrutinized?
Would you want your most cherished ideas openly scrutinised by your colleagues? Scientists do. In fact, peer review is an essential part of the scientific method. But that still doesn’t make it nice.
Peer-review is regarded by many as an indispensable, if sometimes unwieldy, cog of the science machine. It’s what makes science ‘go’. But to outsiders it can seem a bizarre process.
Why? Because it means giving the game away. Imagine you are a scientist and your grant application or new paper is being peer-reviewed by others in your field. Essentially this means inviting potential competitors to wade about in your cherished scientific ideas, searching for difficulties and rooting out flaws. Think about this, the people you are trying to beat, up to their armpits in your precious, hard-earned data.
This kind of thing just doesn’t happen anywhere else. Imagine mobile phone companies demanding full-access to each other’s boardrooms. Or competing supermarkets inviting each other to pore over business plans, and offer ‘helpful’ criticism.
Of course it’s not quite as bonkers as this. But you get the general idea. And scientists have to do this on a routine basis. Research works by having new theories, ideas, and findings put through the wringer. To be tested, debated, and tested again. It sounds uncomfortable. Indeed, it often is. And it’s not without its faults.
Even with checks in place, the peer-review process can still be inefficient and sometimes even subverted for ulterior motives, whether deliberately or not. Scientists are only human, after all. Much recent discussion revolves around how peer-review could be improved, or in some circumstances, done away with altogether. But it’s currently the best way we have of separating the most promising from the mundane, the flawed from the real, of helping the best science shine through.
Worldwide Cancer Research funds around 50 new research projects every year. It’s a good amount of research, but its also less than 10 per cent of the total number of applications we receive, and demand is increasing. We’d like to fund more. But as usual it comes down to cold hard cash. We just don’t have the money to fund everything we’d like to. In the mean-time our peer-review process makes sure we can sift out the best research applications to fund.
Effective peer-review takes a long time and a lot of effort. At Worldwide Cancer Research a single grant application can take as long as 6 months for to make it through our ‘gold-standard’ peer-review process. Each application is reviewed by four or five separate scientists, who are all working for free. This includes two scientists from our Scientific Committee and then the most promising applications are sent to two or three other experts around the world to really go through things with a fine toothed comb.
"Scientific funding committees rarely have sufficient in depth knowledge to accurately assess and compare the relative merits of all the different research applications. Hence, peer reviewers around the world are essential to judge the potential importance and likelihood of success of a research project. As a result, their views and suggestions are much appreciated by funding panels and applicants alike. Applicants would have much less faith in the system as a whole, and the rigour of a particular funder, if proposals did not receive specialist peer review." Says Professor Andrew Fry, Director of Research at the University of Leicester and Worldwide Cancer Research Scientific Committee member.
Because our remit is so broad, we have a particularly large Dragons’ Den of stellar scientists on our Scientific Committee (our Funding Committee) to review our applications. If you think the likes of Duncan Bannatyne and Deborah Meaden are a tough crowd, imagine pitching to 24 of the top science brains in Europe, all out to decide whether your idea is worth that precious funding
Swap the starkly-lit warehouse for a modest hotel meeting room in Edinburgh, imagine the pitches made in writing instead of person, and you begin to get the idea.
Admittedly most scientists don’t look quite so fearsome as the Dragons. But they are just as deadly serious when it comes down to business. They even ask virtually the same questions. Is this research new? Is it needed? Is anyone else doing it? Is the applicant able to do what they say they will- that is, are they properly skilled-up? Are they asking for the right amount of money? And perhaps most importantly, could this research really make a difference? But unlike the real Dragons, scientific peer-reviewers don’t stand to financially gain from all the review work they put in. The scientist must often perform these duties over and above their own day-to-day research activities, and that can mean a lot of late nights reviewing papers.
So what’s it like to have your work scrutinised like this in such excruciating detail? During my own time in research I remember waiting for reviewer comments with bated breath. Did the reviewer agree that this research was as exciting as I thought it was? A review can make or break your chances of funding or publication. A ‘good’ review meant celebrations. A ‘bad’ review commiserations and back to the drawing board.
I might not have always agreed with the reviewers’ points, sometimes I found them downright infuriating. But after I’d calmed down, I usually reluctantly agreed that the points made were valid, and necessary. Ultimately, I have to admit, they made me a better scientist.
“In general, studies are almost always improved by critical input by other scientists.” Says Paul Coffer, Professor of Cell Biology at UMC Utrecht and Worldwide Cancer Research Scientific Committee member. “Peer-review is not just important, it is essential. Without it we would have the scientific literature overflowing with poorly performed research and uninterpretable clinical trials. A solid peer-review system allows us to see the wood from the trees and moves science forward.”
And that’s just the point. Scientists may well be working in competition, they may even like to get a sneaky peek at others’ ideas, and reviewing can give an excellent opportunity to do just that. But ultimately everyone’s working towards the same goal, to improve science, to drive forward knowledge and, in the case of cancer research, to improve lives.
So it turns out the dragons might not be so scary after all.
This article also appears on Helen's blog for The Huffington Post