Terence Kealey: Science Notebook
Download 'Too Hot', an exclusive Specials track from iTunes
Which is more likely to be reliable, a news story in The Times or a peer-reviewed paper in The Lancet? As the recent volte-face over the dangers of vitamin supplements shows, the newspaper story may be the more credible.
Peer-reviewed science is forever doing these volte-face (can you remember if coffee is good or bad for you this week?) but The Times is not suddenly going to announce that, er, sorry, it has discovered that China is in fact a parliamentary democracy.
Peer review is the modern sacred cow. “Has it been peer-reviewed?” people ask, breathlessly, when presented with a finding. But peer review is not definitive.
In a recent letter to the journal Science, Robert Zucker, of the University of California, explained the rules of peer review. Basically, they are: “Give the guy a break.” Professor Zucker wrote: “Do not reject a paper with a brilliant new idea simply because the evidence is not as comprehensive as could be imagined.”
You might suppose no scientist would publish a brilliant new idea (“vitamin supplements kill you”) without comprehensively validating it, yet the vitamin paper did not report what the patients died of.
Actually, no one knows. This is very uncomprehensive evidence indeed, because when we do eventually discover the causes of death, we may learn they owe nothing to vitamins, and science will do yet another volte-face.
Professor Zucker also wrote: “Authors need not exclude every possible explanation for their results.” The vitamin paper was not an original study, but simply an overview of 67 previously published reports in which someone died while taking supplements. But the study ignored 405 other reports in which nobody died, so one possible explanation for the results is that the authors omitted inconvenient data. The authors dismissed the omission with hypothetical statistics, yet the paper passed peer review.
Moreover, the patients were not taking natural vitamins but artificial ones, and the reports all differed in terms of dose, combination of vitamins and duration (weeks or years). The study raises so many questions that we still do not know what supplements might actually do. Yet all this is compatible with peer review.
But we can expect no more from science. Nature is too complicated for any paper to be exhaustive, so scientists have settled on a workaday, not comprehensive, standard of peer review. In defence of that workaday standard, Professor Zucker advised journals not “to reflexively demand that more be done” by authors. The public may treat a Lancet paper as ex cathedra but scientists view it simply as work in progress, which is why The Lancet is less definitive than The Times.
As for vitamin supplements, most doctors instinctively dismissed them as worthless for healthy people decades ago.
Terence Kealey is Vice-Chancellor of the University of Buckingham
Win a luxury weekend to Newcastle and its neighbour Gateshead, find out more here
Risk, resilience and embracing new technology
Industry sectors news at a glance. Interactive heatmap, video and podcast
Discover the power of collective thinking. Submit a solution and be in with a chance to win a Media Hub Home Entertainment System
The inside track on current trends in the charity, not for profit and social enterprise sectors
Everything the Business Traveller needs to know to make a better trip
Make the most of the summer and enter our fabulous photographic competition, you could win a £5000 holiday
Corsica is an island of beauty and contrast, an ideal holiday destination
Enjoy further reading from Travel to Fashion, Business to Sport, discover more
Shortcuts to help you find sections and articles
The clever way to lease a new car is with Car leasing made simple™
2009
per month on 36-month
Personal Contract Hire (PCH)
2008
42850
Car Insurance
£24,250 - £30,346
MI5
London
£60,000
The Environment Agency
Bristol
Up to £90K
Boots
Midlands
OTE £85k
Credit Protection Association
Nationwide Opportunities
Completely London
Luxury Condo's in Manhattan with NYC views
The best new homes in Wimbledon?
Nationwide
Fabulous Cruise And Cruise & Stay Offers Including Virgin Atlantic Flights Prices Start From Only £699pp!
Last Minute Cruise And Cruise & Stay Offers. Med From £499pp, Caribbean From £699pp!
5 star quality at a 3 star price.
8 fabulous Canadian cities ...you won’t find cheaper
Contact our advertising team for advertising and sponsorship in Times Online, The Times and The Sunday Times, or place your advertisement.
Times Online Services: Dating | Jobs | Property Search | Used Cars | Holidays | Births, Marriages, Deaths | Subscriptions | E-paper
News International associated websites: Globrix Property Search | Property Finder | Milkround
Copyright 2009 Times Newspapers Ltd.
This service is provided on Times Newspapers' standard Terms and Conditions. Please read our Privacy Policy.To inquire about a licence to reproduce material from Times Online, The Times or The Sunday Times, click here.This website is published by a member of the News International Group. News International Limited, 1 Virginia St, London E98 1XY, is the holding company for the News International group and is registered in England No 81701. VAT number GB 243 8054 69.
I wish people would make the effort to learn how a systematic review/ meta analysis actually works, before embarrassing themselves by spouting in public.
Graeme, Manchester, UK
This latest slur against vitamins is just another part of the attempt by drug companies to remove competition. Vitamins prevent illhealth and these companies don't want healthy people, they want chronically sick customers dependant on their drugs. They need to be stopped, or we'll all be the victims
Mr Davies, Stockport, Cheshire
"No profits in vitamins."
Kevin, Lincoln
Good Lord. Haven't we been over this?
1) Huge profit in vitamins. Multi-billion dollar industry largely owned by "big pharma".
2) Cochrane Library refuses all corporate funding, so by far your best chance at unbiased science.
Think things through, Kevin.
Tom Chivers, London,
What volte face? Real nutrionists and dieticians (as opposed to pill salesman and "nutritional therapists") have known for ages that very few people are deficient in vitamins, and that some can be harmful in high doses, Naturally the pill industry does not like the excellent Cochrane review.
David Colqhoun, London, UK
"But the study ignored 405 other reports..." and so it should have done. Not sure why? Go read the original at http://mrw.interscience.wiley.com/cochrane/clsysrev/articles/CD007176/pdf_fs.html
"...yet the paper passed peer review." I feel sure this article would not have done
Dr Dave, Exeter,
No profits in vitamins, and if there's even a small chance they keep people healthy and away from big pharms expensive drugs then this kind of manipualtion of the scientific process is inevitable. Each vitamin must be examined in isolation to determine (contradictive) effect. This study is flawed.
kevin, Lincoln, UK
I'm sure if one is looking for absolutism, then science isn't the way to go. Science is all about probabilities and the preponderance of evidence. In science new evidence changes old analyses and conclusions.
However, if absolutism is what one wants then the way to go is religion and journalism.
pv, Treviso, Italy
Another problem with peer review is that orthodoxy can make it difficult for genuine new ideas to surface and flourish.
After all, most peer reviewers are senior members of the scientific community, and many will have built their careers on the prevailing scientific orthodoxy.
Chris Palmer, Southampton, England
So let me see if i understand, Scientists put forward a paper attempting to move an idea forward. This is reviewed by peers who compete for grants for research into the same field?WOW this could be really messy, maybe what you need is something really scary like a big asteroid or global warmi...,OH
steve, Sunshine Coast, Australia
many scientific papers are not perfect.
wilson, guangdong, china
In essence, Scientists are people with their foibles just like the rest of us.I trust science but not those pitching for news attention or funding.
Finding the right notes to play the pipers tune is a fact of both science and journalism.Scepticism is better for you than vitamins ever were.
robert everitt, wolverhampton,
This article complains that researchers looked at "artificial", not "natural" vitamins. Since the study is assessing the safety of vitamin supplements, this makes perfect sense. If you want to find out if vitamin pills are safe, looking at people taking those pills seems a good place to start.
P Wilson, London,
A few points for those who do not understand meta analysis:
Rejected studies were rejected for scientifically sound reasons such as inadequate methodological or data quality, and especially for the fact that as this was a study about deaths, only those studies which had deaths could be looked at!
Adam Gibb, Manchester, England
I think that a peer-reviewed article would make it clear what evidence the authors looked at, what evidence they didn't, and precisely how they reached their conclusions. One can then judge how solid and how applicable those conclusions are.
Newspaper articles almost never do this, nor should they
Hal, New York, United States
If all this article is stating is that there are times when science is not infallible, and it so happens that an article with no scientific basis guesses the correct answer, then, statistically, this is of course an inevitability.
Dominic Pearson, Dubai, UAE
The really sad thing about Fleet Street these days is that the speculative article n today's Times (presented without a single conditional verb, of course) about Madeleine McCann's movements on the night of her disappearance probably would pass peer review amongst ordinary journalists...
Ian Kemmish, Biggleswade, UK
Surely this article notes merely that findings in peer-reviewed science journals may need to be subjected to further research. This article makes no reference to the credibility of newspapers contrary to the comparative opening paragraph. That is therefore not a credible assertion without evidence.
John Scott, London,
This is all very well, but I think more research is needed.
Christopher Chantrill, Seattle, USA
Hm. I'm sure many scientific papers are flawed. But the same can be said of some of the selective reports that appear in newspapers. The Press (and broadcast media) are not renowned for letting facts and balance stand in the way of a good story.
James, Hong Kong, China