You are here

Peer Review: A Slippery Slope

By: Dr. F. Edward Yazbak

"When looking for health information on the Internet, don’t believe everything you see. Articles published in peer-reviewed medical journals are checked for accuracy, but anyone can put information on the Internet, so there’s no guarantee that the information you find is accurate or up-to-date."

This statement made last month (1) by someone at the National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS) is obviously not always true. The accuracy of peer-reviewed studies about arthritis drugs, specifically, has been in question lately in the news and in the courts.  

The Lancet is a well-known and respected medical publication not only in the U.K., but the world over. In its Oct. 15, 2005 issue, the journal featured an article entitled "Non-steroidal anti-inflammatory drugs and the risk of oral cancer: a nested case-control study." The first sentence of the Medline summary (2) was certainly an attention-getter: "Non-steroidal anti-inflammatory drugs (NSAIDs) seem to prevent several types of cancer"¦."

The authors, all from the department of medical oncology and radiotherapy at the prestigious Norwegian Radium Hospital in Oslo, described in the methods section how they undertook ""¦ a nested case-control study to analyze data from a population-based database, which consisted of prospectively obtained health data from all regions of Norway." Patients with oral cancer were identified among heavy smokers and their matched controls were selected from the remaining heavy smokers, who were cancer-free.

Under findings, the authors reported, "We identified and analyzed 454 (5%) people with oral cancer (279 men, 175 women, mean [SD] age at diagnosis 63.3 [13.2] years) and 454 matched controls (n=908); 263 (29%) had used NSAIDs, 83 (9%) had used paracetamol (for a minimum of 6 months), and 562 (62%) had used neither drug. NSAID use (but not paracetamol use) was associated with a reduced risk of oral cancer (including in active smokers; hazard ratio 0.47, 95% CI 0.37-0.60, p<0.0001). Smoking cessation also lowered the risk of oral cancer (0.41, 0.32-0.52, p<0.0001). Additionally, long-term use of NSAIDs (but not paracetamol) was associated with an increased risk of cardiovascular-disease-related death (2.06, 1.34-3.18, p=0.001). NSAID use did not significantly reduce overall mortality (p=0.17)."

The interpretation by the authors was, "Long-term use of NSAIDs is associated with a reduced incidence of oral cancer "¦ even in smokers."

Because of their size, Scandinavian countries have been able to research whole populations. I became acquainted with "population-based" studies when they became à la mode in order to "prove" that MMR and thimerosal did not cause autism. In fact, the "big MMR study from Denmark" by K. M. Madsen, MD, (3) published in the New England Journal of Medicine (NEJM) in November 2002, may have been the most famous "population-based" study of all times - until we published our opposing research in the Journal of American Physicians and Surgeons in the fall of 2004. (4)

With respect to The Lancet study, I needed to be sure what exactly "a nested case-control study" is. It certainly sounds more impressive than just a good old case-control study. So, with all due respect to NIAMS, I looked up the definition on the Internet and thought that I could trust the following one from Stanford University: "A nested case-control study is a type of case-control study that draws its cases and controls from a cohort population that has been followed for a period of time." (5) The explanation that followed was even more apropos to this discussion:

"A nested-case control study depends on the pre-existence of a cohort that has been followed over time. This cohort, at its inception or during the course of follow-up, has had exposure information and/or biospecimens collected of interest to the investigator. The investigator identifies cases of disease that occurred in the cohort during the follow-up period. The investigator also identifies disease-free individuals within the cohort to serve as controls. Using previously collected data and obtaining additional measurements of exposures from available biospecimens, the investigator compares the exposure frequencies in cases and controls as in a non-nested case-control study."

Because I had obsessively smoked a pipe until 1994 and because I had a wonderful dentist who worried about every little white spot in my mouth for 30 years, the subject of oral cancer has always interested me. A Medline review revealed that the lead author of the Norwegian study, alone or with others, had written extensively about the subject of oral malignancies since his first article in 2000. He published three studies in 2001, four in 2002, six in 2003, one in 2004 and three in 2005 before October. The 2004 study, "The influence of resection and aneuploidy on mortality in oral leukoplakia," was published in the NEJM; the others in Norwegian and English-language cancer and medical Journals. The October 2005 study was the only one published in The Lancet and the only one pertinent to this discussion, so far. (2)

Although I did not understand all the mysterious statistical formulae and every detail of the findings (above), I was reassured that since they were published in The Lancet, they must have been deemed accurate by more experienced statisticians who had "peer-reviewed" the paper pre-publication.

For a few weeks, I was really happy about those few tablets of Advil I had taken when I hurt my back years ago. They had apparently saved my life.

But suddenly, The Lancet's NSAID/oral cancer study started unraveling in Norway. The BBC announced on Jan. 16, 2006 that the lead author had made up patients and case histories for the study. (4)

According to the BBC report, Stein Vaaler, director of external relations at the Norwegian Radium Hospital, said that the lead author had ""¦ published an article in The Lancet in October last year whose data is totally false, actually totally fabricated"¦. His database had been completely fabricated on his computer." The Norwegian daily Dagbladet reported that of the 908 people in the study, 250 shared the same birthday.

The hospital set up a commission to investigate how the data passed peer-review locally and to scrutinize previous articles by the author.

What was stranger than the revelations about the latest study was the reaction of the editor of The Lancet. According to the BBC, a very composed Richard Horton, MD, said that "he would be speaking to the co-authors of the study to seek their permission to retract the paper." He also described the fabrication of data as a "terrible personal tragedy" for the author. 

Denying any fundamental problem with the process in general, Horton was quoted as saying, "The peer-review process is good at picking up poorly designed studies, but it is not designed to pick up fabricated research"¦. Just as in society you cannot always prevent crime, in science you cannot always prevent fabrication."

Now that must have really surprised whoever at NIAMS had been so positive that "articles published in peer-reviewed medical journals are checked for accuracy."

I am aware that the British are known to be calm and composed people, but I have to say that Horton's reaction puzzled me, almost as much as the 250 patients in the study who had the same birthday.

Horton was not as kind two years ago to Andrew Wakefield, MD. Understandably, the MMR vaccine issue had a much higher profile in the U.K. than a few cases of oral cancer in Norway. Then, Horton told the BBC he had discovered that Wakefield had a "fatal conflict of interest." In medical terminology, "fatal" is as bad as it gets. The editor's rage had coincided with an "allegation of research misconduct" in the Sunday Times.

The allegation(s) did not involve the quality of Wakefield's work or the accuracy of his findings, rather a perceived conflict of interest. According to Horton "If we had known the conflict of interest Dr. Wakefield had in this work, I think that would have strongly affected the peer reviewers about the credibility of this work and, in my judgment, it would have been rejected." (6)

On Feb. 27, 2004, attorneys for Wakefield refuted Horton's claims and demanded a full apology: "Our client entirely rejects your assertion that his work for the Legal Aid Board gave rise to a conflict of interest in relation to the paper published in The Lancet. The plain implication of the statements you have made is that our client, for nearly six years, withheld not only from The Lancet but also his colleagues the fact that he had also been engaged by the Legal Aid Board to conduct research. This, as you know, is not true. There was no secret and our client made no secret of his work for the Legal Aid Board."

The lawyers were simply reminding Horton that in a letter published in The Lancet in May 1998, Wakefield had fully explained the situation and denied that a conflict of interest had ever existed, as far as his research was concerned.  

"It is a matter of grave concern that, six years after all the relevant facts were within your knowledge, you choose not only to dismiss our client’s work but also to cast doubt on his honesty," the lawyers added. 

Families of children with autism were horrified at Horton's accusations and described the situation as a "witch-hunt." The total amount of the funding in question: £55,000. (6)

The Norwegians' recent research problems followed even more disturbing scientific news from South Korea. On Jan. 10, 2006, Reuters reported that a panel investigating world-renowned researcher Hwang Woo-suk had accused him and his team of faking two landmark papers on embryonic stem cells in 2004 and 2005. (7)

The Korean scandal was, in fact, of a much higher profile: Time magazine had named Hwang, a South Korean veterinarian and an expert in cloning, one of the "people who mattered" in 2004. Hwang, who had extensively researched animal cloning, had succeeded in cloning "human embryos capable of yielding viable stem cells that might one day cure countless diseases." (8) For that, he had been awarded South Korea's top science prize for 2004. The "Stem Cell Man," as he was called, had been quoted as saying, "If I were a woman, I would definitely donate my eggs to scientific research without hesitation." (9)

After it was revealed that data in his 2005 landmark paper on producing tailored embryonic stem cells had been "intentionally fabricated," Hwang resigned last December from his post as professor at Seoul National University.

The Other Side of the Story

In addition to "missing" problems related to facts, accuracy and calculations, the overrated medical peer-review process offers an even bigger risk: turning down good research that may not be "mainstream" or threatens golden cows.

We had initially submitted our response to the Madsen MMR study in 2004 to the New England Journal of Medicine, where it was promptly turned down by the editor. We later submitted it to another medical journal focusing on vaccines where one of the reviewers, an employee of the Centers for Disease Control and Prevention (CDC), rejected it. In the first instance, there was no review; in the second, we obviously lacked a peer.

Obviously, the opposite is also true. When the director of the CDC's National Center on Births Defects and Developmental Disabilities (10) wrote a personal letter to the editor of a medical journal that concluded, "I feel this is a very important study that deserves thoughtful consideration by the Journal. Its findings provide one strong piece of evidence that thimerosal is not causally linked to autism. Thank you for your timely consideration," he guaranteed the manuscript in question a prompt review by like-minded peers. 

Clearly, editors of medical journals do their very own peer-review of "letters to the editor."

Because it was published in the NEJM, Madsen's "big MMR study from Denmark" (3) was quoted as being convincing proof that MMR vaccination did not cause regressive autism, in spite of the fact that it was co-authored and funded by the CDC.

Three days after the study was published, Samy Suissa, Ph.D., a professor of epidemiology, biostatistics and medicine at McGill University wrote to the NEJM: "Madsen et al. observed an adjusted rate ratio of autistic disorder after vaccination of 0.92 relative to no vaccination, when the crude rate ratio (my computation) was 1.45 (95% confidence interval 1.08-1.95). Moreover, the rate by time since vaccination increases to a high of 27.3 two years after vaccination (rate ratio 2.5) and decreases thereafter to 11.4 per 100,000 per year. It is stated that adjustment for age eliminated these rate increases, but the corresponding data are unusual. Indeed, the rates of autistic disorder by age at vaccination, although not the age at follow-up, are 18.9, 14.8, 24.6, 26.9 and 12.0 per 100,000 per year respectively for ages<15,15-19,20-24, 25-35 and >35 months.

These rates are all above the overall rate of 11.0 for the reference group of no vaccination over all ages. It is then somewhat implausible for the adjusted rate ratio to fall below 1, unless the risk profile by age in the unvaccinated is vastly different than in the vaccinated (effect-modification). In this case, the adjustment for age could have been artificial. It would be useful then to present rates on subjects 24-29 months since vaccination and on the unvaccinated (crude rate ratio 2.5) stratified by age. Otherwise, one could be tempted to conclude that the figure is in fact suggestive of an association between MMR vaccination and the risk of autism."

Suissa's letter was never published in the journal and was only made public two years later in a commentary by Carol Stott et al (11) that was published in the same issue of Journal of American Physicians and Surgeons as our response.

Not only had the peer-review process at the NEJM missed the inaccuracy in Madsen's calculations, but the editor's refusal to publish Suissa's critique prevented the world from learning that there had, indeed, been a statistically significant 45 percent excess risk of autism in recipients of the MMR vaccine and, therefore, an association between MMR and autism in the studied Danish population. (11)

One can only wonder whether Horton would have allowed recent comments by a noted British physician and scientist to be printed in The Lancet, even as a letter to the editor. In The Mail on Sunday on Feb. 5, 2006 was a long article by Sue Corrigan, "Former science chief: "˜MMR fears coming true'."

In it, Peter Fletcher, former chief scientific officer at the U.K. Department of Health, suggested that the government had shown "utterly inexplicable complacency" over the MMR vaccine.

According to Corrigan, Fletcher had also said that there was a steady accumulation of evidence from scientists worldwide that the measles, mumps and rubella jab is causing brain damage in certain children and that some very powerful people in positions of great authority in Britain and elsewhere had staked their reputations and careers on the safety of MMR and were willing to do almost anything to protect themselves.

Having agreed to be an expert witness for parents' lawyers, Fletcher had examined thousands of documents that had never been made public. His informed opinion is, "Clinical and scientific data is steadily accumulating that the live measles virus in MMR can cause brain, gut and immune system damage in a subset of vulnerable children."

Fletcher went on to say, "It is the steady accumulation of evidence, from a number of respected universities, teaching hospitals and laboratories around the world that matters here. There’s far too much to ignore. Yet government health authorities are, it seems, more than happy to do so." He added that there is "a very strong clinical signal that some children are immunologically at risk from MMR"¦. Why is the government not investigating it further - diverting some of the millions of pounds spent on advertising and PR campaigns to promote MMR uptake into detailed clinical research instead?" (12)

* * * *

There obviously is a lot of misinformation on the Internet and people should be careful and selective. There is also wonderful medical and scientific information available.

Most universities and medical centers have first-class web sites and excellent information available to the public. 

In the field of continuing medical education, specifically, the material available on the Internet is consistently of the highest quality and of great clinical relevance; it is also free. I would dare say that a practicing physician who only reads "peer-reviewed" medical journals would be less generally informed about the latest developments than a colleague who also looks at the enormous good material on the Internet. This is particularly true when it comes to vaccine-related problems including regressive autism and shaken baby syndrome "¦ and indeed any information that could be perceived as likely to "rock the boat."

Probably the best-kept secret in medicine is the fact that many doctors do not even open the "peer-reviewed" medical journals that they receive and have paid for. They simply let them pile up on their desks or on the shelves of their bookcases, for effect.

Sometimes, that may be a good idea.

References

  1. Don’t Believe Everything You Read
    http://www.niams.nih.gov/hi/topics/howto/howto.htm#how3
    U.S. Department of Health and Human Services
    Public Health Service
    National Institutes of Health
    National Institute of Arthritis and Musculoskeletal and Skin Diseases
    NIH Publication No. 05"“4745
    January 2005
  2. Sudbo J, Lee JJ, Lippman SM, Mork J, Sagen S, Flatner N, Ristimaki A, Sudbo A, Mao L, Zhou X, Kildal W, Evensen JF, Reith A, Dannenberg AJ. Non-steroidal anti-inflammatory drugs and the risk of oral cancer: a nested case-control study. The Lancet. 2005 Oct 15-21;366(9494):1359-66. link
  3. Madsen KM, Hviid A, Vestergaard M, Schendel D, Wohlfahrt J, Thorsen P, Olsen J, Melbye M. A population-based study of measles, mumps, and rubella vaccination and autism. N Engl J Med. 2002 Nov 7;347(19):1477-82
  4. An Investigation of the Association between MMR Vaccination and Autism in Denmark. Goldman GS, Yazbak FE. J .Am Phys Surg  2004;9(3):70-75http://www.jpands.org/vol9no3/goldman.pdf
  5. link
  6. http://news.bbc.co.uk/1/hi/health/3510721.stm
  7. http://www.alertnet.org/thenews/newsdesk/SEO165628.htm
  8. http://www.time.com/time/asia/2004/personoftheyear/people/hwang_woo_suk.html
  9. http://times.hankooki.com/lpage/nation/200406/kt2004060416354752920.htm
  10. The CDC finances, writes and helps publish Danish research.
    Another useless CDC-supported autism study
  11. http://www.jpands.org/vol9no3/stott.pdf
  12. link