Research Integrity and Evidenced Based Medicine

By Bruce W. McNulty, MD
This article is part of our MedRIB project and was published March 13, 2008.

As we discussed in our previous blog entry, central to our understanding of medical research integrity is the notion of Evidenced Based Medicine (EBM). Physicians in practice like to think that they treat their patients based on firm scientific knowledge that follows from sound clinical studies such as those that are typically performed in a double-blind type fashion to avoid the possibility of chance and bias. Past reports have shown, however, that physicians often base their practices on other things: their anecdotal experience, advice from colleagues, local practice patterns, and advice from industry sales representatives. Following these studies, a few years back, a movement started called ‘Evidence Based Medicine’. Its concept is clear and straight forward: treat patients based on the best, most highly rated medical evidence.

The goal of those involved in EBM is to create easily disseminated information for physicians that will help them consistently manage the day to day medical issues that they encounter in their practice. For instance, The American Heart Association (AHA) may bring experts together to review current literature and then provide evidence-based guidelines on the treatment of victims of acute myocardial infarctions, i.e., heart attacks. The AHA experts may then go as far as creating goals for doctors and institutions to meet in terms of percentages of patients that get certain recommended treatments or medical interventions. The extent to which these goals are met can then be used for assessment purposes to generate a physician or hospital “report card” by insurance companies or the federal government’s Centers for Medicare and Medicaid Services (CMS). Regional data may become public record, and reported on the front page of local papers with top performing hospitals being able to tout their record of excellence. CMS has also begun a reimbursement system called pay for performance in which the payment of hospitals and physicians can be either positively or negatively affected based on meeting EBM goals. More can be found at the CMS website at: http://www.cms.hhs.gov/MedicaidSCHIPQualPrac/04_P4P.asp. Thus the stakes are high for both health care providers, and the patients they serve.

What happens, however, if the studies on which evidence based guidelines are created are biased? Clearly, the financial implications are significant. Let us restate our question: What happens if a scientist’s or company’s financial interests exert undue influence on the research that is being conducted, presented, and published? There is growing evidence that such vested interest often has a profoundly negative influence on the conduct of medical research. This blog will look back at some examples of what has occurred in this regard and, where appropriate, we will try to propose some solutions to this very serious problem.

As a start, let’s consider a paper recently published in the New England Journal of Medicine (January 17, 2008, Volume 358:252-260, No. 3): “Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy” by Erick H. Turner, M.D., Annette M. Matthews, M.D., Eftihia Linardatos, B.S., Robert A. Tell, L.C.S.W., and Robert Rosenthal, Ph.D. A link to the summary of this article can be found at: http://content.nejm.org/cgi/content/short/358/3/252?query=TOC

This article looks at studies that were registered with the FDA on the efficacy of antidepressants. The authors analyzed the registered studies and were able to divide the studies into those that showed a positive or beneficial effect of antidepressants vs. those whose results were either negative or questionable at best. They then looked at which studies were then actually published in the medical literature for doctors to see, and found that the vast majority of positive studies were published, and the majority of negative ones were not. They also found that many of the questionable studies, in their opinion, were often ‘spun’ to be more beneficial than the data supported in their conclusions. Turner et al. found that if one contrasted published studies on the effectiveness of antidepressants to all the registered FDA studies, 94% of the published literature provided evidence in support of the benefit of antidepressants, whereas if unpublished studies were included in the analysis, only 51% of clinical trials showed positive results from the use of antidepressants.

This presents a clear illustration of the issue of publication bias. For a variety of reasons, articles showing a benefit of treatment may be more likely to be published than those showing no benefit. It may be that a journal would prefer to publish a study that has a positive outcome and could affect patient management. It may be that those doing the work are selectively submitting positive studies to journals because the work is being sponsored by the manufacturer of the medication or treatment being tested. What the medical practitioner is left with in this case however is published data showing the vast majority of studies supporting the use of antidepressants. Following the creed of Evidenced Based Medicine, a physician would seem to be on solid ground by basing treatment on the published findings. CMS and medical societies could point to sound published evidence to create treatment goals for the diagnosis of depression. Yet clearly the entire story has not been told by the published literature.

It would seem that there is a solution to publication bias: that any study that wants to be published must register prior to the initiation of the trial. And even if not published, all data from such trials must be readily available to medical practitioners as well as the experts creating guidelines for us to follow. Evidence based medicine must be based on all the evidence available, not just that selected by those that may have a vested interest in the results.