In the USA at least, the data legally belong to trialists on the grounds that it requires work to create knowledge from data. But science, particularly medical science, is essentially an enterprise conducted for moral reasons. We need to do not just what is legal but what is right. As such, we must take into account the probable wishes of the patients who give us their blood, fill in our questionnaires and die on our trials. It is difficult to believe that any patient on my trial, who completed complex questionnaires so diligently over such a long period of time, would really have wanted me to keep the data for myself rather than share it with others for the benefit of medical science in general. Vickers AJ. Whose data set is it anyway? Sharing raw data from randomized trials. Trials. 2006;7:15.
Every day, patients and their caregivers are faced with difficult decisions about treatment. They turn to physicians and other healthcare professionals to interpret the medical evidence and assist them in making individualized decisions. Unfortunately, we are learning that what is published in the medical literature represents only a portion of the evidence that is relevant to the risks and benefits of available treatments. In a profession that seeks to rely on evidence, it is ironic that we tolerate a system that enables evidence to be outside of public view. Krumholz HM. Open Science and Data Sharing in Clinical Research: Basing Informed Decisions on the Totality of the Evidence. Circulation: Cardiovascular Quality and Outcomes. 2012;5: 141-142
We are all patients, and will all face questions about what medical treatments to pursue. Some questions are trivial and unimportant, others can mean the difference between life and death. We rely on evidence-based medicine to give us reliable information about the risks and benefits associated with medical interventions, but a disturbing amount of evidence indicates that the medical literature is not always reliable. Many clinical trials are not published within a reasonable time after completion or are never published at all. Missing data leads to systematic reviews that are based on only a portion of the trials that were conducted, which can affect the results in unknown and unpredictable ways. Missing data may in some cases hold important information about risk, as in the case of Vioxx (rofecoxib). Merck had data several years before Vioxx was withdrawn from the market that showed the drug increased the risk of heart attacks, but most of the data was unpublished and out of public view. In other cases, clinical trials are published but the data are reported in a misleading and biased way, as when a negative trial is presented so as to appear positive, or analyses showing harm are omitted.
What is to be done? What can we do to make evidence-based medicine more evidence-based? Four commentaries in the March 2012 issue of Circulation: Cardiovascular Quality and Outcomes discuss how making clinical research data available outside individual drug and device companies or research groups could greatly add to the depth and reliability of our knowledge. Currently, with certain exceptions, access to most clinical trial data is restricted to the investigators or the funders. Harlan Krumholz, in an editor’s perspective, outlines the key concepts:
Now is the time to bring data sharing and open science into the mainstream of clinical research, particularly with respect to trials that contain information about the risks and benefits of treatments in current use. This could be accomplished through the following steps:
Post, in the public domain, the study protocol for each published trial. The protocol should be comprehensive and include policies and procedures relevant to actions taken in the trial.
Develop mechanisms for those who own trial data to share their raw data and individual patient data.
Encourage industry to commit to place all its clinical research data relevant to approved products in the public domain. This action would acknowledge that the privilege of selling products is accompanied by a responsibility to share all the clinical research data relevant to the products’ benefits and harms.
Develop a culture within academics that values data sharing and open science. After a period in which the original investigators can complete their funded studies, the data should be de-identified and made available for investigators globally.
Identify, within all systematic reviews, trials that are not published, using sources such as clinicaltrials.gov and regulatory postings to determine what is missing.
It must be acknowledged that there are many obstacles — political, cultural, financial — to accomplishing these goals. Some of these obstacles are discussed in the other three commentaries, which are open access and which I urge you to read:
Spertus, JA. The Double-Edged Sword of Open Access to Research Data. Circulation: Cardiovascular Quality and Outcomes. 2012;5;143-144.
Ross JS, Lehman R, Gross CP. The Importance of Clinical Trial Data Sharing: Toward More Open Science. Circulation: Cardiovascular Quality and Outcomes. 2012;5;238-240.
Gotzsche PC. Strengthening and Opening Up Health Research by Sharing Our Raw Data. Circulation: Cardiovascular Quality and Outcomes. 2012;5;236-237.
Whatever the difficulties, the current situation is clearly intolerable. Patients deserve reliable information on the risks and benefits of medical treatments and the subjects of clinical trials deserve that their contributions be fully used to benefit other patients.
I just want to highlight this profile on my friend Harlan Krumholz in Yale Alumni Magazine. Harlan is a Yale cardiologist who is a leader in the field of outcomes research — figuring out what works in the real world and applying those lessons to improve outcomes for patients. In the process, he has not been afraid to take on powerful interests, as he did by testifying in the Vioxx trials (described in Snigdha Prakash’s new book All the Justice Money Can Buy) and publishing articles on how the Vioxx debacle happened and what we can learn from it (see here, here, here and here).
I first contacted Harlan after reading about his presentation on the ENHANCE trial at the March 2008 American College of Cardiology meeting. The ENHANCE trial was designed to test whether ezetimibe, a drug that lowers LDL, added any benefit to a statin in slowing the progression of atherosclerosis, as measured by carotid intima-media thickness. My daughter, who has heterozygous familial hypercholesterolemia, had been on ezetimibe at one point, so I was particularly interested in the trial. Unfortunately, ezetimibe added no benefit at all and Harlan, representing a panel of cardiologists, was not afraid to state that it is not enough to know a drug’s effect on laboratory markers such as LDL. Rather, we need to know whether a drug improves clinical outcomes, such as heart attacks, strokes and death. As Harlan put it,
There are 3 possibilities with this drug. Eventually—one day, when outcomes studies are finally done—we may recognize that it is an effective medication for reducing cardiovascular risk. The ENHANCE study makes that less likely, but it is not impossible.
It could be that ezetimibe is simply an expensive placebo, and its principal harm is that it drains precious resources from our health care system and possibly leads people to use fewer of the drugs that have been shown to be beneficial. The ENHANCE study suggests that this may be true.
Third, it could be harmful. We do not know enough about the clinical risks of this drug. It is well tolerated and there are no obvious safety problems, but we cannot say if there is an increased risk of acute myocardial infarction or death or another important health problem.
* * * * *
This study heralds the need for clinical research to guide us in decisions for our patients; ideally, this work must be done early in the drug’s development. It is not right that we are this far down the line with this drug and we have so much uncertainty about its balance of risks and benefits. We must understand the effect of new drugs on people and that relying on a drug’s effect on a set of laboratory tests may not tell the whole story. We have learned this lesson before. It appears that we must learn it again.
Addendum: Just published in Circulation: Cardiovascular Quality and Outcomes, this editor’s perspective by Harlan Krumholz entitled “Patient-Centered Medicine: The Next Phase in Health Care.” Here’s an excerpt:
What matters most to patients are outcomes: Did I recover? Is my quality of life better? Patients want to know what has been accomplished by the tests and treatments they have undergone and what has been achieved by the time and resources that have been expended. It is time for us to fully embrace patient-centered medicine, which is ultimately outcomes oriented, with a focus on what patients experience and, among the range of medically reasonable options, gives precedence to what patients prefer.
In a recent post, I discussed a panel discussion on May 14, 2011, at the American Heart Association Quality of Care and Outcomes Research in Cardiovascular Disease and Stroke conference. The discussion addressed lessons from experiences with three drugs that were withdrawn or greatly restricted because they caused cardiovascular (CV) harm — rofecoxib (Vioxx), rosiglitzone (Avandia) and sibutramine (Meridia). I summarized the introduction by Sanjay Kaul and the presentations by Steve Nissen and Milton Packer. In this post I will discuss the presentations by statistician Dean Follmann of National Institute of Allergy and Infectious Diseases, NIH, and Ellis Unger of the Center for Drug Evaluation and Research, FDA.
Follmann’s presentation was similar to one he gave at the July 2010 joint meeting of the Endocrinologic and Metabolic Drugs Advisory Committee and Drug Safety and Risk Management Advisory Committee that was held to discuss Avandia. Follmann discussed the hierarchy of study designs, with randomized controlled trials (RCTs) that are double blind superiority trials being at the top. In such a design, randomization ensures that the groups are similar and double blinding ensures that the investigators can’t favor one arm over another. In addition, in a superiority trial the incentives encourage good study conduct because sloppiness (e.g. missing data, loose inclusion criteria, lack of adherence) makes it more difficult to show that the drug is effective. At the next level of reliability, according to Follman, are RCT noninferiority trials and meta-analyses. In a noninferiority trial, the goal is to conclude that a drug is not “unacceptably worse” than a comparator. In Follmann view, the incentives in a noninferiority trial “encourage sloppiness,” since sloppiness will tend to make the two arms more similar and thus meet the goal of noninferiority. (The RECORD trial was a noninferiority trial and was used to assess the safety of Avandia.) A meta-analysis is a quantitative synthesis of RCTs. In Follmann’s view, the quality of evidence of a meta-analysis is a bit less than a RCT, because (1) there may be unpublished trials that are not available for inclusion in the meta-analysis, (2) studies may be heterogeneous in population, endpoints, and comparators, and (3) the decisions on how to conduct the meta-analysis (e.g., what to include, how to analyze, endpoint definition) are made with knowledge of the potential safety signal. For example, to counter the Nissen-Wolski and FDA Avandia meta-analyses, which used myocardial infarction (MI) as the endpoint, GlaxoSmithKline chose a wider endpoint of serious and nonserious ischemia, resulting in a smaller hazard ratio. In addition, GSK used a “very unconventional and some would say illegitimate method of analyzing the data,” according to Follmann. Follmann also stated that it was a “revelation” to him to learn from Nissen’s presentation that GSK had done previous meta-analyses that had similar results as the Nissen-Wolski meta-analysis.
Follmann stated that the next study type in the hierarchy is observational studies. Because, observational studies are not randomized, drug choice may be based on patient characteristics, doctor preference, and unquantifiable factors. Statistical adjustment is done, but the result is less reliable than a RCT. Below observational studies are the FDA’s Adverse Event Reporting System (AERS) and data collected for other purposes, such as data collected by HMOs or the Centers for Medicare & Medicaid Services (CMS). In summary, Follmann stated that assessing a post marketing safety signal is difficult. RCTs are the best data source but are not always available.
Ellis Unger’s first remark was that Nissen had a “retrospectoscope in his back pocket” and was being a “Monday morning quarterback” with respect to the FDA’s actions concerning Vioxx and Avandia. He pointed out that the FDA has to make decisions in real time, which is not so easy, and he is not convinced that the FDA did the wrong thing, based on what it knew at the time. He does agree with the ultimate outcome for Vioxx, Avandia and Meridia.
With respect to Vioxx, Unger stated that at the time of approval it was known that there were associations between Vioxx and hypertension and edema, but in the preapproval trials there were no differences with respect to MI and stroke. The VIGOR trial showed a hazard ratio of 1.94 for the composite endpoint of death, MI and stroke. For non-fatal MI, the hazard ratio was 4.51 (p < 0.05). He does not believe the VIGOR data were enough that Vioxx should have been removed from that market at that point (2000). Unger next discussed the APPROVe trial, which was stopped two months early due to an excess in serious thrombotic events in the Vioxx group (RR 1.92), and resulted in the voluntary removal of Vioxx from the market. In the wake of Vioxx’s withdrawal, the FDA held a joint meeting of the Arthritis and Drug Safety and Risk Management Advisory C0mmittees on February 16-18, 2005 to discuss Cox-2 inhibitors. Unger summarized the data presented at the meeting as follows: (1) “all Cox-2-selective agents seem to increase CV risk (no ranking)” and (2) “available data do not support greater CV risk for selective agents as compared to non-selective agents.” After the meeting, the FDA added labeling warning of the potential for increased risk of CV thrombotic events to all NSAIDs.
With respect to rosiglitazone, Unger stated that the evidence of cardiovascular risk is “neither robust nor conclusive” and “remains an open question,” while acknowledging that there were “multiple signals of concern from various sources of data, without reliable evidence to refute risk.” He stressed the limitations of the Nissen/Wolski meta-analysis, including that the results were based on a relatively small number of events. Interestingly, Unger said that the FDA was more worried about the finding for cardiovascular death (odds ratio 1.64, p = 0.06) than the finding for MI (odds ratio 1.43, p = 0.03), even though the result for CV death was not statistically significant. Unger views the ADOPT and DREAM trials as being neutral on cardiovascular death, with both showing trends for increased MI.
With respect to the RECORD trial, Unger criticized the open-label design and possibility of ascertainment bias but also stated that the results for all-cause death are “unlikely to be influenced by bias,” and showed a favorable trend for rosiglitazone. With respect to MI, the results were “inconclusive,” as neither the GSK nor the FDA analysis showed a statistically significant increase in MIs. Unger stated that viewed as a means to test the two hypotheses generated by the Nissen/Wolski meta-analysis — rosiglitazone causes MI and increases the risk of CV death — RECORD “does not substantiate the findings of the Nissen/Wolski meta-analysis.” (For more on Unger’s views on RECORD, see his slides from the 2010 advisory committee meeting on rosiglitazone here). Finally, Unger noted that the David Graham epidemiological study of Medicare patients did not find a statistically significant higher risk of MI with rosiglitazone as compared to pioglitazone. Why didn’t the FDA take rosiglitazone off the market instead of leaving it on the market with restricted access? Unger cited conflicting data on the existence and magnitude of risk, the need for detailed re-adjudication and analysis of RECORD, the fact that some patients are currently taking rosiglitazone and want to stay on it even with knowledge of the risk.
With respect to sibutramine (Meridia), a weight loss drug that is an inhibitor of norepinephrine, serotonin and dopamine reuptake, Unger noted that at approval in 1997 the drug was known to increase blood pressure and heart rate and result in miscellaneous ECG changes, but the adverse effects were deemed “monitorable.” The European regulators, however, required a post-marketing cardiovascular outcomes study. This was the SCOUT trial, a large randomized, double-blind, placebo-controlled trial in obese patients over age 55 with a history of coronary artery disease, peripheral vascular disease, or stroke and/or Type 2 diabetes with at least one other risk factor. The primary endpoint was a composite of CV death, resuscitation after cardiac arrest, non-fatal MI and non-fatal stroke, which occurred in 11.4% of the patients on sibutramine and 10.0% of the patients on placebo (HR 1.16, p = 0.02). Following this trial, sibutramine was removed from the market in the U.S. and Europe.
Unger noted that post-marketing safety used to focus on rare, severe events that were detectable from spontaneous reporting. In recent years, there has been greater interest in small increases in common but serious events, such as MI, stroke, and CV death. Quantification of common risks is challenging with longer, larger studies required. If the drug is for a symptomatic condition such as depression or pain, it is difficult to keep patients from dropping out of the trial. It is difficult to interpret the results of a trial when there have been a lot of dropouts.
Unger stated that when the FDA reviews clinical trial data they are interested in imbalances in virtually any safety issue so we ”always see safety signals because we look at 150 adverse events.” They have to consider a number of issues in assessing causality: whether there is a plausible mechanism of action, whether it has been observed in other related drugs, whether there is a dose-response relationship, etc.
Comment: I think the problem of post approval safety is not entirely solvable, because there will always be safety signals that crop up after drugs are approved. However, I am in sympathy with Dr. Nissen’s view that safety signals should be investigated and acted on as early as possible, and preferably before approval.
Also, on the topic of Vioxx specifically, I suggest the following for further reading:
Joseph S. Ross, MD, MHS; Kevin P. Hill, MD, MHS; David S. Egilman, MD, MPH; Harlan M. Krumholz, MD, SM. Guest Authorship and Ghostwriting in Publications Related to Rofecoxib: A Case Study of Industry Documents From Rofecoxib Litigation. JAMA. 2008;299(15):1800-1812.
Keven P. Hill, MD, MHS; Joseph S. Ross, MD, MHS; David S. Egilman, MD, MPH; Harlan M. Krumholz, MD, SM. The ADVANTAGE Seeding Trial: A Review of Internal Documents. Annals of Internal Medicine. 2008;149(4):251-258.
Joseph S. Ross, MD, MHS; David Madigan, PhD; Kevin P. Hill, MD, MHS; David S. Egilman, MD, MPH; Yongfei Wang, MS; Harlan M. Krumholz, MD, SM. Pooled Analysis of Rofecoxib Placebo-Controlled Clinical Trial Data: Lessons for Postmarket Pharmaceutical Safety Surveillance. Archives of Internal Medicine. 2009;169(21): 1976-1985.
Joseph S. Ross, MD, MHS; David Madigan, PhD; Marvin A. Konstam, MD; David S. Egilman, MD, MPH; Harlan M. Krumholz, MD, SM. Persistence of Cardiovascular Risk After Rofecoxib Discontinuation. Archives of Internal Medicine. 2010;170(22):2035-2036.
Snigdha Prakash, All the Justice Money Can Buy: Corporate Greed on Trial (2011) (book by former NPR reporter Snigdha Prakash on the Vioxx saga — focuses on a particular Vioxx trial).