Reference
Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. PLOS ONE, 10(8), e0132382. https://doi.org/10.1371/journal.pone.0132382
Info
FirstAuthor:: Kaplan, Robert M.
Author:: Irvin, Veronica L.
~
Title:: Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time
Year:: 2015
Citekey:: KaplanIrvin_2015_LikelihoodNullEffects
itemType:: journalArticle
Journal:: PLOS ONE
Volume:: 10
Issue:: 8
Pages:: e0132382
DOI:: 10.1371/journal.pone.0132382
Link
Abstract
Background We explore whether the number of null results in large National Heart Lung, and Blood Institute (NHLBI) funded trials has increased over time. Methods We identified all large NHLBI supported RCTs between 1970 and 2012 evaluating drugs or dietary supplements for the treatment or prevention of cardiovascular disease. Trials were included if direct costs >$500,000/year, participants were adult humans, and the primary outcome was cardiovascular risk, disease or death. The 55 trials meeting these criteria were coded for whether they were published prior to or after the year 2000, whether they registered in clinicaltrials.gov prior to publication, used active or placebo comparator, and whether or not the trial had industry co-sponsorship. We tabulated whether the study reported a positive, negative, or null result on the primary outcome variable and for total mortality. Results 17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 (χ2=12.2,df= 1, p=0.0005). There has been no change in the proportion of trials that compared treatment to placebo versus active comparator. Industry co-sponsorship was unrelated to the probability of reporting a significant benefit. Pre-registration in clinical trials.gov was strongly associated with the trend toward null findings. Conclusions The number NHLBI trials reporting positive results declined after the year 2000. Prospective declaration of outcomes in RCTs, and the adoption of transparent reporting standards, as required by clinicaltrials.gov, may have contributed to the trend toward null findings.
Yellow: Interesting
Highlight ( Page 2)
supplements funded between 1970–2012
Highlight ( Page 5)
A total of 55 trials were analyzed
Highlight ( Page 6)
Fig 1 plots the relative risks of the primary outcome by the publication year of the main outcome paper. Because it was an extreme outlier, the CAST study is excluded from the figure. Prior to publication in 2000, studies often showed benefits of treatments with the notable exception of CAST (not shown in figure). Following 2000, confidence intervals for relative risk ratios included 1.0 in all cases, with the exceptions of the PREVENT and the SANDS trials (benefit) and the Women’s Health Initiative (Harm). In addition, the variability in RRs was considerably reduced after the year 2000 (Fig 2).
Highlight ( Page 6)
more recent trials may have evaluated their treatment drug against clinically effective alternatives instead of placebos
Highlight ( Page 8)
A final explanation for the trend toward null reports is that current authors face greater constraints in reporting the results of their studies. In our review, the year 2000 marks the beginning of a natural experiment. After the year 2000, all (100%) of large NHLBI were registered prospectively in ClinicalTrials.Gov prior to publication. Prior to 2000 none of the trials (0%) were prospectively registered.
Highlight ( Page 8)
Prospective declaration of the primary outcome variable is important because it eliminates the possibility of selecting for reporting an outcome among many different measures included in the study
Highlight ( Page 8)
Had the prospective declaration of a primary outcome not have been required, it is possible that the number of positive studies post-2000 would have looked very similar to the pre-2000 period.
Highlight ( Page 8)
Beginning in approximately 2000, the likelihood of showing a significant benefit in large NHLBI funded studies declined. Among the explanations we evaluated, the requirement of prospective registration in Clinicaltrials.gov is most strongly associated with the observed trend toward null clinical trials. The decline is not easily explained by the increased use of active comparators or a decline in industry sponsorship.
Highlight ( Page 8)
It is widely noted that journals favor publication of statistically significant findings[2]. Bias in favor of publishing positive outcomes is not a likely explanation for our results. We focused on large trials because previous analyses by NHLBI reported that 97% of trials with annual budgets over $500,000/year were published [3], thus removing publication bias as a rival explanation
Highlight ( Page 9)
It has been argued that there have been few efficacious drugs in the pipeline[4,5]. Since about 1998, there has been a systematic decline in the number approvals for new cardiovascular drugs[6].
Highlight ( Page 9)
Our analysis is limited to large NHLBI-funded trials and to studies on cardiovascular outcomes in adults. We focused on NHLBI because the Institute has championed transparency and allowed us full access to all trials.
Highlight ( Page 10)
We cannot say that trend toward null trials to preregistration in ClinicalTrials.gov is causal. Our analysis included only a small number of trials and the design of the study does not allow causal inferences.
Highlight ( Page 10)
The transparency of RCTs is likely to have improved following the FDA Modernization Act of 1997, which created the ClinicalTrials.gov registry[8], a service that required registration of studies that test drugs, biologics, or devices for the treatment of serious or life threatening diseases[9–11]. Registered studies must provide: the study’s purpose, recruitment status, design, eligibility criteria, locations and pre-specified primary and secondary outcomes[11].