Editorial: The Inappropriate Use of Hypothesis Testing to Infer Safety of Calcium

Channel Blockers

Frank E Harrell Jr Professor of Biostatistics and Statistics Director, Division of Biostatistics and Epidemiology Department of Health Evaluation Sciences University of Virginia School of Medicine Charlottesville, Virginia 22908, USA

                fharrell@virginia.edu

December 16, 1997

Reicher–Reiss, Behar, Boyko, Mandelzweig, Kaplinsky, and Goldbourt (RBBMKG) and the SPRINT Study Group have analyzed 5–year follow–up data from the SPRINT trial, concluding that “our results do not support an association between nifedipine therapy and a late harmful effect on long– term mortality”, “the main finding of this study is that low dose short– acting nifedipine therapy used in a randomized fashion for one year was not associated with increased mortality during a five–year follow–up”, and “no carry over excess mortality effect appeared after one year of use.” In fact the results support a significant excess mortality as well as no effect and a reduction in mortality. There are two fatal flaws in the authors’ analyses and a number of minor errors.

Before addressing RBBMKG’s analyses a number of other issues that are very pertinent to studies of the safety or harm of calcium channel blockers or other compounds that will not be addressed here should be listed.

1. What should be the length of follow–up for analyzing survival outcomes?

2. How long should the drug be given so that adequate exposures are assured?

3. What dose should be studied?

4. Should patient compliance be accounted for?

5. Which clinical characteristics should patients have so that they reflect clinical practice?

6. Should the results of one study be pooled with findings from other studies so as to maximize the precision of survival estimates? If so, how similar should the studies be with respect to patient entry criteria, dosing, and follow–up?

Some of the less important potential errors in RBBMKG are as follows.

1. Even when subjects are intended to be followed 5 years, it is often the case that the follow–up duration varies from patient to patient. If that is the case, there are no simple numerators and denominators to use for computing the survival fraction. The authors have reported simple survival fractions but these may need to be replaced with Kaplan–Meier estimates.

2. The authors are referred to “risk ratios” where hazard ratios may be more appropriate.

3. The paper did not take into account the effect of concomitant medications.

4. The dose of nifedipine may have been too low to detect harm.

Now on to the severe problems with RBBMKG’s analysis, in order of increasing importance. First, the authors highlighted 5–year effects of nifedipine on survival although the drug was only intended to be given for one year. It is only reasonable to study the effect of chronic dosing calcium channel blockers for the duration of dosing. The authors have done a formal test for constancy of effect of nifedipine over the entire 5–year period. Had post 1–year compliance data (both drop–out and drop–in) been available it would have been preferable to model the potentially changing drug effect to correspond with the 0–5 year compliance pattern. At any rate, tests of constant effect do not necessarily have adequate power to allow one to conclude that effects are constant if the test is not “statistically significant.” It would be interesting to compute the nifedipine:placebo hazard ratio for survival experiences restricted to one year. The crude mortality estimates (0.05 vs. 0.059) indicate some evidence for increased one–year mortality with nifedipine.

The second, and by far the most major problem with RBBMKG is that they misused null hypothesis tests and P –values. In traditional (frequentist) statistical inference, a P –value is used to gauge the extent of evidence against a null hypothesis (H_{0}: hazard of death for nifedipine equals hazard of death for placebo). P –values can only provide evidence against a hypothesis, and they can never provide evidence in favor of a hypothesis[6]. P –values can be large because the drug effect is small or because the power is low (e.g., the sample size is inadequate), but one cannot tell from the data which of these conditions holds as one does not know the true drug effect. When one wishes to conclude that two drugs have similar effects, specialized statistical methods must be used. Ware and Antman[8] wrote an eloquent editorial explaining the difference between studies (and statistical methods) used to assess efficacy and those used to assess equivalence. See also [4].

Confidence intervals, althought difficult to interpret, can solve some of the problems of P –values and point–hypothesis tests. RBBMKG presented 0.95 confidence limits for the nifedipine:placebo hazard ratio of [0.81, 1.22], yet they failed to use the confidence limits in interpreting the study findings! The confidence limits indicate that the results are consistent with a 19% reduction in hazard as well as a 22% increase in the hazard of death due to nifedipine. As the upper confidence interval is above 1.0, one cannot make a conclusion about the safety of nifedipine from this study. It should be noted that the confidence level to be used for such analyses is somewhat arbitrary. The study supports a harmful effect of nifedipine, a beneficial effect, and no effect; i.e., the study is not informative.

Bayesian statistical methods are being increasingly used to solve many of the problems with P –values and hypothesis tests, and to provide probability intervals that are more meaningful and interpretable than confidence intervals[7, 5, 3]. The Bayesian approach involves quantifying the state of evidence about a treatment prior to the experiment, and updating the evidence using the data from the experiment to obtain a current probability distribution for the treatment effect. This probability distribution can be used to compute many probabilities of interest, and with this Bayesian approach it is as easy to measure evidence for safety as it is to measure evidence for harm. One can do this by computing for example the probability that the hazard ratio exceeds 1.0 or the probability that it is less than any given value.

To do a full Bayesian analysis one needs the raw data for the study, but an approximate analysis can be carried out as follows[2]. Here we have a special case where to 3 significant digits the estimated hazard ratio for nifedipine:placebo is 1.00. So if we assume that the log hazard ratio estimate has a normal distribution with mean zero (as did RBBMKG in computing confidence intervals) and if we assume a prior distribution for the log hazard ratio that is symmetric about zero^{1}, the current probability distribution for the log hazard ratio is symmetric with mean zero (and with variance depending on the variance of the prior distribution). The area under a symmetric probability curve to the left of its mean is \frac{1}{2}. Then the current state of evidence about the safety of nifedipine from RBBMKG is quantified by a probabilty of \frac{1}{2} that nifedipine is safe. In other words, from SPRINT alone we are now at the point of maximum uncertainty about the safety or efficacy of nifedipine. The Bayesian calculation is far more useful than a failure to reject H_{0}, and it provides far less comfort than a large P –value.

An approximate Bayesian re–analysis of the observational data for the 2607 patients not included in SPRINT is as follows. The hazard ratio estimate is 1.08 with 0.95 confidence limits [0.93, 1.27]. The (natural) log hazard ratio is then log(1.08) = 0.077 with confidence limits [−0.0726, 0.2390]. The standard deviation of the log hazard ratio equals the width of this confidence interval divided by 2 × 1.96, i.e., 0.0795. If we use a non–informative prior distribution for the log hazard ratio, i.e., we assume that all nifedipine:placebo differences in log hazard are equally likely, the current (posterior) distribution of the population log hazard ratio is approximately normal with mean 0.077 and standard deviation 0.0795. The probability of safety is the probability that the hazard ratio is less than 1.0 or the probability that the log hazard ratio is less than 0. For the normal distribution with mean and standard deviation listed above, this probability of safety is 0.166.

1It is equally likely, before SPRINT, that nifedipine decreases the log hazard of death by θ as that it increases the log hazard of death by θ.

Even allowing the authors to analyze 5–year mortality patterns for 1– year dosing in the randomized patients, their own confidence limits show that the data are consistent with a 22% increase in hazard from nifedipine. The 1–year results are more pessimistic: The point estimate for the 1–year mortality ratio (0.059/0.05) indicates an 18% increase in mortality, and an upper 0.95 confidence limit for this mortality increase would show that the 1–year data are consistent with an increase in the probability of death by 1 year that is much higher than 18%. And this is for a low dose of nifedipine.

The SPRINT trial was designed as an efficacy trial and not as an equivalence study. Equivalence studies generally require significantly higher sample sizes, and equivalence cannot be demonstrated using standard hypothesis tests about a null treatment effect. Equivalence (or actually “equal or better”) can be inferred using the upper confidence limit, by using a Bayesian approach, or by using a hypothesis test where the null treatment effect is not assumed to be zero. The pitfalls of the RBBMKG analysis can be summed up in the title of Altman and Bland’s excellent editorial “Absence of evidence is not evidence of absence” [1].

References

[1] D. G. Altman and J. M. Bland. Absence of evidence is not evidence of absence. British Medical Journal, 311:485, 1995.

[2] D. Faraggi and R. Simon. Large sample Bayesian inference on the parameters of the proportional hazard model. Statistics in Medicine, 16:2573– 2585, 1997.

[3] L. Freedman. Bayesian statistical methods. British Medical Journal, 313:569–570, 1996.

[4] B. Jones, P. Jarvis, and A. F. Ebbutt. Trials to assess equivalence: The importance of rigorous methods. British Medical Journal, 313:36–39, 1996.

[5] R. J. Lilford and D. Braunholtz. The statistical basis of public policy: A paradigm shift is overdue. British Medical Journal, 313:603–607, 1996.

[6] M. J. Schervish. p values: What they are and what they are not. American Statistician, 50:203–206, 1996.

[7] D. J. Spiegelhalter, L. S. Freedman, and M. K. B. Parmar. Bayesian approaches to randomized trials. Journal of the Royal Statistical Society Series A, 157:357–416, 1994.

[8] J. H. Ware and E. M. Antman. Equivalence trials (editorial). New England Journal of Medicine, 337:1159–1161, 1997.