\documentstyle[12pt]{article}
\begin{document}
\title{Editorial: The Inappropriate Use of Hypothesis Testing to Infer Safety
of Calcium Channel Blockers}
\author{Frank E Harrell Jr \\
Professor of Biostatistics and Statistics \\
Director, Division of Biostatistics and Epidemiology \\
Department of Health Evaluation Sciences \\
University of Virginia School of Medicine \\
Charlottesville, Virginia 22908, USA \\
{\tt fharrell@virginia.edu}}
\date{\today}
\maketitle
Reicher--Reiss, Behar, Boyko, Mandelzweig, Kaplinsky, and Goldbourt
(RBBMKG) and the SPRINT Study Group have analyzed 5--year follow--up
data from the SPRINT trial, concluding that ``our results do not
support an association between nifedipine therapy and a late harmful
effect on long--term mortality'', ``the main finding of this
study is that low dose short--acting nifedipine therapy used in a
randomized fashion for one year was not associated with increased
mortality during a five--year follow--up'', and ``no carry over excess
mortality effect appeared after one year of use.''  In fact the
results support a significant excess mortality as well as no effect
and a reduction in mortality.  There are two fatal flaws
in the authors' analyses and a number of minor errors.

Before addressing RBBMKG's analyses a number of other issues that are
very pertinent to studies of the safety or harm of calcium channel
blockers or other compounds that will not be addressed here should be
listed.  
\begin{enumerate}
\item	What should be the length of follow--up for analyzing survival
		outcomes?
\item	How long should the drug be given so that adequate exposures
		are assured?
\item	What dose should be studied?
\item	Should patient compliance be accounted for?
\item	Which clinical characteristics should patients have so that
		they reflect clinical practice?
\item	Should the results of one study be pooled with findings from
		other studies so as to maximize the precision of survival
		estimates?  If so, how similar should the studies be with
		respect to patient entry criteria, dosing, and follow--up?
\end{enumerate}

Some of the less important potential errors in RBBMKG are as follows.
\begin{enumerate}
\item	Even when subjects are intended to be followed 5 years, it is
		often the case that the follow--up duration varies from
		patient to patient.  If that is the case, there are no simple
		numerators and denominators to use for computing the survival
		fraction.  The authors have reported simple survival fractions
		but these may need to be replaced with Kaplan--Meier
		estimates.
\item	The authors are referred to ``risk ratios'' where hazard
		ratios may be more appropriate.
\item	The paper did not take into account the effect of
		concomitant medications.
\item	The dose of nifedipine may have been too low to detect harm.
\end{enumerate}

Now on to the severe problems with RBBMKG's analysis, in order of
increasing importance.  First, the authors highlighted 5--year effects
of nifedipine on survival although the drug was only intended to be
given for one year.  It is only reasonable to study the effect of
chronic dosing calcium channel blockers for the duration of dosing.
The authors have done a formal test for constancy of effect of
nifedipine over the entire 5--year period.  Had post 1--year
compliance data (both drop--out and drop--in) been available it would
have been preferable to model the potentially changing drug effect to
correspond with the 0--5 year compliance pattern.  At any rate, tests
of constant effect do not necessarily have adequate power to allow one
to conclude that effects are constant if the test is not
``statistically significant.''  It would be interesting to compute the
nifedipine:placebo hazard ratio for survival experiences restricted to
one year.  The crude mortality estimates (0.05 vs.\ 0.059) indicate
some evidence for increased one--year mortality with nifedipine.

The second, and by far the most major problem with RBBMKG is that they
misused null hypothesis tests and $P$--values.  In traditional ({\em
frequentist}) statistical inference, a $P$--value is used to gauge the
extent of evidence against a null hypothesis ($H_{0}$: hazard of death
for nifedipine equals hazard of death for placebo).  $P$--values can
{\em only} provide evidence against a hypothesis, and they can
never provide evidence in favor of a hypothesis\cite{sch96pva}.
$P$--values can be large because the drug effect is small or because
the power is low (e.g., the sample size is inadequate), but one cannot
tell from the data which of these conditions holds as one does not
know the true drug effect.  When one wishes to conclude that two drugs
have similar effects, specialized statistical methods must be used.
Ware and Antman\cite{war97equ} wrote an eloquent
editorial explaining the difference between studies (and statistical
methods) used to assess efficacy and those used to assess equivalence.
See also \cite{jon96tri}.

Confidence intervals, althought difficult to interpret, can solve some
of the problems of $P$--values and point--hypothesis tests.  RBBMKG
presented 0.95 confidence limits for the nifedipine:placebo hazard
ratio of $[0.81, 1.22]$, yet they failed to use the confidence limits
in interpreting the study findings!  The confidence limits indicate
that the results are consistent with a 19\% reduction in hazard as
well as a 22\% increase in the hazard of death due to nifedipine.  As
the upper confidence interval is above 1.0, one cannot make a
conclusion about the safety of nifedipine from this study.  It should
be noted that the confidence level to be used for such analyses is
somewhat arbitrary.  The study supports a harmful effect of
nifedipine, a beneficial effect, and no effect; i.e., the study is not
informative.

Bayesian statistical methods are being increasingly used to solve many
of the problems with $P$--values and hypothesis tests, and to provide
probability intervals that are more meaningful and interpretable than
confidence intervals\cite{spi94bay,lil96sta,fre96bay}.  The Bayesian
approach involves quantifying the state of evidence about a treatment
prior to the experiment, and updating the evidence using the data
from the experiment to obtain a current probability distribution for
the treatment effect.  This probability distribution can be used to
compute many probabilities of interest, and with this Bayesian
approach it is as easy to measure evidence for safety as it is to
measure evidence for harm.  One can do this by computing for example
the probability that the hazard ratio exceeds 1.0 or the probability
that it is less than any given value.  

To do a full Bayesian analysis one needs the raw data for the study,
but an approximate analysis can be carried out as
follows\cite{far97lar}.  Here we have a special case where to 3
significant digits the estimated hazard ratio for nifedipine:placebo
is 1.00.  So if we assume that the log hazard ratio estimate has a
normal distribution with mean zero (as did RBBMKG in computing
confidence intervals) and if we assume a prior distribution for the
log hazard ratio that is symmetric about zero\footnote{It is equally
likely, before SPRINT, that nifedipine decreases the log hazard of
death by $\theta$ as that it increases the log hazard of death by
$\theta$.}, the current probability distribution for the log hazard
ratio is symmetric with mean zero (and with variance depending on the
variance of the prior distribution).  The area under a symmetric
probability curve to the left of its mean is $\frac{1}{2}$. Then the
current state of evidence about the safety of nifedipine from RBBMKG
is quantified by a probabilty of $\frac{1}{2}$ that nifedipine is
safe.  In other words, from SPRINT alone we are now at the point of
maximum uncertainty about the safety or efficacy of nifedipine.  The
Bayesian calculation is far more useful than a failure to reject
$H_{0}$, and it provides far less comfort than a large $P$--value.

An approximate Bayesian re--analysis of the observational data for the
2607 patients not included in SPRINT is as follows.  The hazard ratio
estimate is 1.08 with 0.95 confidence limits $[0.93, 1.27]$.  The
(natural) log hazard ratio is then log(1.08) = 0.077 with confidence
limits $[-0.0726, 0.2390]$.  The standard deviation of the log hazard
ratio equals the width of this confidence interval divided by $2
\times 1.96$, i.e., 0.0795.  If we use a non--informative prior
distribution for the log hazard ratio, i.e., we assume that all
nifedipine:placebo differences in log hazard are equally likely, the
current (posterior) distribution of the population log hazard ratio is
approximately normal with mean 0.077 and standard deviation 0.0795.
The probability of safety is the probability that the hazard ratio is
less than 1.0 or the probability that the log hazard ratio is less
than 0.  For the normal distribution with mean and standard deviation
listed above, this probability of safety is 0.166.
%pnorm(0,.077,.0795)

Even allowing the authors to analyze 5--year mortality patterns for
1--year dosing in the randomized patients, their own confidence limits
show that the data are consistent with a 22\% increase in hazard from
nifedipine.  The 1--year results are more pessimistic: The point
estimate for the 1--year mortality ratio (0.059/0.05) indicates an
18\% increase in mortality, and an upper 0.95 confidence limit for
this mortality increase would show that the 1--year data are
consistent with an increase in the probability of death by 1 year that
is much higher than 18\%.  And this is for a low dose of nifedipine.

The SPRINT trial was designed as an efficacy trial and not as an
equivalence study.  Equivalence studies generally require
significantly higher sample sizes, and equivalence cannot be
demonstrated using standard hypothesis tests about a null treatment
effect.  Equivalence (or actually ``equal or better'') can be
inferred using the upper confidence limit, by using a Bayesian
approach, or by using a hypothesis test where the null treatment
effect is not assumed to be zero.  The pitfalls of the RBBMKG analysis
can be summed up in the title of Altman and Bland's
excellent editorial ``Absence of evidence is not evidence of
absence'' \cite{alt95abs}.

\bibliographystyle{abbrv}
\bibliography{/doc/survrisk/feh}
\end{document}



