In many disciplines such as physics, scientific investigation is based on experimentation until sufficient evidence is obtained about an effect of interest. In contrast, biomedical research, especially clinical research, has traditionally been based on making assumptions about the magnitude of the effect one wishes to detect, the inter-subject variability, and the probability of “false positive” and “false negative” findings, then computing a sample size. The sample size is often thought of as a constant. The idea of experimenting until one has “the answer” is foreign to most clinical trialists and clinical statisticians. For these reasons, the 10 July 2006 speech by Scott Gottlieb, Deputy Commissioner for Medical and Scientific Affairs, Food and Drug Administration [4] is welcomed news that we expect to have far reaching implications for conducting biomedical research in a more flexible, efficient, and ethical fashion.

Some of the ways in which modern experimental designs can come into play include adding and dropping treatment arms after the study has begun, adding a new endpoint, stopping a study early because of a large difference between treatments, increasing (from zero) the probability that certain types of subjects are not randomized, extending an equivocal study, changing doses, and improving covariate adjustment to better account for heterogeneity in patient responses. Adaptive randomization, in which responses from previously randomized subjects are used to modify the treatment allocation ratio dynamically, is one of the most dynamic approaches. One could say in such studies that a secondary endpoint is the final fraction of subjects randomized to a specific treatment. Flexible trials can respond to new information from other studies. Fully flexible designs allow statisticians to finally admit that sample size estimation is just a game (and expert clinicians to admit they really don’t know the correct dose to use), although these designs can frighten budget directors. Adaptive randomization can better address ethical issues surrounding the exact meaning of equipois when writing an informed consent document, because most gains in knowledge are incremental rather than discontinuous; the evidence base increases during a trial.

Good experimental design and rigorous statistical analysis has advanced research on therapeutics, but it has become increasingly apparent in the past decade that traditional frequentist approaches to study design and assessment of evidence are not optimal scientifically. Many a fixed sample size study with simple treatment allocation has resulted in equivocal results. In some of these cases, extending the study, dropping treatment arms, changing doses, and otherwise altering the study in midstream would have helped sponsors and the academic community better optimize treatment strategies for groups of patients and for individuals. And even though a large number of clinical trials utilize group sequential methods for possible early termination for reasons of safety or extreme efficacy, or employ multiplicity adjustments for multiple endpoints, such approaches are mathematically complex, insufficiently flexible, and have an element of arbitrariness. As examples, consider multiplicity adjustment and the design of group sequential trials. When multiple hypotheses are tested (e.g., superiority on one endpoint or non-inferiority on another), there is no unique solution to the optimal testing procedure. In group sequential trials, some planning on the frequency of “looks” at the data or the rate of spending type I error (α) must be done beforehand, and there is an α-penalty for multiple looks. With regard to extending a study, there is no concensus among statisticians in how P -values are adjusted for multiple looks.

The difficulty in all of these examples is caused by emphasis on “backwards” probabilities, i.e., probabilities of observing results as or more extreme the the results of the study if the null hypothesis were true. These probabilities involve the consideration of data other than those obtained; the treatment effect is fixed at zero and a universe of data distributions is entertained. Computation of the probability of type I error entails “what would have occurred following results that were not observed at analyses that were never performed” [2]. To state this as a sports analogy, a watcher of an American football game who is considering whether to turn off the TV with his team ahead by 10 points at the end of the third quarter is interested in the probability his team will win the game. An optimum decision is reached by analyzing data on past games in which the lead is 10 with 15 minutes to go, to obtain a “forward” probability. A frequentist hypothesis tester on the other hand might consider all games ending in a tie or loss for his team, computing the probability of being ahead by 10 points 15 minutes before the end. If the viewer wanted to make repeated decisions about stopping, the frequentist would be forced to compute the probability of being 10 points or more ahead at some point in the game, given a loss or tie at the end. The schedule of “looks” influences these calculations, but the Bayesian viewer could compute the probability of victory every millisecond without altering the meaning of each of these probabilities. In statistical terms, Bayesian calculations involve the “parameter space” and frequentist hypothesis testing involves the (all too theoretical) “sample space” that must consider investigator and analyst intentions not carried out.

One could say that the Bayesian approach trades a large set of arguments related to arbitrariness of type I error calculations, non-uniqueness of multiplicity adjustments to P -values, and exact definition of the universe of sample space calculations for arguments over the choice of the statistical model and the state of knowledge before the experiment began (prior distribution for the parameter(s) of interest, e.g., treatment effect). Once the sample size is moderately large, the information in the current data overwhelm the information from the prior, making the choice of prior less important. Until that point, there are several reasonable approaches to selection of priors including using prior distributions that would convince a reasonable skeptic [7].

The elegance of the Bayesian approach is accompanied by its ability to solve real-world, complex problems. For example, to summarize evidence that a new drug is either clinically superior to standard therapy on endpoint A or noninferior on endpoint B, one would directly calculate the posterior (an ironic name, better termed current) probability that the drug is at least 10% better then control on A or no more than 5% worse on B. In practice this is done using a technique such as Gibbs sampling wherein tens of thousands of bivariate samples of the treatment effects on both endpoints are generated in a few seconds conditional on the current data, and one merely computes the proportion of these hitting the clinical target [5], as each patient is accrued, at each death, or at each hour. In a three endpoint study, one can likewise easily compute the probability that a new drug is better than control on at least two endpoints or that it is dramatically better on at least one endpoint. Such calculations are exceedingly difficult and arbitrary in the frequentist domain. And the only dishonest analyses done by a Bayesian would be failure to include new data in the probability calculations after results become less impressive, changing the prior distribution after seeing the data, or changing the model if the results are not favorable.

Full flexibility cannot be obtained in the frequentist paradigm. With the increased interest of the FDA and some pharmaceutical companies on flexible study design and execution, it is time to fully embrace the Bayesian approach. Certainly, challenges remain, including unblinding, more complex study execution, and more investment in statistical design and specification of objective statistical analysis plans, with a cessation of using “off the shelf” designs and analysis plans. Clinical statisticians at pharmaceutical companies will need to assume new levels of responsibility. The additional investment will be paid for many times over by the resulting flexibility and potential for decreasing the number of subjects exposed to inferior treatment arms.

Study design and analysis issues discussed above mainly target efficacy. There are many important issues relating to safety assessment in clinical trials. Regardless of whether a statistician is a Bayesian or frequentist, it is safe to say that clinical chemistry, hematology, electrocardiogram, and adverse event data are some of the most poorly analyzed data in clinical research. We hope that the Critical Path initiative will begin a push for efficient and comprehensive analysis of clinical safety data. Detection of early safety signals is imperative, and there is room for pharmaceutical companies to do a much better job in analyzing pure safety data. Before an enormous effort is put into the discovery of new safety biomarkers, some of which can be used to modify ongoing trials, existing inexpensive clinical safety data should be fully analyzed. It is time to cease reliance on such oversimplifications as computing the proprtion of patients with a liver function parameter exceeding three times the upper limit of normal separately at each followup visit, an approach that has horrendous statistical performance. For truly binary data such as many types of adverse events, new Bayesian multilevel modeling approaches [1] can increase sensitivity in detecting safety signals, especially thematic ones dispersed across many rare adverse events. Adaptive randomization can result in a small fraction of subjects ultimately being placed in one of the treatment arms. As the most meaningful safety assessments are comparative rather than absolute ones, it will be even more important in the future to analyze safety data efficiently.

One of the side effects of adoption of flexible clinical trial design and analysis is the necessity for sponsors to utilize a much wider array of software than is currently used at pharmaceutical companies. For example, commercial statistical packages do not provide any of the tools needed for designing or properly analyzing adaptively randomized trials. We see a pressing need for sponsors to make full use of the wide array of free analytic software that is already available (e.g., BUGS [3] and R [6]), and to join academia in developing modern open source software tools to meet the needs of flexible clinical research.

References

[1] S. M. Berry and D. A. Berry. Accounting for multiplicities in assessing drug safety: A three-level hierarchical mixture model. Biometrics, 60:418–426, 2004.

[2] S. S. Emerson. Stopping a clinical trial very early based on unplanned interim analysis: A group sequential approach. Biometrics, 51:1152–1162, 1995.

[3] W. R. Gilks, A. Thomas, and D. J. Spiegelhalter. A language and program for complex Bayesian modeling. The Statistician, 43:169–177, 1994. http:

//www.mrc-bsu.cam.ac.uk/bugs.

[4] S. Gottlieb. Speech before 2006 Conference on Adaptive Design. http:

//www.fda.gov/oc/speeches/2006/trialdesign0710.html, July 10 2006.

[5] F. E. Harrell and T. Shih. Using full probability models to compute probabilities of actual interest to decision-makers. International Journal of Technology Assessment in Health Care, 17:17–26, 2001.

[6] R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria, 2005.

ISBN 3-900051-07-0, http://www.R-project.org.

[7] D. J. Spiegelhalter, L. S. Freedman, and M. K. B. Parmar. Bayesian approaches to randomized trials. J Roy Stat Soc A, 157:357–416, 1994.