Time for Bayes in the Era of Flexible Therapeutic Research
Frank E. Harrell Jr.  and Donald A. Berry 

In many scientific disciplines such as physics, investigation is based on experimentation until sufficient evidence is obtained about an effect of interest. In contrast, biomedical research, especially clinical research, has traditionally been based on making assumptions about the magnitude of the effect one wishes to detect and the inter-subject variability, setting the probability of “false positive” and “false negative” findings, and then computing a sample size. The sample size is usually thought of as a constant. The idea of a physicist experimenting until she has “the answer” is foreign to most clinical trialists and clinical statisticians. For these reasons, the 10 July 2006 speech by Scott Gottlieb, Deputy Commissioner for Medical and Scientific Affairs, Food and Drug Administration [1] is welcomed news that we expect to have far reaching implications for conducting biomedical research in a more flexible, efficient, and ethical fashion. A large number of highly underutilized experimental designs and inferential techniques exist to meet the needs of flexible therapeutic research.
Some of the ways in which modern experimental designs can come into play include adding and dropping treatment arms after the study has begun, adding a new endpoint, stopping a study early because of a large difference between treatments, increasing (from zero) the probability that certain types of subjects are not randomized, extending an equivocal study, changing doses, and tuning covariate adjustment to better account for heterogeneity in patient responses. Adaptive randomization, in which responses from previously randomized subjects are used to modify the treatment allocation ratio, is one of the most dynamic approaches. One could even say in adaptive studies that a secondary endpoint is the final fraction of subjects randomized to a specific treatment. Flexible trials can respond to new information from other studies. Fully flexible designs allow statisticians to finally admit that sample size estimation is just a game (and expert clinicians to admit they really don’t know the correct dose to use), although these designs can frighten budget directors. Adaptive randomization can better address ethical issues surrounding the exact meaning of equipoise when writing an informed consent document, because most gains in knowledge are gradual and continuous rather than coming in large discrete chunks; the evidence base increases during a trial, which is at odds with the static text found in most informed consents.
The frequentist approach to statistics, and frequentist statisticians, have had an enormously important impact on clinical research and regulatory standards. As recently as 50 years ago our understanding of good medical practice was driven largely by anecdote and case studies. In large part because of the influence of frequentist ideas, and the associated extreme rigor, modern medicine now is appropriately called a science. Moreover, we have achieved a plateau of understanding regarding what constitutes "good science." This is both comforting and frustrating. It is frustrating because of the difficulty in moving a field from a place where its leaders are comfortable in the knowledge that they are doing good things. But improvements and advances are possible, and we must explore the range of possibilities. For this reason the recent presentation by Scott Gottieb is most refreshing. Quite generally, the FDA’s Critical Path is a ramp that can lead to a level substantially above the present plateau. 
Although good experimental design and rigorous statistical analysis has advanced research on therapeutics, it has become increasingly apparent in the past decade that traditional frequentist approaches to study design and assessment of evidence are not optimal scientifically. Many a fixed sample size study with simple treatment allocation has resulted in equivocal results. In some of these cases, extending the study, dropping treatment arms, changing doses, and otherwise altering the study in midstream would have helped sponsors and the academic community better optimize treatment strategies for groups of patients and for individuals. And even though a large number of clinical trials utilize group sequential methods for possible early termination for reasons of safety or extreme efficacy, or employ multiplicity adjustments for multiple endpoints, such approaches are mathematically complex, insufficiently flexible, and have an element of arbitrariness. For example, when multiple hypotheses are tested (e.g., superiority on one endpoint or non-inferiority on another), there is no unique solution to the optimal testing procedure. In group sequential trials, some planning on the frequency of “looks” at the data or the rate of spending type I error (a) must be done beforehand, and there is an a-penalty for multiple looks. With regard to extending a study, there is no consensus among statisticians in how P-values are adjusted for multiple looks. Sample size re-estimation, a current fad in biostatistics, does not have a solid theoretical basis in the frequentist world.
The difficulty in all of these examples is caused by emphasis on “backwards” probabilities, i.e., probabilities of observing results as or more extreme as the study results if the null hypothesis were true. Nonstatisticians seldom seek backward probabilities but rather forward ones such as the probability that a patient on drug A has a longer life expectancy than a patient on drug B. Feinstein [2] believed that embracement of (backward) P-values “...is a lamentable demonstration of the credulity with which modern scientists will abandon biologic wisdom in favor of any quantitative ideology that offers the specious allure of a mathematical replacement for sensible thought.” P-values condition on what is unknowable (the complete absence of an effect) and not what is known (the study data). They involve the consideration of data other than those obtained; the treatment effect is fixed at zero and a universe of data distributions is entertained. Significance levels entail “what would have occurred following results that were not observed at analyses that were never performed” [3]. To state this as a sports analogy, a watcher of an American football game who is considering whether to turn off the TV with his team ahead by 10 points at the end of the third quarter is interested in the probability his team will win the game. An optimum decision is reached by constructing a model of this setting, one that is informed by data. The model would allow calculation of the probability of victory when the lead is 10 with 15 minutes to go, to obtain a “forward” probability. A frequentist hypothesis tester on the other hand might consider all games ending in a tie or loss for his team, computing the probability of being ahead by 10 points 15 minutes before the end. If the viewer wanted to make repeated decisions about stopping, the frequentist would be forced to compute the probability of being 10 points or more ahead at some point in the game, given a loss or tie at the end. The schedule of “looks” influences these calculations, but the Bayesian viewer could compute the probability of victory every millisecond without altering the meaning of each of these probabilities. In statistical terms, Bayesian calculations are made in the “parameter space” and frequentist hypothesis testing involves the (all too theoretical) “sample space” that must consider fixed values of all parameters and investigator and analyst intentions not carried out. Repeated assessment of evidence over time cannot result in sampling to a foregone conclusion because of the martingale property of Bayesian posterior probabilities. The only dishonest Bayesian analyses would involve collection of new information and then refusing to use it because it is unfavorable, changing the prior distribution after seeing the data, or changing the statistical model to obtain some desired answer. Prespecification of prior and model, or at least prespecifying the recipe for building the model are required for scientific validity.
In a sense the Bayesian approach trades a large set of arguments related to arbitrariness of type I error calculations, non-uniqueness of multiplicity adjustments to P-values, and exact definition of the universe of sample space calculations for arguments over the choice of the statistical model and the state of knowledge before the experiment began (prior distribution for the parameter(s) of interest, e.g., treatment effect). Unless there is extensive knowledge about the treatment effect before the experiment (i.e., the prior distribution is sharp), information in the current data overwhelm information from the prior as the sample size becomes large, making the choice of prior less important. In the absence of pertinent prior knowledge, there are several reasonable approaches to selection of priors including using prior distributions that would convince a reasonable skeptic [4]. In exchange for specifying a prior probability distribution for the treatment effect, the Bayesian approach allows direct calculation of forward probabilities such as the probability of efficacy.
The elegance of the Bayesian approach is accompanied by its ability to solve real-world, complex problems. For example, to summarize current evidence that a new drug is either clinically superior to standard therapy on endpoint A or non-inferior on endpoint B, one could directly calculate the posterior (better termed current) probability that the drug is at least 10% better then control on A or no more than 5% worse on B. In practice can be done using a technique such as Gibbs sampling wherein tens of thousands of bivariate samples of the treatment effects on both endpoints are generated in a few seconds conditional on the current data, and one merely computes the proportion of these hitting the clinical target [5], as each patient is accrued, at each death, or at each hour. In a three endpoint study, one can likewise easily compute the probability that a new drug is better than control on at least two endpoints or that it is dramatically better on at least one endpoint. Such calculations are exceedingly difficult and arbitrary in the frequentist domain. An accessible article about Bayesian methods may be found in [6].
Full flexibility cannot be obtained in the frequentist paradigm. With the increased interest of the FDA and some pharmaceutical companies on flexible study design and execution, it is time to fully embrace the Bayesian approach. Certainly, challenges remain [7], including unblinding, more complex study execution, and more investment in statistical design and specification of objective statistical analysis plans, with a reduction in the use of “off the shelf” designs and analysis plans. Great care is necessary in designing adaptive trials so that we do not sacrifice scientific rigor. For example, since treatment assignments or treatment arms may change during an adaptive trial, the fact that one treatment arm seems to be assigned more frequently may be observed by treating physicians and patients who are candidates for the trial. Such information may change the patient population or concomitant treatment in such as way as to bias the results and potentially threaten the integrity of the trial. Adaptive Bayesian designs tend to be complicated and so there is more room for error than in conventional balanced randomized trials. Moreover, the experience with and understanding of adaptive designs by medical researchers are limited. These researchers have learned to base their conclusions on evidence. The substantial efficiencies from using Bayesian approaches will not be accepted generally until we are able to demonstrate them in actual clinical trials, while at the same time demonstrating that we have not sacrificed scientific rigor. 
Designing a Bayesian clinical trial requires substantial upfront effort, and clinical biostatisticians at pharmaceutical companies will need to assume new levels of responsibility. For example, in a seamless phase II/III drug trial, the design team has to prospectively identify the trial results that would lead to moving to phase III. Some sponsors worry that their decision making responsibility would be assumed by the trial’s Data Monitoring Committee (DMC). But this is not so. With advance planning the DMC serves as an automaton, ensuring that the trial proceeds as per protocol (unless unanticipated safety concerns develop). This means that the protocol has to seriously address actions to take in all eventualities. This is work, which would seem to be a negative. However, in our experience, this advance effort has payoffs beyond enabling an efficient design. Namely, the process of analyzing the potential data before they occur serves to better define the purposes of and consequences of the trial. And it leads to better understanding of the different points of view among members of the design team. So designing an adaptive Bayesian trial has a net benefit even if it is not run!  We believe that additional upfront investments (which constitute a small part of study budgets anyway) will be paid for many times over by the resulting flexibility and potential for decreasing the number of subjects exposed to inferior treatments.
Study design and analysis issues discussed above were focused on efficacy. There are many important issues relating to safety assessment in clinical trials. Regardless of whether a statistician is a Bayesian or frequentist, it is safe to say that clinical chemistry, hematology, urinalysis, electrocardiogram, and adverse event data are some of the most poorly analyzed data in clinical research. We hope that the Critical Path initiative will begin a push for efficient and comprehensive analysis of clinical safety data. For example, the other principal focus (besides innovative trial design) of the Critical Path is using biomarkers, which could be better used to identify patients who are particularly susceptible to serious adverse effects. But before an enormous effort is put into the discovery of new safety biomarkers, some of which can be used to modify ongoing trials, existing inexpensive clinical safety data should be fully analyzed. It is time to cease reliance on such oversimplifications as computing the proportion of patients with a liver function parameter exceeding three times the upper limit of normal separately at each followup visit, an approach that has horrendous statistical performance. For truly binary data such as many types of adverse events, new Bayesian multilevel modeling approaches [8] can increase sensitivity in detecting safety signals, especially thematic ones dispersed across many rare adverse events. Adaptive randomization can result in a small fraction of subjects ultimately being placed in one of the treatment arms. As the most meaningful safety assessments are comparative rather than absolute ones, it will be even more important in the future to analyze safety data efficiently.
One of the side effects of adoption of flexible clinical trial design and analysis is the necessity for sponsors to utilize a much wider array of software than is currently used at pharmaceutical companies. For example, commercial statistical packages do not provide the tools needed for designing or properly analyzing adaptively randomized trials. We see a need for sponsors to make full use of the wide array of free analytic software that is already available (e.g., BUGS [9] and R [10]), for academic statisticians to make modern open source software tools available to meet the needs of flexible clinical research, and for sponsors to share their experiences in dealing with the FDA regarding such software use as well as regarding the larger issues of design and execution of flexible clinical trials.  
References
[1]	Scott Gottlieb. Speech before 2006 Conference on Adaptive Design. http://www.fda.gov/oc/speeches/2006/trialdesign0710.html, July 10 2006.
[2]	Alvan R. Feinstein. Clinical Biostatistics. C. V. Mosby, St. Louis, 1977.
[3]	Scott S. Emerson. Stopping a clinical trial very early based on unplanned interim analysis: A group sequential approach. Biometrics, 51:1152–1162, 1995.
[4]	David J. Spiegelhalter, Laurence S Freedman, and Mahesh K. B. Parmar. Bayesian approaches to randomized trials. J Roy Stat Soc A, 157:357–416, 1994.
[5]	Frank E. Harrell and Tina Shih. Using full probability models to compute probabilities of actual interest to decision-makers. International Journal of Technology Assessment in Health Care, 17:17–26, 2001.
[6]	Donald A. Berry. Bayesian clinical trials. Nat Rev, 5:27–36, 2006. Editorial, p. 3.
[7]	Scott S. Emerson. Issues in the use of adaptive clinical trial designs. Stat Med, 25:3270–3296, 2006.
[8]	Scott M. Berry and Donald A. Berry. Accounting for multiplicities in assessing drug safety: A three-level hierarchical mixture model. Biometrics, 60:418–426, 2004.
[9]	W. R. Gilks, A. Thomas, and D. J. Spiegelhalter. A language and program for complex Bayesian modeling. The Statistician, 43:169–177, 1994. http://www.mrc-bsu.cam.ac.uk/bugs.
[10]	R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria, 2005. ISBN 3-900051-07-0, http://www.R-project.org.