Tom Louis 2020-06-01

I agree with everything you write in the context of strict Bayesian analysis, but want to add two points; one regarding prior elicitation and one using the Bayesian formalism but not strict Bayes; both using “backwards Bayes.”

Prior elicitation: It can be effective to present datasets to experts, describe the priors that would lead to various decisions and ask which subset of these priors seem “reasonable.” Doing this for several, created datasets can help home in on each elicitee’s prior or priors that can be used in the analysis. So, this is Bayesian design in that the prior is part of the design.

Threshold analysis (see the attachment): Not strictly Bayes, but very much in the spirit of threshold utility analysis, for a given dataset, map out the classes of priors lead to various decisions (e.g., reject or not) as a data summary. Then, use these to focus decision-making. I first saw the approach in Mosteller & Wallace disputed authorship of a subset of the Federalist papers. Their “backwards Bayes” summary went something like, “Unless your prior probability of the author being Hamilton was greater than 0.99, the data will persuade you that the author was Madison.” [I may have Hamilton and Madison switched, but you get the point.]