Spiegelhalter list of resource: https://wintoncentre.maths.cam.ac.uk/resources/resources-civil-servants-and-government-officials/ https://understandinguncertainty.org ---------------------------------------------------- Kappen, T.H. 2017-06-05 AttachmentsJun 5 (1 day ago) to K.G.M., Frank, Derek Dear Frank, Derek and Carl, I apologize for the delay, but the literature on this subject is pretty diverse, so I had to make sure first that I was still up to date and on the right track. There are many different views on risk perception/probability/decision making/judgment, but my personal opinion is that there is a lot of overlap between all these views. That being said, I will provide you with my personal (simplified) take on this problem to keep it comprehensible. To me your question is a comparison of either a) physicians estimations/decisions to a prognostic tool; or b) a prognostic tool to a physicians' estimations. In other words, do you want to improve the model or do you want to understand the physicians better? Trying to do both is not impossible, but this tends to make your study a whole lot more difficult. In the summary below, I will explain why I think this problem should be expressed by distinguishing probability from risk from decisions, and how this relates to a prediction tool's predictive probability and actual clinical decision making (of course with some appropriate literature). With this I hope to help you on your way, but please feel free to reach out when things are not clear. Also, if you don't agree with what I wrote, please mention it. I tried to explain it as concise as I could, so a lot of nuance may have been lost. Kind regards Teus P.S. I removed the CC-ed persons to prevent unnecessary mailbox pollution. I’ll leave it up to you to forward this email to them. SUMMARY Medical literature on 'risk perception' Within the medical literature there is no real nice summary (that I know of) on the methodology of risk perception research. There are definitely examples of risk perception research within the medical literature, but they mainly consider patients' risk perceptions, whereas just a 'handful' of studies study the perception of risk by healthcare workers. There are a lot of studies on physicians' perspectives that consider perception of a specific clinical problem rather than focusing on risk (e.g. 'Emergency Physicians' Perceptions and Decision-making Processes Regarding Patients Presenting with Palpitations'). So for a more theoretical and methodological view on risk perception, we'll have to look outside of medicine. Risk perception: hazard vs probability? To me the standard reference on risk perception would be 'The perception of risk' by Paul Slovic (2016, Routledge). The introduction of the book gives a nice overview on the 'risk perception', while Chapter 22 gives a nice example of how to study risk perception through vignettes and Chapter 23 nicely discusses how probability is related to risk. In this book, four different uses of 'risk' are distinguished: using it to mean a hazardous activity (‘bungee jumping is a serious risk’) to mean an adverse consequence (‘the risk of letting your parking meter expire is getting a ticket’) - which is not a true hazard to mean probability (‘What is the annual risk of death at age 80?’) - it might be used for a hazard or an adverse consequence, but in itself is a probability Risk as‘risk’ is a blend of the probability and the severity of consequences - it is in fact a blend of 'hazard', 'adverse consequence' and 'probability' This book generally considers the latter form, because its use is mainly to understand human decision making. There is a nice summary on qualitative vs quantitative methods in Sjöberg, Lennart. "The methodology of risk perception research." Quality & Quantity 34.4 (2000): 407-418. Probability perception: the role of intuition - or better - the role of decision making When trying to better understand or improve the prediction model, you are mainly focusing on probability perception - or - expert probability estimation. This often becomes a bit problematic, because definitions of risks are often mixed. For example, in medical literature it is not uncommon to compare very contextual human judgments or decisions (based on physicians' intuitions or 'gestalts' (sorry Carl) to decisions that are (almost) strictly based on a probability from a prediction tool. As I encountered during one of my studies, decision making is something completely different than probability estimation (Kappen, Teus H., et al. "Barriers and facilitators perceived by physicians when using prediction models in practice." Journal of Clinical Epidemiology 70 (2016): 136-145.). So now it hopefully becomes a bit more clear on why it is important to know who or what you want to study: when you want to understand a human, it is no real problem to compare the 'gestalt' to predictive probability: you simply have to understand the context, hazard and consequences of those decisions. However, when comparing human vs prediction models in terms of their predictive capabilities it becomes more problematic: although prediction models definitely can outperform humans in predictive performance, comparing gestalt to predicted probability is not the way to do this, because it is difficult (if not impossible) for a human to think without context or hazards. We would be better able to compare tool and expert probability estimations, when we can a better grasp of expert judgment in terms of probability. There is an entire field dedicated to this, stemming from the Bayesianist need to define a prior. A nice book on this is O'Hagan, Anthony, et al. Uncertain judgements: eliciting experts' probabilities. John Wiley & Sons, 2006, and some articles on background and pitfalls: 1) Statistical methods for eliciting probability distributions. PH Garthwaite, JB Kadane, A O'Hagan. Journal of the American Statistical Association 100 (470), 680-701) ; 2) Morgan, M. Granger. "Use (and abuse) of expert elicitation in support of decision making for public policy." Proceedings of the National Academy of Sciences 111.20 (2014): 7176-7184. Conclusion: from probability to decision I hope that what I wrote above is not too apprehensive. But I felt that this introduction was necessary to draw the parallel between probability/risk/decisions and the various types of studies one can do to study these. The distinction between human vs model (or computer) is in fact more of a distinction between probability vs decision - or - between theory and practice. And the types of studies vary in whether they come close to actual decisions or stick with probability. This is a highly simplified list that lists study types according how closely they can mimic actual practice: Expert calibration studies and other eliciting-studies on expert judgments. They use iterative approaches to get a 'calibration curve' for experts on how they estimate a probability. Although they use context to better frame the questions, this is really about probability. Surveys: you can ask about probabilities, but humans are not very good at that. So surveys that ask for probability estimates from humans in other ways than described in the 'eliciting approaches' are typically unreliable. You definitely can use surveys to ask about decisions, but there is a substantial risk of bias, because without proper context humans tend to give different answers than in a real-life decision. Vignettes: vignettes provide more context and are therefore much more reliable in getting to 'the real decision' behavior. However, people are typically only willing to accept a certain degree of complexity in a vignette. Matt Weinger and colleagues at Vanderbilt are very experienced in these types of studies. In-depth interviews: if done on itself typically these are very time-consuming and difficult to prevent bias. They can be used in conjunction with an intervention study (see 7) Real-time simulation: this approaches real-life decision making in a more complex and detailed way. Controlled decision experiments: often done in psychology, often unethical in medicine. And within psychology there are still some nice examples of study-introduced biased behavior. Understanding real-time decision making: often done as a mixed methods approach to a prospective observational study or an intervention study (see my Barriers and Faciliators study). One (quantitatively) observes human behavior and then asks questions on 'how' and 'why'. However, typically there is no time to go into details at the time of the decision (which might also influence the decision), so typically it is done (just) after the decision. This assumes that people can reflect on their decision and that they remember it correctly, which is - unfortunately - often not true. In other words, nothing is perfect, but these are more or less your options. Regarding intervention studies there are some new iterative approaches that one could consider, but I left this out for now. In attached the articles as PDF, the books can be found on Google Books.