From : semerson@uw.edu
To : "Frank Harrell"< fh@fharrell.com>,"Jonathan Siegel"< jonathan.siegel@bayer.com>
Date : Mon, 10 Aug 2020 08:38:11 -0500
Subject : Re: Your comments at the FDA WG estimands talk at JSM

Jonathan, Frank:

Based on both of your comments below, I think I am largely in agreement with each of you, and it would not surprise me that there is commonality between the two of you.

I tried to make distinctions between scientific questions (questions we are curious about) and questions that are answered scientifically. This was the distinction I was making in my question to Elena. We should definitely have important scientific questions, and it is the purview of the "sponsor" or principal investigator to decide what they are interested in. But when it comes to answering the question in a scientifically rigorous manner, I invoke Richard Feynman's view that it is incumbent on every good scientist to be their own worst critic. A scientific experiment is about deciding whether a testable hypothesis might be true, not proving that it is true. A comprehensive series of experiments might end up convincingly establishing that it is true.
But it is generally a sequence of experiments that are needed to try to eliminate alternative hypotheses sequentially. (I will spare you The Scientist Game that I use to illustrate how many scientists are not immediately good at removing confounding factors.)

In my classes I use the example of a perfectly good "scientific question" in that I am curious about the answer: Why is REM sleep necessary? The question was about REM in particular, so the issue was to have adequate sleep (whatever adequate might mean in this setting), just not REM sleep. An experiment was attempted in which subjects were awakened whenever they went into REM sleep. As the experiment was described to me (and these are recollections from medical school lectures, so I do not have references), the subjects soon started going into REM sleep as soon as they fell asleep. Hence, in the end, the question could not be answered through that experiment, because the factor of interest could not be separated out. Other approaches would have to be tried. For instance perhaps a drug could be found that blocked REM sleep. But then, we would want to be sure to separate effects of REM sleep deprivation from off-target effects of the drug. So maybe we want lots of different approaches to REM deprivation, and then try to examine commonality.

I think the above goes to the iterative approach that I think you are describing, and I mentioned such approaches in my talk as well, both within and across experiments.

But certainly in my example above, I would not be willing to try any sort of imputation to imagine what would have happened in the subjects who went into REM more rapidly had they behaved more like the people who perhaps had slightly longer delay in their changes in sleep patterns.

In my undergraduate physics curriculum, we often championed the ingenious methods that individual physicists used to derive experiments that were in fact convincing to their audiences. (Michelson-Morley comes to mind).

But I do think that we as statisticians provide a unique viewpoint, and I do spend a lot of my time in consulting trying to derive the RCT designs (it almost always takes more than one RCT) that will provide the regulatory evidence. And I regularly deal with "stakeholders" that I have not considered all the clinical and scientific issues. This is why collaboration on RCTs is so interesting. No single person has the full knowledge to find the best path forward. (I'll spare you my New Guinea highlander and the airplane comparison.)

But it is also true that many of the "stakeholders" are driven by strong personal bias (both financial and academic) that has them advocating for what are to my mind inappropriate paths. It is quite often the case that in my collaborations/consultations, clinicians make strong claims about "the most relevant clinical outcome", but then retreat quickly when asked about alternative physiologic and pathophysiologic mechanisms as well as the full spectrum of clinical behaviors.

And I think the ICH E-9 amendment gives many the impression that what I consider bad scientific method is OK. I very much advocate more for Frank's statement that it comes down to utilities. It is in fact very hard sometimes to weight the utilities of particular events in a way that will convince all in the audience, and that is why we sometimes have to resort to sensitivity analyses. And if we are lucky, the results will not be sensitive to alternative formulations, and we can feel confident that a wide spectrum of our scientific audience has been persuaded. (I invoke the need to use a spectrum of Bayesian priors to address the population of scientists--consensus priors don't cut it.)

I do very much appreciate your comments/questions.

Best,
Scott

______________________________
Scott S. Emerson, M.D., Ph.D.
Professor Emeritus of Biostatistics
University of Washington
(V) 206-459-5213    semerson@uw.edu
7055 54th Avenue NE, Seattle WA 98115

From: Frank Harrell <fh@fharrell.com>
Sent: Sunday, August 9, 2020 1:01 PM
To: Jonathan Siegel <jonathan.siegel@bayer.com>
Cc: Scott S Emerson <semerson@uw.edu>
Subject: Re: Your comments at the FDA WG estimands talk at JSM
 
Hi Jonathan - I didn't get to hear Scott's comments but have to throw in my own $.02 based on what I'm seeing at FDA and in the industry.  Some of the emphasis on estimands has turned into a math stat exercise where how patients value outcomes is being ignored.  I'm speaking especially about missing data situations where the main outcome of interest is interrupted by death or rescue therapy etc.   The marjority of these situations are IMHO not served by complex estimands that involve "what ifs" but rather using patient utilities or constructing ordinal outcome scales where treatment failures, death, etc. are place in appropriate places.

Frank
Twitter: @f2harrell







---- On Sun, 09 Aug 2020 13:14:30 -0500 Jonathan Siegel <jonathan.siegel@bayer.com> wrote ----

Dear Professor Emerson,

 

I have long been following your career, particularly your publications on issues in non-proportional hazards in group sequential designs.

 

You made some comments at Thursday afternoon’s estimands session that I may simply have misunderstood.

 

As I understood it, you remarked that many questions posed about clinical outcomes are unscientific questions, and they are unscientific because they can’t be reliably answered.

 

My view has always been that good science involves trying to understand both ones real problems – which are often complex and unsolvable, tractable only if we make simplifying assumptions that never completely hold – and ones tools, which are often tractable only because they make simplifying assumptions that don’t fully address the actual problem at hand.

 

So in my view, good science often involves identifying questions that are hard to answer with the available tools. Our situation is often like that of the fabled Wise Men of Chelm, who lost their keys in a dark alley, but are looking for them under a lamppost because it’s so much easier to see there. Positing that ones keys are in the dark alley is not bad science. Looking for ones keys under the lamp-post because that’s where it’s most reliable to look is not good science. And this is so even if it really is impossible to look in the dark alley at the moment. It is always possible that hypothesizing that the real problem lies in the dark alley and not under the lamppost may lead someone to invent the equivalent of a flashlight and hence the ability to find out reliably.

 

I note this because one of my difficulties with statistics as a profession has been that it often operates like a branch of mathematics rather than a science. It all too often operates deductively, starting with a set of tools and seeing what can be done with them, much as mathematics starts with a set of axioms and seeing what can be produced. Science operates inductively, starting with simple observation, and proceeding from simple description to developing and testing causal theories.

 

So from this view, focusing on whether a problem can be reliably solved as the primary criterion not only doesn’t define what science is, I’m not even sure it’s good science. (The term “Wise” in stories about the Wise Men of Chelm was used somewhat ironically). Our real problems will often be hard and sometimes impossible to solve. The applied statistician often has to take the simple tool that most closely approximates the complex problem at hand, and do so with eyes open – trying to understand both the real problem and the limitations of the tool.

 

In my talk on estimands in JSM 2019, I approached this issue by discussing a feedback loop between goals and feasible solutions analogous to the Deming Cycle. I believe clinical research often requires acknowledging that available solutions are rarely perfect fits for the real problems. But rather than deductively treating the available solutions as the given and characterizing misfits -- real problems for which there are no good solutions available as “bad science” -- the better course would be to treat the problems as the given, describe them accurately, see what available solutions most closely fit them, and then describe the problems that the available solutions are actually solving accurately, acknowledging that this will often be somewhat different from the problem that was posed. I understand that part of any consulting task is helping a client clarify the problem. Clients may not start out being able to state their problem clearly.

 

I think some humility is in order. Just because we’re statisticians doesn’t mean we’re the ones who understand the real problem. And our present humble and imperfect tools and their limitations are not the boundaries of science. It is always possible that someone will come up with a solution that solves something closer to the real problem at hand than anything available. If we are attentive to both the real problem and the limitations our tools, perhaps we will be the ones who can come up with a better solution. If we can’t, explaining the real problem clearly, even when we can’t solve it, may help someone else devise a solution. And even if that doesn’t happen,  it’s still good science to describe and understand what our problems are.

 

As Oliver Wendall Holmes put it, “I would not give a fig for the simplicity this side of complexity. I would give my right arm for the simplicity on the other side of complexity.” The applied scientist must accept unsolvable complexity as part of science in order to be able to find a path to the simplicity on the other side.

 

Sincerely,

 

Sincerely,

 

Jonathan

 


 


 

Jonathan Siegel

Director, Oncology Clinical Statistics US

_______________________

 

Bayer: Science For A Better Life

 

Bayer US

Oncology Clinical Statistics US

100 Bayer Blvd

Building B100, 1B2250

Whippany, NJ 07981

Tel:         +1 862-404-3497

Fax:        +1 862 404-3131

Mobile:    +1 862 309 3347

E-mail:    jonathan.siegel@bayer.com

Web:      http://www.bayer.com

 

 

 

 




The information contained in this e-mail is for the exclusive use of the intended recipient(s) and may be confidential, proprietary, and/or legally privileged.  Inadvertent disclosure of this message does not constitute a waiver of any privilege.  If you receive this message in error, please do not directly or indirectly use, print, copy, forward, or disclose any part of this message.  Please also delete this e-mail and all copies and notify the sender.  Thank you.