21 Reproducible Research

Disconfirmation bias: giving expected results a relatively free pass but rigorously checking non-intuitive results

An excellent article on how to do reproducible research is Munafò et al. (2017) for which the pdf file is openly available. The link to an excellent video by Garrett Grolemund is on the right.

This article by Gelman discussed the role that statistical “significance” plays in non-reproducible research.

21.1 Non-reproducible Research

  • Misunderstanding statistics
  • Investigator moving the target
  • Lack of a blinded analytic plan
  • Tweaking instrumentation / removing “outliers”
  • Floating definitions of response variables
  • Pre-statistician “normalization” of data and background subtraction
  • Poorly studied high-dimensional feature selection
  • Programming errors
  • Lack of documentation
  • Failing to script multiple-step procedures
    • using spreadsheets and other interactive approaches for data manipulation
  • Copying and pasting results into manuscripts
  • Insufficient detail in scientific articles
  • No audit trail

21.2 General Importance of Sound Methodology

21.2.1 Translation of Research Evidence from Animals to Humans

  • Screened articles having preventive or therapeutic intervention in in vivo animal model, \(> 500\) citations (Hackam & Redelmeier (2006))
  • 76 “positive” studies identified
  • Median 14 years for potential translation
  • 37 judged to have good methodological quality (flat over time)
  • 28 of 76 replicated in human randomized trials; 34 remain untested
  • \(\uparrow\) 10% methodology score \(\uparrow\) odds of replication \(\times\) 1.28 (0.95 CL 0.97-1.69)
  • Dose-response demonstrations: \(\uparrow\) odds \(\times\) 3.3 (1.1-10.1)

Note: The article misinterpreted \(P\)-values

21.2.2 Other Problems

  • Rhine and ESP: "the student’s extra-sensory perception ability has gone through a marked decline’’
  • Floating definitions of \(X\) or \(Y\): association between physical symmetry and mating behavior; acupuncture
  • Selective reporting and publication bias
  • Journals seek confirming rather than conflicting data
  • Damage caused by hypothesis tests and cutoffs
  • Ioannidis: \(\frac{1}{3}\) of articles in Nature never get cited, let alone replicated
  • Biologic and lab variability
  • Unquestioning acceptance of research by the “famous”
    • Weak coupling ratio exhibited by decaying neutrons fell by 10 SDs from 1969–2001

21.2.3 What’s Gone Wrong with Omics & Biomarkers?

  • Gene expression-based prognostic signatures in lung cancer: Ready for clinical use? (Subramanian & Simon (2010))
  • NSCLC gene expression studies 2002-2009, \(n \geq 50\)
  • 16 studies found
  • Scored on appropriateness of protocol, stat validation, medical utility
  • Average quality score: 3.1 of 7 points
  • No study showed prediction improvement over known risk factors; many failed to validate
  • Most studies did not even consider factors in guidelines
    • Completeness of resection only considered in 7
    • Similar for tumor size
    • Some only adjusted for age and sex

21.2.4 Failure of Replication in Preclinical Cancer Research

  • Scientists at Amgen tried to confirm published findings related to a line of research, before launching development
  • Identified 53 ‘landmark’ studies
  • Scientific findings confirmed in only 6 studies
  • Non-reproduced articles cited far more frequently than reproduced articles

Begley CG, Ellis LM: Raise standards for preclinical cancer research.
Nature 483:531-533; 2012 Natural History of New Fields

Each new field has a rapid exponential growth of its literature over 5-8 years (“new field phase”), followed by an “established field” phase when growth rates are more modest, and then an “over-maturity” phase, where the rates of growth are similar to the growth of the scientific literature at large or even smaller. There is a parallel in the spread of an infectious epidemic that emerges rapidly and gets established when a large number of scientists (and articles) are infected with these concepts. Then momentum decreases, although many scientists remain infected and continue to work on this field. New omics infections continuously arise in the scientific community.

21.3 System Forces