Michael Cibulka <mtcibulka@gmail.com> LTTE

In order to study the performance of the authors' data analysis strategy, we simulated 500 datasets that are very similar to the 80-subject dataset used in the paper, except that all of the data were random, with no true associations of any of the 46 candidate variables with the binary outcome.  31 of the random variables were simulated binary predictors and the remaining 15 were normally distributed continuous variables, the same setup as in the paper.  49 subjects were "successes" and 31 subjects were "failures" for each of the 500 simulations.  

The mean number of candidate predictors passing the P < 0.1 univariable screening phase was 3.5.  In one of the 500 simulated datasets, 10 variables were significant.  The mean number of candidate predictors in the final logistic model was 2.7, with one model having 8 "significant" predictors.  Recall that none of the predictors were in fact related to the outcome, by the simulation design.  More importantly, the authors' strategy resulted in models with apparent predictive accuracy even when no signal is present.  The mean area under the ROC curve was 0.73 and the mean Nagelkerke R^2 was 0.22.  One random model had an apparent ROC area of 0.92 and one had an R^2 of 0.62.  The 75th percentile of ROC areas was 0.78 and of R^2 was 0.30.

When it is not difficult to develop a somewhat discriminating model
using a proposed strategy when absolutely no signal is present, the
strategy is deficient.  Quite simply, univariable screening and stepwise variable selection, like virtually all strategies using statistical measures of association to guide model selection, are almost never appropriate strategies.  This is the classic "double dipping" problem (ref below).

Upon further inspection we found other serious problems with the statistical approach used in the paper.  The original outcome variable was dichotomized, which is generally an invalid approach (Fedorov ref.).    The authors did not correct the Hosmer-Lemeshow test for overfitting.  Cutoffs for computing sensitivity, specificity, and likelihood ratios were derived from the same dataset used to compute these measures, resulting in overstatement of results.


@Article{fed09con,
  author = 		 {Fedorov, Valerii and Mannino, Frank and Zhang, Rongmei},
  title = 		 {Consequences of dichotomization},
  journal = 	 PS,
  year = 		 2009,
  volume = 	 8,
  pages = 	 {50-61},

@Article{kri09cir,
  author = 		 {Kriegeskorte, Nikolaus and Simmons, W. Kyle and Bellgowan, Patrick S. F. and Baker, Chris I.},
  title = 		 {Circular analysis in systems neuroscience: the dangers of double dipping},
  journal = 	 NN,
  year = 		 2009,
  volume = 	 12,
  number = 	 5,
  pages = 	 {535-540},