Overview of Risk Adjustment for Provider Profiling

Frank Harrell
Department of Biostatistics
Vanderbilt University School of Medicine

Basis for Provider/Hospital Profiling

A Note on the Medicare Star System

Methods of Risk Adjustment

Indirect Adjustment

Direct Adjustment

Comparison of Direct and Indirect Adjustment

Indirect adjustment through the use of observed to expected estimates has a long history but was never a good idea. It is overconfident by not taking uncertainty of coefficient and risk estimates into account. But more serious is the failure to answer the right question, which can result in inadequate “leveling of the playing field” and unreliable outcome quality rankings of providers. For mortality reporting for hospitals, the expected number of deaths for a hospital is the sum of all the predicted mortality risks for the patients in that hospital. For example, if there were only 3 patients, one with a risk of 0.5 and the other two with a risk of 0.25, the expected number of deaths from among the three patients is 1.0. The expected risks come from pooling all data from all hospitals and fitting, for example, a binary logistic regression model. Expected risk are pooled estimates averaged over all hospitals (ironically, including the hospital being evaluated, but this matters little when the number of hospitals is very large). The use of expected number of deaths does not answer the question of what happened to the higher risk (0.5) patient vs. what happened to the lower risk (0.25) patients. It answers the question of what is the expected number of deaths were all the single hospital’s patients reassigned at random to different hospitals across the country. Most importantly, it does not answer the question that is most needed in medical decision making: what is likely to happen to the current patient were she treated at another hospital vs. would would happen to her at the hospital in question.

Contrast that with direct adjustment that includes random effects for all hospitals in a single unified model. This is a patient-specific model that allows one to predict what would happen to an individual patient were she treated at hospital y instead of hospital x. When a patient has the option of choosing a hospital, this is precisely the input needed. Observed to expected ratios average over all types of patients, resulting in estimates that may not apply to any individual patient, and through Simpson’s paradox may not even rank hospitals correctly on the average.

Observed:expected (standardized mortality ratios) may sometimes be useful for selecting optimum care for a diverse batch of patients, but not for individual patients.