External validation of the SAPS II, APACHE II and APACHE III prognostic models in South England: a multicentre study

Objective: External validation of three prognostic models in adult intensive care patients in South England. Design: Prospective cohort study. Setting: Seventeen intensive care units (ICU) in the South West Thames Region in South England. Patients and participants: Data of 16,646 patients were analysed. Interventions: None. Measurements and results: We compared directly the predictive accuracy of three prognostic models (SAPS II, APACHE II and III). using formal tests of calibration and discrimination. The external validation showed a similar pattern for all three models tested: good discrimination, but imperfect calibration. The areas under the receiver operating characteristics (ROC) curves, used to test discrimination, were 0.835 and 0.867 for APACHE II and III, and 0.852 for the SAPS II model. Model calibration was assessed by Lemeshow-Hosmer C-statistics and was X-2 =232.1 for APACHE II, X-2 =443.3 for APACHE III and X-2 =287.5 for SAPS II. Conclusions: Disparity in case mix, a higher prevalence of outcome events and important unmeasured patient mix factors are possible sources for the decay of the models' predictive accuracy in our population. The lack of generalisability of standard prognostic models requires their validation and re-calibration before they can be applied with confidence to new populations. Customisation of existing models may become an important strategy to obtain authentic information on disease severity, which is a prerequisite for reliably measuring and comparing the quality and cost of intensive care.