Predicting 30-day mortality after hip fracture surgery

Data sources

The study used data collected by the NHFD as part of the Royal College of Physicians and Association of Anaesthetists of Great Britain and Ireland collaborative ‘sprint’ audit on anaesthetic practice. The Anaesthetic Sprint Audit of Practice (ASAP) ran in parallel with standard NHFD data collection over three months in 2013 at 95 hospitals in England, Wales and Northern Ireland.¹⁴ During this time, hospitals submitted data on additional audit fields beyond the standard NHFD dataset, which enabled the estimation of individual patients’ risk of 30-day mortality using both NHFD and Nottingham models.

Ethical approval was not required since the NHFD has Section 251 approval from the Health Research Authority’s Confidentiality Advisory Group to collect details of hip fracture patients and link their data with the Office for National Statistics (ONS) death register.

Patient population

The study included patients who were admitted for hip fracture, between 1 May 2013 and 31 July 2013, to a NHS hospital in England, Wales or Northern Ireland that participated in ASAP. Patients with known operation type were selected for analysis if they were aged between 60 and 110 years, so long as their mortality status at 30 days after surgery could be confirmed by linking their record with the ONS death register.

The risk models

The NHFD model contains six variables:² age; gender; American Society of Anesthesiologists (ASA) physical status grade; ability to walk indoors; fracture type; and source of admission. The Nottingham model contains seven variables: age; gender; number of comorbidities; abbreviated mental test score on admission; haemoglobin concentration; living in an institution; and malignant disease. The Nottingham model was developed with age stratified into three categories (< 66, 66 to 85, and ⩾ 86 years). This study however used the slightly different categories that were applied in the ASAP study.⁷ These corresponded to the categories used in the NHFD model. There was also some overlap between the two models in the other patient risk factors that they contain (Table I).

Statistical analysis

We used a standard approach for assessing and improving prediction models¹⁵ whereby a model is refined in one dataset before its performance is examined in a different group of patients.

We used a ‘development’ dataset of patients admitted between 1 May 2013 and 15 June 2013 to recalculate (recalibrate) the coefficients of the risk factors in the NHFD and Nottingham models. Each patient’s probability of death within 30 days of surgery for hip fracture could then be predicted from the regression coefficients of the two models.^16,17

The resulting risk equations for the two models were applied to the ‘validation’ dataset of patients admitted between 16 June 2013 and 31 July 2013 to examine their predicted risk of death.^16,17

The ‘discrimination’ of a model describes its ability to differentiate between patients who survived or died using the area under the receiver-operating characteristic (ROC) curve. We interpreted values of this ‘c-statistic’ as indicating poor model performance if they were below 0.70, as moderate for 0.70 to 0.79, and as good performance for 0.80 to 0.89.^12,18

The ‘calibration’ of a model compares the predicted and observed mortality rates between groups of patients who are grouped based on their predicted mortality risk. We assessed calibration visually and with the Hosmer-Lemeshow test, with patients allocated to eight predicted risk groups. Goodness of fit was considered adequate if the p-value was less than 0.05.¹⁹

We addressed missing data using multiple imputation by chained equations (MICE), that is, missing data assumed to be missing at random. Rubin’s rules were applied to produce 20 imputed datasets.²⁰ All analyses were carried out using STATA version 14.1 (StataCorp LP, College Station, Texas).

Model recalibration and validation

The development dataset was used to derive coefficients for the NHFD and Nottingham models and produce the following risk scores:

NHFD risk score = exp (- 4.723 + (0.435 x age 70 to 79) + (0.518 x age 80 to 89) + (1.060 x age 90 to 110) + (0.894 x male) + (-0.275 x not admitted from home) + (1.164 x ASA 3) + (1.933 x ASA 4 or 5) + (0.358 x walk with at least one aid) + (-0.055 x extracapsular fracture)).

Nottingham risk score = exp (- 3.955 + (- 0.407 x age 70 to 79) + (0.485 x age 80 to 89) + (1.083 x age 90 to 110) + (0.883 x male) + (0.558 x Hb ⩽ 10 g dl⁻¹) + (0.662 x cancer) + (- 0.537 x AMTS ⩾ 7) + (0.263 x living in institution) + (0.755 x ⩾ 2 comorbidities)).

The NHFD and Nottingham models displayed similar levels of performance (Table III). Both were moderately able to distinguish between patients at high and low risk of death within 30 days, with a c-statistic of 0.71 for the NHFD model and 0.70 for the Nottingham model in the validation dataset.

Model calibration was similar in both models, with a slightly reduced range of predicted risk in the validation dataset (Fig. 2). The Nottingham model displayed a better fit (p-value = 0.599 for Hosmer-Lemeshow test) than the NHFD model (p-value = 0.029), but both were inconsistent in their estimations of mortality risk for patients in the eight risk groups (Fig. 2).

Fig.

Observed versus predicted 30-day mortality by risk group, Nottingham and National Hip Fracture Database (NHFD) models. Figure 2a) shows the development dataset, while b) shows the validation dataset.

Main findings

Case-mix adjustment is important in the development and refinement of prediction models, as potential confounding caused by differences in the populations of individual hospitals must be addressed. The risk model used by the NHFD to adjust 30-day mortality rates for individual hospitals features the same six patient factors as previous NHFD risk models, but with different categories for some variables, and is based on a regression equation rather than a classification tree.²¹

In this evaluation of the NHFD risk model, its performance was comparable with one of the most commonly used and well validated outcome prediction tools for hip fracture – the Nottingham Hip Fracture Score. Both models displayed moderate discriminative power, with the NHFD model achieving a slightly higher c-statistic of 0.71 in the validation dataset. The two models also displayed moderate levels of calibration, although there was some inconsistent estimation of mortality risk for patients in different risk groups when applied to the validation dataset. These findings are consistent with studies that have found limited goodness of fit for the Nottingham model and the model described by Holt et al.^5,10

Strengths and limitations

Previous studies have excluded certain patient groups, such as those who underwent total hip arthroplasty,⁵ but this study included patients who had any type of surgery for hip fracture. The findings could therefore be applied to the whole hip fracture population. The study has several other strengths. It used better quality data than earlier validation studies of the Nottingham model because a high level of data completeness was achieved in the ASAP, where hospitals were excluded if their case ascertainment was 80% or less. We adopted the more robust approach of multiple imputation to manage missing data,²² while other validation studies limited themselves to complete case analysis by excluding records with missing data.^6,10

Validation of the Nottingham model using the original and subsequently recalibrated risk score was not feasible as the regression coefficients used to calculate the current Nottingham Hip Fracture Score are not publicly available.⁸ Comorbidities were only recorded in the ASAP dataset in terms of whether they were present or not, so it was not possible to determine whether patients without specific recorded comorbidities truly did not have those conditions.

Finally, both models include a selection of patient characteristics associated with death following surgery for hip fracture,²³ but they do not capture all potential factors of interest. Future refinement of either model should be guided by the availability and reliability of recorded variables.

Implications for further research and clinical practice

This study has demonstrated that the NHFD model performs comparably with the Nottingham model, and meets current standards for hip fracture outcome prediction in the United Kingdom hip fracture population. Therefore, the NHFD model could be considered for use as an alternative to the Nottingham model, especially if variables (such as haemoglobin concentration or the nature and number of comorbidities) are poorly recorded or missing.

There is room for improvement in both models. When used to calculate risk-adjusted outcomes of hospitals, there is a small chance of wrongly classifying a hospital as an outlier. However, compared with making predictions for individual patients, the size of the error will be small. This is because the risk adjustment process aims to estimate the average risk of all patients treated at different hospitals, and the differences in the average risks among hospitals is much less than the differences in the risks among individual patients.

If these models are to be used to predict mortality at the individual patient level, we suggest that they be further refined and validated. For example, regular recalibration of both models is appropriate given the trend in decreasing 30-day mortality rate over recent years (7.1% in 2015 compared with 8.0% in 2013).^1,13 Similarly, changes to the profile of hip fracture patients will need to be taken into account in future updates to the models.²⁴ However, the risk of death faced by an individual patient cannot be fully defined by any model based on risk factors present on admission, since it will obviously also depend on the quality of care they subsequently receive. The importance of prompt surgery and peri-operative orthogeriatric assessment are just two examples of aspects of care that impact on patients’ mortality risk.

While mortality is the most commonly measured patient outcome, there are many other outcomes relevant to the assessment of care delivered to hip fracture patients. For example, functional independence and quality of life in the intermediate future following hip fracture are of interest and importance to patients and their families, care providers and policy makers. Future studies would benefit from assessing the performances of these tools in their ability to predict differences among patients on these outcomes. This would lead to a better understanding of how different patient groups (such as those at relatively high risk or low risk of death within 30 days of surgery) are affected by hip fracture in the months after completing hospital treatment.