Assessing treatment outcomes using a single question

Patients and Methods

Over a five-year period (January 2007 to December 2011) we prospectively collected data for all patients undergoing primary total hip or knee replacement (THR and TKR) at a single orthopaedic centre. Regional ethics approval was obtained (ref: 11/AL/0079). Surgery was carried out by a multiplicity of consultant orthopaedic surgeons and their supervised trainees. All data was collected independently from the clinical teams by the associated university outcomes assessment unit.

As the intended use of this metric is to compare departments and services, we evaluated both THR and TKR to ascertain whether the surgical procedure undertaken had any influence on the eventual recommendation score (assuming that the patient experience of healthcare delivery at our unit was otherwise identical), not to investigate differences between the procedures.

During the study period, 6912 patients underwent hip or knee replacement. Questionnaires were completed by 6186 (89.5%) patients (3265 THR and 2921 TKR). All data were included in the analysis. We also recorded the patient’s age, gender and presence of co-morbidities and post-operative length of stay. The patient’s clinical outcome and satisfaction were assessed using patient-reported questionnaires.

The Oxford Hip or Knee Score^14,15and Medical Outcomes Study Short Form 12 (SF-12) health assessment¹⁶were completed by all patients pre-operatively and then by postal follow-up at six and 12 months post-operatively.

After 12 months, patients were asked to rate their overall satisfaction with their operated hip or knee on a four-point scale (very satisfied, satisfied, unsure or dissatisfied). They were also asked to rate specific facets of their surgical outcome with the following questions, answered on a six-point scale (excellently, very well, well, fairly, poorly, don’t know):

1) How well did the surgery relieve the pain in your affected joint? 2) How well did the surgery increase your ability to perform regular activities? 3) How well did the surgery allow you to perform heavy work or sport activities? 4) How well did the surgery meet your expectations?

Patients indicated their satisfaction with the care they received at the hospital with the question: 5) Rate your overall hospital experience using the response scale; excellent; very good; good; fair; poor or unknown. These responses were scored from 1 (excellently) to 5 (poorly), excluding the ‘don’t knows’.

We also asked: Would you recommend this operation to someone else? with the possible responses of: definitely yes; possibly yes; probably not; certainly not; or not sure. The patient response to this question was coded to the Net Promoter Score using previously described methodology for the validated five-point response version of the Net Promoter Score;¹⁰ definitely yes (1), probably yes (2), unsure (3), probably not (4), definitely not (5). A score of 1 is classified as a promoter, 2 as passive and 3–5 as a detractor.¹⁷

Results

The overall Net Promoter Score for joint replacement was 60, individual scores for THR and TKR were 71 and 49 respectively (Table I) indicating that different procedures generate different subsequent recommendation responses. A significantly greater proportion of THR patients were classified as promoters (compared with TKR patients), and significantly more TKR patients classified as detractors compared with THR patients (Table II, p < 0.001). Patients classified as ‘promoter’ were more likely to be male (χ² = 21.389, p < 0.001). The detractors reported an extra co-morbidity compared with the promoters (p < 0.001). The length of post-operative stay was statistically greater in the detractor group, though this reflected the large data volume, as the mean number of days was the same between promoters and detractors (Table II).

Pre-operatively scores were similar between the groups, though the Oxford Scores and SF-12 MCS were significantly lower in the net detractor group. The Oxford Scores differed by a single point (less than the minimal clinically important difference of the measure)¹⁸ and the MCS scores were within a standard deviation of the population normative score,¹⁶ suggesting that there was little practical significance in these differences (Fig. 1). Post-operative patient outcome scores for all measures were significantly different between Net Promoter Score classifications. Post-hoc analysis showed that there were significant differences (p < 0.001) between promoter/passive, promoter/detractor and detractor/passive with the detractors reporting worse outcomes (p < 0.001).

Figs. 1a - 1c

Histograms showing a) the Oxford Knee Score by Net Promoter Score, b) the SF-12 Physical Component Score, and c) the SF-12 Mental Component Score by Net Promoter Score.

Overall satisfaction varied significantly between Net Promoter Scores. Those who were classified as promoters reported a median satisfaction score of 1 (very satisfied), those classified as passive reported a median score of 2 (satisfied) and those who were classified as detractors reported a median score of 3 (uncertain). These differences were significant (Kruskall–Wallis, p < 0.001) (Table 2).

A strong correlation was seen between overall satisfaction and whether the patient would recommend the operation to another (rho = 0.637). This results in a correlation of determination (R²) of 0.406 between the two measures, suggesting that overall satisfaction response explains around 40% of the variation in Net Promoter Score and vice versa.

Variables that were shown to be significantly different between Net Promoter Scores in the univariate analysis were entered into stepwise logistic regression model using either promoter or detractor as the dependent variable (n = 4803). A model containing four variables was found to predict Net Promoter Score of detractor or promoter; whether the hip or knee joint was replaced, the satisfaction with pain relief, meeting of expectations, and the hospital experience (Table III). The model correctly predicted 95% of Net Promoter Score responses (98% for promoters and 72% for detractors). The Hosmer–Lemeshow goodness-of-fit test showed that there was no statistically significant difference between the observed probabilities and those predicted by the model (χ2 = 3.591, df = 7, p = 0.825). The model could therefore be considered as good fit. Patients who had undergone knee surgery were over twice as likely to be classified as detractors compared with hip surgery. A change of one point on the satisfaction scales relating to pain relief, expectations and hospital experience (where 1 = very satisfied and 4 = dissatisfied) meant that a patient was over twice as likely to be a detractor.

We specifically assessed the volume of missing data in the satisfaction questionnaire response to determine if the ‘recommend to a friend’ question was poorly received by patients. In our post-operative survey, the ‘would you recommend this operation to someone else?’ question demonstrated a completion rate of 96.5%, and was the most consistently completed question in our satisfaction survey (Table IV).

Discussion

This study assesses the Net Promoter Score/friends and family test score in the context of orthopaedic services, focusing on joint replacement. Assessing patient outcome survey responses over a six-year period, we found high levels of patient satisfaction with total joint replacement and show that most patients who have had either a hip or knee replacement would be happy to recommend the procedure to someone else.

Perhaps surprisingly, the Net Promoter Scores achieved by joint replacement are comparable to the most popular and successful commercial organisations.¹⁹ Table I highlights the scores achieved by selected companies on the Satmetrix European benchmark performance index (2012). The scores presented are for the best-performing companies in their particular category (mobile phones, televisions, internet providers, insurance companies and utility providers). While this concept of comparing services is commonplace in business, it is new to the orthopaedic literature.

We report different Net Promoter Scores for hip and knee replacement, which suggests that different scores are likely to be identified for other, differing, procedures (such as back or hand surgery). This indicates that case-mix must be taken into account when comparing scores between hospitals and across various departments. This mirrors differences between different types of business and highlights why businesses are primarily concerned with competitor comparison within their particular sector.

The new ‘friends and family test’ is described as a measure of patient satisfaction. This is perhaps not completely accurate. We found a strong correlation between our patients response to the ‘overall satisfaction’ question and the ‘recommend to another’ question, however, only 40% of the variation in one response can be explained by the other; suggesting that although these concepts are clearly related, they are not the same. To try to better understand the Net Promoter Score/friends and family response, we carried out multivariate regression modeling using all available covariate data (age, gender, Oxford Score, etc.) in an attempt to predict better the promoters and detractors. Whether patients would recommend the procedure to another was determined by four factors, namely, meeting pre-operative expectations of surgery, achieving pain relief following surgery, the hospital experience and whether the hip or the knee joint was replaced. These factors drove a model that was able to explain 95% of the variation in the recommendation response.

The factors previously reported to influence clinical outcomes including patient age, gender, co-morbidities, length of post-operative stay and PROM scores, did not help explain the ‘recommendation’ response. This highlights that an overarching ‘recommendation’ metric as a single and generic concept is only in part related to clinical effectiveness or to specific aspects of surgery; the patients’ experience of the delivery of healthcare is just as pertinent to this outcome measure response. We have previously found similar predictor variables when modelling the factors that influenced overall patient satisfaction.²⁰ This suggests a similarity between satisfaction and recommendation responses though the predictive association we report between these variables here suggests that promotion of joint replacement is indeed a separate metric. It may be that psychological and emotional factors are the key difference between these two responses, where patients may be broadly content and satisfied with what has happened to them, they would not necessarily suggest another person be put through the same procedure. The recommendation question may then be a more discerning metric in assessing the quality of the delivery of care than questions relating to personal satisfaction.

Pre-operative physical scores were the same in both promoter and detractor groups, though the detractors demonstrated statistically lower mental health scores prior to surgery. The actual MCS scores reported were within a standard deviation of the population normative mental health score¹⁶ and so the clinical relevance of this finding is difficult to interpret. Lower mental health scores have consistently been linked to worse outcomes,^21,22 however, a definitive association is lacking. Broadly, those who report worse mental health pre-operatively are less satisfied post-operatively, yet this did not influence the predictive modelling of the ‘recommendation to another’ response.

PROM scores are useful tools for the assessment of clinical outcome, but they mainly depend on pain relief,^23,24 and do not capture whether or not the patient is satisfied. This analysis highlights that while pain relief is very relevant to whether the patient would promote joint replacement, it is not the only factor. Baker et al²⁵ suggest that failure to meet optimistic expectations is associated with dissatisfaction following joint replacement; our results support this assertion. We also found the hospital experience of healthcare delivery to be equally relevant to our predictive modelling, with similar odds ratios achieved for meeting expectations, clinical outcome (pain relief) and hospital experience. As all operations were carried out in the same hospital environment it is likely that the difference in promoter response between THR and TKR reflects the acknowledged differences in outcome between hip and knee replacement²and merely highlights that any ‘net promoter’ style metric should take into account what operation is being performed.

Strengths of this study include a large patient cohort from a single NHS orthopaedic centre with many surgeons treating a large regional population. We have used validated, reliable instruments for assessing change in health status and outcome of joint replacement, and our data has been collected prospectively with high rates of follow-up. Though probably reflective of other interventions, the wider generalisability of these findings to emergency settings and the breadth of medical services provided by the NHS is unknown. It seems likely that this model will be valid for most interventions, however, the relative importance of individual factors may vary with clinical outcomes and expectation from different treatments. Broadly, patients who report higher clinical outcome scores also report better levels of satisfaction,^2,20,24 and are more likely to recommend joint replacement as a treatment. Studies in general medicine have also found conflicting associations between the patient’s experience of intervention and the technical quality of the care delivered as measured by other means.^26,27

A recent study suggested that while a single question can be asked, patients found the recommendation concept difficult, which could manifest in limited response rates to surveys which assess this. Our experience in the field of joint replacement is in stark contrast to this, where 96.5% of our patients responded to this question over a five-year period, making it the most consistently reported item in our satisfaction questionnaire. The context of how the question is asked may be important. We would recommend, as a minimum, a relevant clinical outcome score and some form of experience measure both to set the question and to help analyse the results.

Achieving pain relief, the meeting of pre-operative expectation and the hospital experience are the key domains that explain whether a patient would recommend joint replacement services: each domain contributes a similar amount to the overall response. The difference between THR and TKR suggests that no overarching score for a department should be used without case-mix adjustment. THR and TKR each give Net Promoter Scores which compare favourably with the services from the most successful commercial organisations.