Abstract
Aims
Patients undergoing limb reconstruction surgery often face a challenging and lengthy process to complete their treatment journey. The majority of existing outcome measures do not adequately capture the patient-reported outcomes relevant to this patient group in a single measure. Following a previous systematic review, the Stanmore Limb Reconstruction Score (SLRS) was designed with the intent to address this need for an effective instrument to measure patient-reported outcomes in limb reconstruction patients. We aim to assess the face validity of this score in a pilot study.
Methods
The SLRS was designed following structured interviews with several groups including patients who have undergone limb reconstruction surgery, limb reconstruction surgeons, specialist nurses, and physiotherapists. This has subsequently undergone further adjustment for language and clarity. The score was then trialled on ten patients who had undergone limb reconstruction surgery, with subsequent structured questioning to understand the perceived suitability of the score.
Results
Ten patients completed the score and the subsequent structured interview. Considering the tool as a whole, 100% of respondents felt the score to be comprehensible, relevant, and comprehensive regarding the areas that were important to a patient undergoing limb reconstruction surgery. For individual questions, on a five-point Likert scale, importance/relevance was reported as a mean of 4.78 (4.3 to 5.0), with ability to understand rated as 4.92 (4.7 to 5.0) suggesting high levels of relevance and comprehension. Flesch-Kincaid reading grade level was calculated as 5.2 (10 to 11 years old).
Conclusion
The current SLRS has been shown to have acceptable scores from a patient sample regarding relevance, comprehensibility, and comprehensiveness. This suggests face validity, however further testing required and is ongoing in a larger cohort of patients to determine the reliability, responsiveness, precision, and criterion validity of the score in this patient group.
Cite this article: Bone Jt Open 2021;2(9):705–709.
Take home message
The Stanmore Limb Reconstruction Score has shown face validity in the pilot testing phase.
Further validation is ongoing in a larger patient cohort.
Introduction
Validated and effective outcome measures are an essential element of performing good quality research in orthopaedic surgery.1 Increasingly, such measures are also being used in developing registry data as well as potential use in commissioning.2 Patient-reported outcomes measures (PROMs) allow further understanding into the patient perspective of a disease and its treatment. Such information can guide effectiveness of treatment,3 and help to counsel patient expectations for surgery.4
There are a wide range of PROMs that have been designed. These can include scores based on broader health-related quality of life (e.g. 36-Item Short Form Health Survey questionnaire (SF-36), EuroQol five-dimension questionnaire (EQ-5D)),5,6 those specific to an anatomical area or joint (e.g. Oxford Hip Score, American Academy of Orthopedic Surgeons (AAOS) lower limb score),7,8 or scores which are targeted at a particular condition (e.g. Boston Carpal Tunnel Syndrome Questionnaire, Western Ontario Shoulder Instability Index).9,10 The patient undergoing limb reconstruction surgery has particular challenges, both due to the underlying condition (deformity, complex trauma, infection, nonunion) and the treatment process itself. As well as pain and functional limitations, psychological, cosmetic, and socioeconomic issues may affect the healthcare-related quality of life.11-16 As such, existing anatomy specific scores, which focus on pain or function alone, may not capture the improvements that correction of deformity may have on social impairment or patient perception of cosmesis.
A recent systematic review demonstrated that no single PROM adequately captures all domains of importance to this patient group.17 This results in existing research either using different collections of instruments to try to effectively understand patient-reported outcomes, or even neglecting patient-reported measures entirely to focus on the more easily attainable radiological outcomes and complication profile. Use of multiple measures not only limits comparisons with other published data, but also limits feasibility of data collection, due to “respondent fatigue” from both the patient and the collecting clinician.18
We have aimed to create and develop such an outcome measure in the hope that this could provide a tool for research and quality improvement by the limb reconstruction community. The first iteration of this has been previously published.17 We present the initial introduction of this measure, along with assessment of face validity in the form of patient comprehension, comprehensiveness, and relevance.
Methods
This project was assessed and approved as a service evaluation project by the local research and ethics panel (Reg No. SE20.46). The methodology of initial score design, current pilot study, and plans for future validation are based on the methods as described by Fizpatrick et al,19 and the more recent COnsensus based Standards for the selection of health status Measurement INstruments (COSMIN) methodology.20,21
A combination of patient and clinician involvement was used throughout the design of the score. Patients currently undergoing limb reconstruction surgery were recruited to take part in a structured interview following attendance at the outpatient clinic. Inclusion criteria at each stage was for patients aged > 18 years who are currently undergoing treatment with external fixator or lengthening nail. Exclusion criteria included limited understanding of English, the need for a translator, or any patients who were unwilling or unable to consent to take part.
Score design
Structured interviews were performed in 20 patients using described techniques to allow exploration of general and then subsequently specific areas relating to their underlying condition and limb reconstruction treatment.22 These interviews were transcribed and conventional content analysis used to identify issues and themes of relevance.23 This was used as a basis to determine the domains of relevance for the structure of the score. Subsequently, interviews with limb reconstruction surgeons (4), clinical nurse specialists (3), and therapists (2) were performed to further guide design of the score and ensure that the content was also thought to be comprehensive and relevant from the clinician perspective. The pain domain was based on a modification of the Brief Pain Inventory.24 The remaining domains, as determined by the categories and themes identified from the content analysis, were assessed using a five-point Likert scale. The preliminary score has been published previously as an appendix to a systematic review of PROMs in limb reconstruction;17 this paper shall focus on the pilot assessment of face validity.
Pilot and refinement
The score was trialled in 11 patients recruited in clinic with the same criteria as the design interviews, but these were new patients not previously involved in the design. They completed a further written questionnaire and short structured interview to assess the comprehensibility and relevance of each individual question, along with comprehensiveness of the score as a whole.
These comments were then used in two iterative group discussions involving limb reconstruction surgeons, clinical nurse specialists, and physiotherapists to further refine the language, appearance, and question structure of the score. The modified score (see Supplementary Material) was then trialled again in ten patients, once again using short structured interviews and a questionnaire to assess the comprehensibility and relevance of each individual question, along with comprehensiveness of the score as a whole. The patients were asked about the acceptability of the test and whether they would be willing to complete this while waiting in clinic. The score was tested with the use of Flesch-Kincaid readability tests (mathematical assessment of structure, sentence length, and word length in a document)25 for objective assessment of comprehension level.
Statistical analysis was performed using SPSS 27 (IBM, USA) for Mac. Unless otherwise stated, categorical variables are expressed as frequency (percentage) and continuous variables are expressed as mean (range) with p < 0.05 considered as statistically significant. Non-parametric group comparisons of data were made with the Mann-Whitney U test.
Demographic details
The demographic details of the initial pilot group, and the second pilot after refinement of the score, are shown in Table I.
Table I.
Demographic details of patients undergoing pilot testing of the score.
| Variable | First pilot group | Second pilot group |
|---|---|---|
| Sex, n | ||
| Male | 5 | 2 |
| Female | 6 | 8 |
| Mean age, yrs (range) | 33 (17 to 68) | 42 (18 to 78) |
| Indication for treatment, n | ||
| Deformity correction/lengthening | 6 | 7 |
| Nonunion/infection | 4 | 3 |
| Trauma | 1 | 0 |
Results
Comprehensibility
The first test group showed that 3/11 patients found at least one question difficult to understand. Clarifications were made to the questions on use of public transport and employment, along with general adjustment for readability. The subsequent test group had no patients (0/10) who found any of the questions difficult to understand. The mean understandability of individual questions was rated as 4.92/5 (4.8 to 5).
The Flesch-Kincaid readability tests demonstrated a Flesch reading ease index of 58.6 for the initial version and 78.2 for the modified version (this index is out of 100, with a higher value representing an easier to read text). The Flesch-Kincaid reading grade level improved from 15.6 initially (equivalent to 18 years old) to 5.2 on the modified version (equivalent to a reading age of 10 to 11 years old).
Relevance
For the initial group, the mean score for relevance of the questions was 4.34/5 (3.8 to 5). The second group showed reported a mean relevance of 4.78 (4.3 to 5) for the modified score, which is a significant improvement (p < 0.001). When taking the score as a whole, 10/10 patients in the second group felt the score to be relevant to their needs. Similarly, 10/10 patients felt that the score was comprehensive of their situation as a limb reconstruction patient.
Feasibility
All patients using the modified score (10/10) felt the score was acceptable to complete while waiting in clinic.
Discussion
This pilot study has demonstrated that the updated version of the SLRS shows good levels of comprehensibility, comprehensiveness, relevance, and acceptability to a patient cohort undergoing limb reconstruction surgery. This is an indication of face validity from the patient perspective, which can be taken as an initial stage of the validation process.
By the nature of a pilot study, the numbers are small, however we feel the results are adequate to consider continuing with further validation. This has included patients with a variety of indications but limited to a single centre. The score has been designed using only adult patients and further work would be required if a paediatric limb reconstruction PROM was to be considered. While a combination of trauma patients and elective deformity/nonunion patients were included in the design of the score, this pilot did not include trauma patients, which is a limitation. For the ongoing further validation of this score, both trauma and elective patients will be included.
The previous systematic review from this unit has suggested that there is at present no adequate score which covers all the needs of the limb reconstruction patient.17 This score has been created with the limb reconstruction patient at its centre from the beginning, to address their specific requirements. These methods have allowed us to endeavour to ensure that all domains of importance to the patient have been captured. Beyond the assessment of pain and physical function, the score includes evaluation of social function, cosmesis, and emotional state, to allow for a more nuanced assessment of the experience of the limb reconstruction patient within a single score. We believe this allows for better understanding of the outcomes of this patient group, both during and after treatment, which may help improve their holistic surgical care. Multiple existing scores using joint-specific and generic quality of life outcome measures could be used to capture elements of these data over the range of domains required. However, this would potentially risk both respondent fatigue18 and, if too time-consuming, could also decrease feasibility of use of the score in a clinical setting.26 A single specific outcome score can avoid these limitations.
At the time of writing, there is a UK group investigating the current state of PROMs in limb reconstruction. They have published their protocol for the theory of design of a new score,27 although they are yet to report on the outcomes of a design process. There is also a proposed paediatric limb reconstruction score from Canada,28 which is currently undergoing validation, although not yet available for general use. As our score has been designed for the adult patient group, this paediatric score may prove complementary for outcomes data collection in limb reconstruction and we look forward to further results as their score is developed.
Further work is underway that will be required to fully validate the SLRS. The requirements for developing an effective and valid patient-reported measure were initially described by Fitzpatrick et al.19 The COSMIN methodology, however, has built further on this and gives a clear and reproducible framework with which to consider the design and validation of such a score. Criterion validity can be tested considering both convergent and divergent validity in comparison to existing (albeit not comprehensive) scores. Reliability to repeated testing, sensitivity to change, internal consistency, and floor and ceiling effects will all need to be examined.
These methods of validation will require a larger number of patients to use the score and take part in the validation process. Due to the complex and subspecialized nature of limb reconstruction, the number of patients in any single unit are small, so ideally this testing process will be through a multicentre collaboration. At the time of writing we have a number of limb reconstruction units working together on the validation project, with an aim to further expand this. Once more centres are using a similar standardized score, more effective comparison will be allowed between published results for techniques, leading to easier collaboration for research, and this may pave the way for further registry data collection of outcomes to everyone’s benefit.
In conclusion, this pilot study has demonstrated the SLRS has acceptable relevance, comprehension and comprehensiveness for the needs of a cohort of limb reconstruction patients. Further work is now ongoing in order to ensure the score is fully validate.
References
1. Swiontkowski MF , Buckwalter JA , Keller RB , Haralson R . The outcomes movement in orthopaedic surgery: where we are and where we should go . J Bone Joint Surg Am . 1999 ; 81-A ( 5 ): 732 – 740 . Crossref PubMed Google Scholar
2. Black N . Patient reported outcome measures could help transform healthcare . BMJ . 2013 ; 346 : f167 . Crossref PubMed Google Scholar
3. Swiontkowski MF . Outcomes measurement in orthopaedic trauma surgery . Injury . 1995 ; 26 ( 10 ): 653 – 657 . Crossref PubMed Google Scholar
4. Stiggelbout AM , Van der Weijden T , De Wit MP , Frosch D , Legare F , Montori VM . Shared decision making: really putting patients at the centre of healthcare . BMJ . 2012 ; 344 : e256 . Crossref PubMed Google Scholar
5. Herdman M , Gudex C , Lloyd A , Janssen M , Kind P , Parkin D . Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L . Qual Life Res . 2011 ; 20 ( 10 ): 1727 – 1736 . Crossref PubMed Google Scholar
6. Ware JE , Sherbourne CD . The MOS 36-ltem Short-Form Health Survey (SF-36 ). Med Care . 1992 ; 30 ( 6 ): 473 – 483 . Google Scholar
7. Dawson J , Fitzpatrick R , Carr A , Murray D . Questionnaire on the perceptions of patients about total hip replacement . J Bone Joint Surg Br . 1996 ; 78-B ( 2 ): 185 – 190 . PubMed Google Scholar
8. Johanson NA , Liang MH , Daltroy L , Rudicel S , Richmond J . American Academy of Orthopaedic Surgeons lower limb outcomes assessment instruments. Reliability, validity, and sensitivity to change . J Bone Joint Surg Am . 2004 ; 86-A ( 5 ): 902 – 909 . Crossref PubMed Google Scholar
9. Levine DW , Simmons BP , Koris MJ , Daltroy LH , Hohl GG , Fossel AH . A self-administered questionnaire for the assessment of severity of symptoms and functional status in carpal tunnel syndrome . J Bone Joint Surg Am . 1993 ; 75-A ( 11 ): 1585 – 1592 . Crossref PubMed Google Scholar
10. Kirkley A , Griffin S , McLintock H , Ng L . The development and evaluation of a disease-specific quality of life measurement tool for shoulder instability. The Western Ontario Shoulder Instability Index (WOSI ). Am J Sports Med . 1998 ; 26 ( 6 ): 764 – 772 . Crossref PubMed Google Scholar
11. McKee MD , Yoo D , Schemitsch EH . Health status after Ilizarov reconstruction of post-traumatic lower-limb deformity . J Bone Joint Surg Br . 1998 ; 80-B ( 2 ): 360 – 364 . Crossref PubMed Google Scholar
12. Giannoudis PV , Harwood PJ , Kontakis G , Allami M , Macdonald D , Kay SP . Long-term quality of life in trauma patients following the full spectrum of tibial injury (fasciotomy, closed fracture, grade IIIB/IIIC open fracture and amputation . Injury . 2009 ; 40 ( 2 ): 213 – 219 . Crossref PubMed Google Scholar
13. Modin M , Ramos T , Stomberg MW . Postoperative impact of daily life after primary treatment of proximal/distal tibiafracture with Ilizarov external fixation . J Clin Nurs . 2009 ; 18 ( 24 ): 3498 – 3506 . Crossref PubMed Google Scholar
14. Montpetit K , Hamdy RC , Dahan-Oliel N , Zhang X , Narayanan UG . Measurement of health-related quality of life in children undergoing external fixator treatment for lower limb deformities . J Pediatr Orthop . 2009 ; 29 ( 8 ): 920 – 926 . Crossref PubMed Google Scholar
15. Burton M , Walters SJ , Saleh M , Brazier JE . An evaluation of patient-reported outcome measures in lower limb reconstruction surgery . Qual Life Res . 2012 ; 21 ( 10 ): 1731 – 1743 . Crossref PubMed Google Scholar
16. Kim SJ , Balce GC , Agashe MV , Song SH , Song HR . Is bilateral lower limb lengthening appropriate for achondroplasia?: midterm analysis of the complications and quality of life . Clin Orthop Relat Res . 2012 ; 470 ( 2 ): 616 – 621 . Crossref PubMed Google Scholar
17. Antonios T , Barker A , Ibrahim I , et al. A systematic review of patient-reported outcome measures used in circular frame fixation . Strategies Trauma Limb Reconstr . 2019 ; 14 ( 1 ): 34 – 44 . Crossref PubMed Google Scholar
18. Rolstad S , Adler J , Rydén A . Response Burden and Questionnaire Length: Is Shorter Better? A Review and Meta-analysis . Value Health . 2011 ; 14 ( 8 ): 1101 – 1108 . Crossref PubMed Google Scholar
19. Fitzpatrick R , Davey C , Buxton MJ , Jones DR . Evaluating patient-based outcome measures for use in clinical trials . Health Technol Assess . 1998 ; 2 ( 14 ): i – 0 . PubMed Google Scholar
20. Mokkink LB , Terwee CB , Knol DL , Stratford PW , Alonso J , Patrick DL . The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: a clarification of its content . BMC Med Res Methodol . 2010 ; 10 : 22 . Crossref PubMed Google Scholar
21. Mokkink LB , Terwee CB , Patrick DL , Alonso J , Stratford PW , Knol DL . The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study . Qual Life Res . 2010 ; 19 ( 4 ): 539 – 549 . Crossref PubMed Google Scholar
22. Britten N . Qualitative interviews in medical research . BMJ . 1995 ; 311 ( 6999 ): 251 – 253 . Crossref PubMed Google Scholar
23. Hsieh HF , Shannon SE . Three approaches to qualitative content analysis . Qual Health Res . 2005 ; 15 ( 9 ): 1277 – 1288 . Crossref PubMed Google Scholar
24. Mendoza T , Mayne T , Rublee D , Cleeland C . Reliability and validity of a modified Brief Pain Inventory short form in patients with osteoarthritis . Eur J Pain . 2006 ; 10 ( 4 ): 353 – 361 . Crossref PubMed Google Scholar
25. Kincaid JP , Braby R , Mears JE . Electronic authoring and delivery of technical information . Journal of Instructional Development . 1988 ; 11 ( 2 ): 8 – 13 . Google Scholar
26. Makhni EC . Meaningful Clinical Applications of Patient-Reported Outcome Measures in Orthopaedics . J Bone Joint Surg Am . 2021 ; 103-A ( 1 ): 84 – 91 . Crossref PubMed Google Scholar
27. Leggett H , Scantlebury A , Sharma H , Hewitt C , Harden M , McDaid C . Quality of life following a lower limb reconstructive procedure: a protocol for the development of a conceptual framework . BMJ Open . 2020 ; 10 ( 12 ): e040378 : 12 . Crossref PubMed Google Scholar
28. Chhina H , Klassen A , Kopec JA , Oliffe J , Cooper A . International multiphase mixed methods study protocol to develop a patient-reported outcome instrument for children and adolescents with lower limb deformities . BMJ Open . 2019 ; 9 ( 5 ): e027079 . Crossref PubMed Google Scholar
Author contributions
J. Wright: Conceptualization, Investigation, Writing – original draft.
A. Timms: Investigation, Writing – review & editing.
S. Fugazzotto: Investigation, Writing – review & editing.
D. Goodier: Conceptualization, Writing – review & editing.
P. Calder: Conceptualization, Writing – review & editing.
Funding statement
No benefits in any form have been received or will be received from a commercial party related directly or indirectly to the subject of this article.
ICMJE COI statement
P. Calder reports travel and accommodation expenses from Smith & Nephew and Nuvasive, unrelated to the study.
Open access funding
Open access funding was supplied by the departmental education and research fund.
Acknowledgements
We would like to acknowledge the input of Amy Barker and thank her for her contribution to creation of the original score as part of her BSc project in Trauma & Orthopaedics.
Ethical review statement
This project was assessed and approved as a service evaluation project by the local research & ethics panel (Reg No: SE20.46)
Follow J. Wright @jwrightortho
Follow the authors @RNOHnhs
Supplementary material
Patient questionnaire.
© 2021 Author(s) et al. This is an open-access article distributed under the terms of the Creative Commons Attribution Non-Commercial No Derivatives (CC BY-NC-ND 4.0) licence, which permits the copying and redistribution of the work only, and provided the original author and source are credited. See https://creativecommons.org/licenses/by-nc-nd/4.0/
