Sensitivity and specificity of modified RUST score using clinical and radiographic findings as a gold standard

Introduction

The ability to accurately determine fracture healing and to predict union is central to patient care and to measure the success of various fracture interventions. Traditional clinical methods for evaluating fracture healing include the absence of pain with weightbearing or with palpation at the fracture site.¹ Radiological criteria of fracture healing include many factors ranging from cortical continuity, visibility of a fracture line, specific number of bridging cortices or simply surgeon impression.^1-3 Radiological scoring systems have also been described.^1-3 One such scoring system is the Radiological Union Scale for Tibia (RUST) fractures, which numerically evaluates and assesses progression to union after intramedullary (IM) nailing of tibia fractures.^2,4 Subsequently, a modified RUST (mRUST) score was described using four scores for cortices rather than three in attempts to better assess union and bridging callus in metadiaphyseal fracture area.³ The mRUST score is considered a reliable method for evaluation of metadiaphyseal fracture healing.³ To our knowledge, the accuracy, sensitivity, and specificity of the mRUSTscore in evaluating fracture union have not been determined. The goal of this study was to determine the time dependent sensitivity and specificity of the mRUSTscore in radiologically predicting fracture union of distal femur fractures, and subsequently to determine optimal cut-off values of mRUST at various time points (six weeks, 12 weeks, 24 weeks, and final follow-up).

Methods

After obtaining Insitutional Review Board approval (IRB ID #: 201612761), a cohort of 143 consecutive adult patients with 146 sets of radiographs of distal femur fractures, including periprosthetic, treated surgically from 2011 to 2016 at a single institution (Department of Orthopaedics and Rehabilitation, University of Iowa Hospitals and Clinics, USA) were identified. We excluded the radiographs with partial articular fractures (n = 14), referred or old nonunion cases (n = 5), incomplete follow-up less than 12 weeks (n = 17), follow-up only one time point (n = 2), unclear shadow of radiographs (n = 2), compression technique was used (n =1), or patients in which bone allograft was used (n = 1). All surgeons employed standard locked plating, angle blade, and nailing techniques. All periprosthetic fractures, type A fractures, and type C fractures with a simple articular line were treated with a small distal incision for plate/nail insertion, indirect reduction of the fracture and percutaneous insertion of proximal shaft screws. The remaining type C fractures were treated with a lateral arthrotomy, with our without lag screw (articular portions only), and percutaneous insertion of proximal shaft screws. The metaphyseal portion of the fracture was not directly reduced, and surgical fixation was bridging to provide relative stability. After applying our exclusion criteria, 104 sets of radiographs of distal femur fractures were included in the study. Of this cohort, 93 fractures (89.4%) were treated with locking plate constructs (51 periprosthetic fractures), one (0.96%) with angle blade plate, and ten (9.6%) with intramedullary nails. Operative fixation was performed by one of three fellowship trained orthopaedic traumatologists (JLM, MDK, MCW). Patients routinely returned for follow-up at two weeks, six weeks, 12 weeks, and 24 weeks, or until fracture union was determined by the surgeons. Clinical history and physical examination were performed at each follow-up visit and recorded in the medical record. Orthopaedic Trauma Association/AO Foundation (OTA/AO) fracture classifications were demonstrated in Table I.

Evaluation of radiological and clinical healing

AP and lateral radiographs of the knee and femur were independently reviewed by three investigators (YP, MCW, MDK). Callus formation of each cortex was evaluated and scored using the mRUST score.³ Assessments were performed on radiographs taken at six weeks, 12 weeks, 24 weeks, and final follow-up for each fracture, which was routinely followed up in the clinic. The mRUST score was assessed using both AP and lateral radiographs and all four cortices were scored as demonstrated in Figure 1 and Table II.

Table II.

Assessment tool for the modified Radiological Union Scale for Tibia fractures (mRUST) scores.

Score per cortex*	Radiological callus criteria
1	No callus
2	Callus present
3	Bridging callus
4	Remodelled, fracture not visible

1. *

   The individual cortical scores (anterior, posterior, medial, and lateral) are added to provide a mRUST. The mRUST score was totalled for each fracture to equal a minimum score of 4 or a maximum score of 16. Low scores indicate poor fracture healing and callus formation, while high scores correlate with fracture healing and remodelling.

Fig. 1

Radiographs illustrating a modified Radiological Union Scale for Tibia fractures (mRUST) score of 15 at final follow-up. Cortical score: lateral = 4, medial = 4, anterior = 3, and posterior = 4.

In order to decrease the effect of a potential learning curve in applying the mRUST score, prior to any radiological assessment, all three investigators reviewed and came to a consensus on how to apply the mRUST score to each cortex.^3,5-7 However, actual scoring of the 104 sets of radiographs was done by the investigators independently from each other. They were allowed access to radiographs of a single time point only, and were blinded to radiographs from the next time point. Once evaluation and scores for a specific time point were documented, the evaluators were then allowed to view radiographs for the next time point for subsequent scoring at six weeks, 12 weeks, 24 weeks, and final follow-up, respectively. In cases of obstructed visualization of the lateral cortex, our observers were instructed to use consolidation of the fracture line at the lateral cortex to best apply the mRUST score. Scores provided by each investigator were summed and divided to calculate the average mRUST score for each follow-up time point.

The ability of the averaged mRUST score to predict eventual fracture union was determined. The gold standard for union was determined by the operating surgeon (JLM, MDK, MCW) as documented in the medical record, and was based on clinical history and exam findings (patient-reported absence of pain, painless ambulation, absence of limp, no tenderness to palpation of fracture site) and evaluation of radiographs (cortical continuity, bridging cortices, visibility of fracture line) at the time of follow-up. Radiographs of all patients documented as fracture union, that had follow-up less than six months were independently reviewed by three authors (YP, BGW, JLM) for confirmation of union. Similarly, nonunion was determined by the operating surgeon as documented in the medical chart based on clinical and radiological findings. All nonunion cases were independently reviewed by three authors (YP, BGW, JLM) to confirm nonunion and determine construct condition, time of nonunion, time of revision, and revision type for nonunion. Mean length of follow-up for both union and nonunion cases was collected from the medical record. Mean follow-up for all patients (n = 104) was 12.5 months (3 to 55; median 9). Mean follow-up for patients with fracture union (n = 92) was 12.1 months (3 to 55; median 8.5 ). The mean time of clinical and radiological finding union from the surgeons from medical record was 11.5 weeks (4 to 28; median 12).

Results

Selection of appropriate cut point

The most appropriate sensitivity and specificity is summarized in Table V.

Table V.

Results of sensitivity, specificity of each cut point of mean modified Radiological Union Scale for Tibia fractures (mRUST) scores in each period time point.

Time point	Cut point	Sensitivity, %	Specificity, %	1-specificity, %
Six weeks	4	100.0	0.0	100.0
	5	92.60	21.3	78.7
	6	85.20	41.3	58.7
	7	51.90	58.7	41.3
	8	44.40	78.7	21.3
	9	22.20	92.0	8.0
	10	14.8	96.0	4.0
	11	7.4	100.0	0.0
12 weeks	5	100.0	0.0	100.0
	7	98.8	37.5	62.5
	8	89.4	62.5	37.5
	9	78.8	81.3	18.8
	10	58.8	93.8	6.3
	11	37.6	100.0	0.0
	12	20.0	100.0	0.0
	13	5.9	100.0	0.0
24 weeks	5	100.0	0.0	100.0
	6	98.6	0.0	100.0
	8	98.6	71.4	28.6
	10	91.9	100.0	0.0
	11	81.1	100.0	0.0
	12	62.2	100.0	0.0
	13	41.9	100.0	0.0
	14	20.3	100.0	0.0
	15	9.5	100.0	0.0
	16	4.1	100.0	0.0
Final	10	100.0	50.0	25.0
	11	97.3	50.0	25.0
	12	89.6	100.0	0.0
	13	74.7	100.0	0.0
	14	62.7	100.0	0.0
	15	41.3	100.0	0.0

1. Bold shows the most appropriate cut points, sensitivity, and specificity selected.

When the mRUST score was assessed as predictive of healing at the six-week and 12-week time points, the optimal cut points were six and nine, respectively. At these earlier time points, there was the sensitivity and specificity of 85.2% and 41.3%, and 78.8% and 81.3% at six weeks and 12 weeks, respectively. The accuracy of six weeks time point was 52.9% with positive predictive value (PPV) was 34.3% and negative predictive value (NPV) was 88.6%. However, the AUC was 0.6593 which is classified as “not good”, according to commonly used classification systems for AUC as a diagnostic test.¹⁰ The accuracy of the 12-week time point was 79.2% with positive predictive value (PPV), and was 97.5% and negative predictive value (NPV) was 41.9%. The AUC at the same cut point was 0.8864.

When the mRUST score was assessed as predictive of healing at the 24-week time point, the optimal cut points were ten. Compared to gold standard for union at this time point, the most appropriate sensitivity and specificity of the mRUST of ten was 91.9% and 100%, respectively. Using the same cut-off point, positive predictive value (PPV) was 100%, while negative predictive value (NPV) was 53.8% with the highest accuracy of 92.6% (summarized in Table VI).

Table VI.

Summary of sensitivity, specificity, accuracy, PPV, NPV, and AUC of each cut point at each time point.

Time point	Cut point	Sensitivity, %	Specificity, %	Accuracy, %	PPV, %	NPV, %	AUC
Six weeks	6	85.2	41.3	52.9	34.3	88.6	0.6593
12 weeks	9	78.8	81.3	79.2	95.7	41.9	0.8864
24 weeks	10	91.9	100.0	92.6	100.0	53.8	0.9826
Final	12	89.6	100.0	90.1	100.0	33.3	0.9805

1. AUC, area under the ROC curve; NPV, negative predictive value; PPV, positive predictive value.

Increasing the cut-off point to 13 at the 24-week time point yielded decreased sensitivity (41.9%), but specificity remained 100%. The AUC was 0.9826 which was classified as “excellent”, according to commonly used classification using AUC for a diagnostic test.⁸ (Table V).

The ROC curve for the mean mRUST score compared to the gold standard at each time point is demonstrated in Figure 2 and 5.

Fig. 2

Receiver operating characteristic (ROC) curve of modified Radiological Union Scale for Tibia fractures (mRUST) score at six weeks.

Fig. 3

Receiver operating characteristic (ROC) curve of modified Radiological Union Scale for Tibia fractures (mRUST) score at 12 weeks.

Fig. 4

Receiver operating characteristic (ROC) curve of modified Radiological Union Scale for Tibia fractures (mRUST) score at 24 weeks.

Fig. 5

Receiver operating characteristic (ROC) curve of modified Radiological Union Scale for Tibia fractures (mRUST) score at final follow-up.

Discussion

The ability to accurately determine fracture healing and to predict union is central to patient care, and to measure the success of various fracture interventions. Traditional clinical methods for evaluating fracture healing include the absence of pain with weightbearing or with palpation at the fracture site.¹ Radiological criteria of fracture healing includes many factors ranging from cortical continuity, visibility of a fracture line, specific number of bridging cortices, or simply surgeon impression.^1-3 Therefore, it is important to have a method to evaluate radiological healing precisely. However, radiological assessment of fracture healing remains difficult as no clear consensus has been reached on assessing or determining bony union. RUST and mRUST were developed in order to assign a numerical value to tibial shaft fracture and metadiaphyseal healing after operative fixation.^3,6

Kooistra et al⁵ showed the RUST score had better reliability when compared with a surgeon’s general impression or the number of cortices bridged by callus in the follow-up of tibial fractures. The mRUST score has been developed in order to evaluate cortical scoring systems in metadiaphyseal fractures.³ Litrenta et al³ showed substantial agreement and increased ICCs for the mRUST compared to the RUST score in the assessment of metadiaphyseal fracture healing in a series of radiographs of distal femur treated with either plate or retrograde nail. They demonstrated that scores of ten and 13 using the RUST and mRUST scores, respectively, resulted in over 90% of reviewers assigning union on the RUST and mRUST, respectively.³ Furthermore, they found that a standard RUST score of ten and a mRUST score of 14 provide an excellent definition of union based on surgeons’ opinion and biomechanical testing in a sheep osteotomy model.¹³ Cooke et al¹⁴ demonstrated that radiologically healed fractures had a mRUST ≥ 13 and a RUST ≥ 10 and had excellent relationship to structural and biomechanical metrics in an animal study. This is clinically relevant, as scores showed high correlation to physical properties of healing and generally distinguished healed versus non-healed fractures.¹⁴ However, there has been no investigation of sensitivity or specificity of the mRUST score in evaluating eventual fracture union. This study demonstrated the highest sensitivity of 91.9%, specificity of 100% and accuracy of 92.6% of mRUST score of ten at 24-week follow-up.

Cut-off points of the mRUST score in the previous studies did not rely on clinical healing as a gold standard. They solely implemented the agreement of reviewer to determine the score that define union radiologically. Some of those were also investigated from animal studies, which may have different results from clinical situation. Additionally, sensitivity and specificity have not been investigated in the literatures.

The current study supplements the literatures by being the first to assess and determine the sensitivity, specificity, and accuracy of the mRUST scoring system. Thus, the current study allows us to quantify how good and reliable mRUST is at predicting bony union. The ROC curve is a graphic representation of the relationship between both sensitivity and specificity, and it helps select the optimal model by determining the best threshold for the diagnostic test. Recognizing which cut-off points (i.e. mRUST scores) provide high sensitivity or specificity (or both) would prove extremely useful in predicting ultimate fracture union. Furthermore, identified cut-off points could further clinical communication between providers, as well as being used for continued research on fracture healing.

Interpretation of the ROC curve is similar to a single point in the ROC space: the closer the point on the ROC curve to the ideal coordinate, the more accurate the test is. The closer the points on the ROC curve are to the diagonal, the less accurate the test is. The faster the curve approaches the ideal point, the more useful the test results are. The AUC provides a way to measure the accuracy of a diagnostic test. The larger the area, the more accurate the diagnostic test is. There are several criteria for determining the most appropriate cut-off value in a diagnostic test with continuous results, based on ROC analysis.

According to the ROC curves in the current study, mRUST performs poorly and should not be used to predict eventual union when used within six weeks postoperatively (AUC = 0.6593). This can be attributed to the timing of callus formation, and mRUST performs poorly in the early postoperative period as callus formation is likely not robust enough to be radiologically scored.

Conversely, the cut-off point of nine is appropriate for 12 weeks follow-up, and demonstrates a mRUST sensitivity and specificity of 78.8% and 81.3%, respectively. If we used the cut-off point at 13, as done in previous studies, the sensitivity significantly decreased to 5.9% and the specificity increased up to 100%. Both examples showing drastic changes in sensitivity and specificity with different mRUST scores, and highlights the need for validated cut-off points showing how minimal differences in scoring can produce drastic changes in sensitivity and specificity. Nonetheless, results show a cut-off point > ten yields 100% specificity and can be reliably used to make union likely possibility.

The cut-off point of ten is appropriate for 24 weeks follow-up yielding a mRUST sensitivity and specificity of 91.9% and 100%, respectively with the highest sensitivity plus specificity. The AUC of 0.9826 indicates this is an excellent diagnostic test at this cut-off point.¹⁰ The accuracy at this cut off point was 92.6%, which was the highest value. Once again increasing to a cut-off point of 13, as used in previous studies, the sensitivity remarkably decreased to 41.9%. Similarly, the specificity increased to 100%. However, these findings are helpful, as a mRUST score of 13, with its corresponding 100% specificity at 24 weeks follow-up, will definitively confirm fracture union.

Finally, when we used a cut-off point of 12, the sensitivity and specificity of the mRUST score was 89.6% and 100%, respectively, at final follow-up, which was an appropriate cut-off point as the AUC was 0.9805, classified as “excellent” according to commonly used classification using AUC for a diagnostic test.¹⁰ The accuracy at this cut off point was 90.1%.

There are several limitations to the current study. The accurate calculation of the mRUST score requires evaluation of all four bony cortices. Evaluation of the lateral cortex in fractures treated with lateral plate constructs can be very difficult as the cortex can be obscured by the implant. This difficulty was noted by each investigator. Knowing this, it is not surprising that ICC for plate constructs showed only moderate agreement. This limitation is clearly emphasized by the fact that mRUST scores for intramedullary nail constructs consistently demonstrated higher agreement than plate constructs. This limitation is not unique to our study as previous authors evaluating nails and plates separately showed ICC scores of 0.74 and 0.59, respectively.³ The authors also showed the lowest agreement dealt with the lateral cortex as full visualization is often difficult.³ In addition, there were noticeable patients losing to follow-up over time, leaving smaller groups of patient at subsequent follow-up. Furthermore, this is a retrospective chart review and subject to imperfections in the medical record. Most notable is the potential variation in subjective evaluation of healing between physicians at follow-up. This, however, in our opinion reflects accurately the practice occurring in routine clinic visits and therefore stands as a satisfactory bank of information. Lastly, with a low prevalence of nonunion (11.5%), the diagnosis of nonunion in the population may result in low accuracy, as accuracy is influence by the prevalence of the disease in a selected population.

To our knowledge, this is the first study to evaluate the sensitivity, specificity, accuracy, and cut-off points of the mRUST score. Our results show good to excellent accuracy ranges in assessing metadiaphyseal femur fracture healing. Most notably, a mRUST score of ten at 24 weeks follow-up yields the highest sensitivity and specificity of 91.9% and 100%, respectively, with the highest accuracy of the mRUST score was 92.6%. For 100% specificity at 24-week follow-up, a mRUST score of at least ten must be used to assure union; however, this significantly reduces sensitivity.

In conclusion, the AUCs ranged from good to excellent according to commonly used classifications using AUC as a diagnostic test at 12 weeks, 24 weeks, and final follow-up. Importantly, the mRUST performs poorly in the early postoperative period (≤ six weeks) and should not be used to reliably assess or predict ultimate bony union in this time period.