Ankle fracture internal fixation performed by cadaveric simulation-trained versus standard-trained orthopaedic trainees: a preliminary, multicentre randomized controlled trial

Study design and participants

This study is a pragmatic, multicentre, assessor-blind educational randomized controlled trial (RCT). It was conducted at nine secondary and tertiary NHS hospitals in England, in accordance with the published trial protocol.¹³ Participants were recruited from three orthopaedic training programmes in the West Midlands, UK. Eligible trainees were in postgraduate years three to five at recruitment (i.e. core trainees year one or two (CT1 to 2), speciality trainees year one or two (ST1 or 2), and speciality trainees year three (ST3)). Exclusion criteria were consent refusal, or unavailability on the course dates. Support for the study was agreed prospectively with training programme directors. Eligible trainees were identified by liaison with programme administrators and invited to participate by email. All participants gave written informed consent.

The study was approved by NHS Research Ethics (15/WM/0464) and Confidentiality Advisory Group (16/CAG/0125). The study was registered prospectively (ISRCTN20431944).

Randomization

Randomization lists were computer generated using a simple blocking scheme (of size 4) prepared by a statistician with no further involvement in the trial. Researchers who collected data and undertook the analysis were blinded to training allocation. It was not feasible to mask participants to their allocation.

Training interventions

The cadaveric-trained group received an intensive two-day cadaveric simulation training course at the start of the surgical training year, where they were taught to perform ankle fracture, hip fracture fixation, hip hemiarthroplasty, and lower limb fasciotomy. These procedures were selected as they are mapped to the UK surgical training curriculum for progression to postgraduate year six.¹⁴

Fresh frozen cadavers were purchased under license from a specialist supplier.¹⁵ A Weber B ankle fracture was simulated by a near complete sawcut and the completion of the fracture with the twist of an osteotome. This produced a simulated fracture surface that could be reduced by keying in and held with a reduction clamp. The participant:faculty and participant:cadaver ratios were 2:1. Effort was made to maximize the fidelity of the simulation by using full surgical dress for participants, surgical drapes for the cadaveric ‘patients’, comprehensive surgical instrument trays and implants. C-arm image intensification and radiographers were available, and scrub nurses were assigned to each station. The surgical training centre was set up as eight simulated operating theatres running in two parallel circuits of four. All participants performed procedures once as first surgeon and once as assistant during the course. Left-sided procedures were performed on day one and right sided procedures on day two. To maximize learning, immediate structured feedback was given after each procedure through the completion of procedure based assessment (PBAs), which is the current gold standard, technical-skill assessment tool in UK surgical training.¹⁶ Maximum use was made of the cadaveric surgical environment, with no pressure of time and no patient safety requirement to ensure that novice surgeons did not operate beyond their current level of competence. After the course, the cadaveric-trained group returned to their respective hospitals and continued to receive standard training.

Standard training was delivered in the working environment with trainees receiving training in the management of these conditions from their educational supervisors, when suitable patients with these conditions presented to the training hospitals.

Trainees allocated to the “standard training” group received the above standard training from the start of the training year. This group then received the cadaveric course towards the end of the training year, a condition mandated in the ethical approval granted for the study. It was considered unfair to exclude half of the trainees from the educational experience of a cadaveric course (although the effect of the training course was not, at the time of approval, known). The effect of the catch-up course on these trainees was not studied.

All procedures performed by residents during the trial follow-up period were supervised by surgeons either directly ‘scrubbed’ or indirectly ‘unscrubbed’, as is usual practice in the UK.

Outcomes

The primary outcome was accuracy of the reduction on the first postoperative radiograph. The parameters for acceptability were, as per the supporting literature; medial clear space ≤ 4 mm, lateral malleolar displacement ≤ 2 mm, medial malleolar displacement ≤ 2 mm, tibiofibular clear space < 5 mm, and talocrural angle < 5° from normal (normal = 80°).¹⁰

These radiological measurements have been demonstrated to have face- and construct validity, and reliability for measuring technical skill in a relevant population for this trial.¹⁷ Radiological measurements for all primary outcomes were obtained by one researcher (HKJ).

Secondary outcome measures were procedure time (knife-to-skin to wound closure, in minutes) and intraoperative radiation dose administered to the patient (in Gym²). Procedure times were obtained from theatre performance data and radiation dose was obtained from the report generated by the radiographer at the end of each case and saved to the hospital PACS (picture archiving and communication system).

Ankle fracture fixations performed by trainees during the ten-month study follow-up were studied. Assisting-only cases were not included. We did not do subgroup analyses according to fracture type or implant choice due to small numbers. As we evaluated radiographs within the setting of an educational trial, we were unable to specify a priori the type of fractures to be included in the study, and ankle fracture is a complex and highly heterogenous injury. We only included medial and lateral malleoli fixations in the analysis.

Statistical analysis

It was not known a priori what the minimal clinically important difference (MCID) was for the primary outcome measures as this is the first trial attempting to measure the impact of a training intervention on the technical results of open ankle surgery performed on real patients.

Given that the surgical training centre could safely accommodate 16 delegates, a maximum sample size was set at 16 in each arm of the trial. No formal power calculations were undertaken. The plan was approved by the acting data monitoring committee. No interim analyses were undertaken.

Our analysis investigated differences in the primary outcome measure i.e. fracture reduction accuracy from the first postoperative radiograph, between the two training groups on an intention-to-treat (ITT; i.e. train) basis.

Multivariate, multilevel mixed effects models were used to assess the effects of the training intervention on reduction accuracy, allowing for potential within-surgeon correlation between repeated observations, treating surgeon clustering as a random effect. The regression models adjusted for important fixed effects including patient condition (ASA grade), patient age (years), and surgeon experience at baseline (number of prior cases).

Our primary inferences were drawn from the ITT (train) analysis, without imputation for missing data. Secondary analysis of the primary outcomes under a ‘per-protocol’ as-trained approach was undertaken, to set in context the results of the ITT analysis and to aid understanding of the impact of receiving training on the primary outcome measures.

All analyses were undertaken using statistical software (Stata version 16; StataCorp, USA). We have presented training effect estimates from all models (our primary analysis) with standard 95% confidence intervals. All hypothesis testing was at the 5% level with no adjustments for multiple testing.

Primary outcomes

Overall, 139 internal fixation operations for ankle fracture were performed by participants during study follow-up, 91 by the intervention group and 48 by the control group. Adequate anteroposterior postoperative radiographs were available for 81 and 45 procedures, respectively.

Under the ITT (train) principle, the surgical reduction achieved by the cadaveric-trained group was superior compared to that by the standard-trained group as measured by acceptable ( ≤ 2 mm) lateral malleolar displacement (odds ratio (OR) 6.82, 95%confidence interval (CI) 2.92 to 15.95; p =< 0.001, linear mixed effect regression), acceptable medial malleolar displacement ( ≤ 2 mm) (OR 5.90, 95% CI 1.00 to 34.67; p = 0.049, linear mixed effect regression) and acceptable tibiofibular clear space (< 5 mm) (OR 12.56, 95% CI 5.25 to 30.05; p =< 0.001, linear mixed effect regression). There were a higher percentage of acceptable medial clear space (78% cases in intervention, 69% cases in control) and talocrural angles (88% cases in intervention, 83% in control) (Table II), but these were not statistically significant in the adjusted model (Table III).

In the additional per-protocol (as-trained) analysis, these findings are consistent with the primary ITT analysis (Table III).

Secondary outcomes

Information on total intraoperative radiation dose was available for 68 cases, and procedure time was available for 124 cases, of 139. Intervention group surgeons administered significantly less radiation to their patients on average than did the control group surgeons (mean difference 0.011 Gym2, 95% CI 0.003 to 0.019; p = 0.009, linear mixed effect regression), Table III. The mean procedure time for the intervention-group was an average of 2.4 minutes faster than for the control group (81.9 vs 84.4 minutes), but this was not statistically significant in the adjusted model (p = 0.686, linear mixed effect regression).

Strengths and limitations

We have attempted to measure the impact of an educational intervention on patient outcomes, which is methodologically challenging. Most simulation research in the literature measures impact using low-level outcomes, such as learner satisfaction and performance in workplace based assessment.^19,20 Evidence of patient benefit is ultimately necessary to show meaningful impact of expensive training interventions.

This study is randomized and multicentre, which increases the internal and external validity of the findings. A variety of clinical environments were included in the study (secondary vs tertiary, major trauma centres), which increases the generalizability of the findings to different educational settings. Similarly, by not restricting our analysis to particular fracture types or implant choice, our findings are applicable to the wide spectrum of injuries treated by extramedullary fixation. We have applied a sophisticated statistical analysis model to adjust for known variables to isolate the training intervention as the exposure of interest.

Limitations of this work include the fact that, for various logistical reasons, there were crossovers between the two arms of the study. This is a reflection of the challenges of conducting real-world educational research. We were surprised that some trainees would not consent to enrolling in the trial and further work needs to be done to understand why. We underestimated the problems with releasing the trainees from their place of work for the cadaveric course. We engaged the programme directors in the conduct of the trial, but not the clinical leads of the orthopaedic departments. Further trials will require ensuring these practical difficulties are predicted for, and overcome, prior to the trial starting.

Of note, the trainees in the intervention group had performed more ankle fracture fixations prior to the course than the control group. This is an important confounder to consider. Randomization should have dealt with this, and it may have been the pragmatic crossovers who introduced an effect. However, there were no measurable differences between those who remained in their allocated groups and those who crossed over, and so the crossover effect is hard to quantify and it may not have been significant.

The prior experience of trainees was accounted for in the regression analysis, and this gives us some confidence that there is still a measurable training effect, even taking prior experience into account. Whether the increased numbers of fixations performed by the training group after the intervention was a result of the intervention or prior experience is unknown. There is likely to be a multiplicative effect as increased experience usually instils confidence and increases competence and training opportunity.

A statistically significant improvement is seen in four of the five primary outcomes and one of the two secondary outcomes, even when controlling for prior experience and undertaking an ITT (i.e. train) analysis. This, with the general trend of all of the outcomes tending to favour cadaveric training, encourages us that the training effect is probably real although the quantum is currently unknown. This should be the subject of further investigation.

The gold standard outcome measurement for clinical results following ankle fracture surgery is arguably the various well-validated patient-reported outcome measures (PROMs). The relevance of radiological measurements for predicting clinical outcome is unclear and our choice of these as primary outcome could reasonably be questioned. We chose to use these as our primary outcome in this study as they are proximate to the time of surgery, the radiographs are taken as a routine part of clinical care and, most importantly, these measurements have been shown to be responsive to changes in surgeon technical skill in prior validation work.¹⁷ There is evidence that loss of congruency of the joint during or after fracture fixation surgery leads to an increased risk of arthritis,⁶ though which measurements are predictive of future osteoarthritis is currently debated.²¹ There are many other factors that influence functional outcome, and we believe any training effect would be obscured by these if using PROMs with small numbers, such as in our study.

For obvious reasons, we were unable to mask participants to their group allocation. We did not undertake a power calculation and used a pragmatic maximum sample size of 16 in each arm of the trial. As we have found a significant difference in our primary outcome between the two training strategies we think it is likely our study was adequately powered to detect this difference.

All cases were supervised by consultant surgeons. We did not measure consultant experience in managing foot and ankle fractures, although this should be balanced between the arms of the study by randomization. We acknowledge that supervisor experience and expertise is likely to influence the measured outcomes.

We had some missing data; some radiographs were not available. Most notably, information on intraoperative radiation dosing was only available for half the cases. This is due to local variation in how this information is captured, some sites do not routinely save the data with the intraoperative fluoroscopy images. The trial protocol did not allow for identifying this data when it was stored by other means. This issue should be dealt with in future studies.

Due to the complex and heterogeneous nature of ankle fractures, our small sample size and the fact this is an educational and not clinical trial, we could not control fracture type or implant choice. We accept these are important variables in the technical difficulty of surgery and as this study is randomized the case-mix should be balanced between the groups. Given the low level of complex fractures we identified in the data (< 10%), we elected to keep the model as parsimonious as possible and thus excluded fracture complexity from the final analysis model.

We have taken a necessarily narrow view of what constitutes a successful ankle fracture fixation for the purposes of this study to isolate the impact of the training intervention on a measurable variable of interest (reduction on postoperative radiograph). We fully accept that a ‘good radiograph’ does not mean a happy patient and the reality is much more complex than this.

Implications of this work

Funding for, and provision of, simulation for surgical training is growing. We need to meet this expansion with robust evidence that this type of training benefits patients. The fact the intervention under test is an educational one should not mean we have lower expectations of evidence of benefit than we would in a clinical trial of, for example, a new treatment for ankle fractures.

There are several priority areas for further research work. The preliminary results presented here need to be replicable in a large-scale RCT, drawing on the lessons we have learned both in the conduct of the trial and the likely training effect. If adequately powered, PROMS should be used as the primary outcome measure.

There is a further question around the role of emerging simulation technologies, such as virtual reality, for which there is a growing evidence base in the trauma²² and lower limb arthroplasty²³ sectors, and how these compare to cadaveric and low-fidelity bench-top ‘plastic bone’ type simulators. Answering this will require further randomized, non-inferiority studies to determine the best type of simulator to use for any given training application.

In conclusion, cadaveric-simulation trained surgeons performed technically superior ankle fracture fixations on real patients, as measured by accuracy of the reduction during surgery. Despite methodological and practical challenges in the trial, this is likely to be a real training effect which requires further study to determine its size and durability. This work indicates that how we train residents has a measurable impact on patients. The clinical meaning of this impact and best application of simulation training for ankle fracture surgery requires further research.