Blood metal ion levels are not a useful test for adverse reactions to metal debris

Article focus

• We report a systematic review and meta-analysis of the performance of blood metal ion testing for adverse reactions to metal debris (ARMDs) in patients with metal-on-metal (MoM) hip implants.

Key messages

• Blood metal ion levels are not useful as a screening test for identifying high-risk patients because ion testing will either lead to a large burden of false positive results or otherwise marginally modify the pre-test probability.

• With the availability of more accurate non-invasive tests such as MRI, there is no role for using blood ion levels to diagnose symptomatic patients.

Strengths and limitations

• Out of 575 references identified by the literature search, only six met selection criteria.

• These six studies were homogeneous in terms of diagnostic performance estimates.

Introduction

Despite the fact that total hip arthroplasty (THA) has been touted as the operation of the 21st century, orthopaedic researchers continue to propose new designs in an effort to improve implant longevity and patient function.¹ One such purported improvement was the use of a large femoral head with a metal-on-metal (MoM) bearing; this design failed and has become a significant public health concern leading to withdrawal and even recall of certain implant designs.^2,3 This is yet another example of the flawed cycle of innovation in arthroplasty where new implants actually underperform relative to the existing standard.⁴ This alarm over MoM hip implants stems from reported high failure rates, and the potential for locally aggressive ion-induced local tissue reactions such as pseudotumours, a type of adverse reaction to metal debris (ARMD).⁵ It has been estimated that over a million patients have received these implants worldwide, thus posing a significant concern in regard to monitoring and advising patients on both short and long term performance.⁵

In the 1980s, MoM bearings were introduced as an improvement on standard metal-on-polyethylene (MoP) bearings. MoM bearings generate less volumetric wear which may translate into longer implant survival.⁶ In addition, MoM bearings facilitate greater hip stability and range of movement from larger head sizes. Furthermore, MoM bearings allow for bone-conserving hip resurfacing (HR) which is important for younger patients who will eventually require revision surgery.⁷ Both MoP and MoM bearings have been associated with elevated systemic levels of metal ions and ARMDs, however, ion levels have been shown to be consistently higher in patients with MoM bearings, and ARMDs appear to be more common in these patients.^8-11

ARMDs span a spectrum of aseptic necrotic effusions that include pseudotumours as well as aseptic lymphocyte-dominated vasculitis-associated lesions (ALVL) (pericapsular hypersensitivity reactions associated with osteolysis).^12-14 When patients present with mechanical symptoms or pain following a MoM hip implant, unlike with a MoP hip implant, expectant management may not be advisable due to the potential for ARMD lesions which can lead to progressive tissue destruction that compromises reconstructive options.^15-17 As these lesions are diagnosed with ultrasound (US) or metal artefact reduction sequence magnetic resonance imaging (MARS MRI), there has been interest in developing an inexpensive and rapid laboratory test.¹⁸

Government agencies worldwide have published recommendations on the surveillance and work-up of patients with MoM hip implants. The United Kingdom’s Medicines and Healthcare Regulatory Agency (Medicines and Healthcare Regulatory Agency 2012),¹⁹ the United States’ Food and Drug Administration (Food and Drug Agency 2013),²⁰ the European Commission’s Scientific Committee on Emerging Newly Identified Health Risks (European Commission Scientific Committee on Emerging Newly Identified Health Risks 2014),²¹ Health Canada (Health Canada 2012)²² and the Therapeutic Goods Administration of Australia (Therapeutic Goods Administration 2012)²³ have recommended close follow-up of patients, even those with well-functioning implants. Surveillance with metal ions has been recommended for patients with the ASR implant (Depuy Synthes, Warsaw, Indiana) in the United Kingdom, with large-diameter THA and small-diameter hip resurfacing (HR) in Australia, and with large-diameter THA and any HR in Europe. The recommended work-up of symptomatic patients by each of these organisations includes blood metal ion assessment. To date, the most detailed guidelines were put forth by the United Kingdom’s Medicines and Healthcare Regulatory Agency which advised that a blood metal ion level > 7 µg/L indicates potential for soft-tissue reaction. As noted by others, ion level cut-offs are arbitrary and not supported by scientific data.^5,24-26

Given the uncertain utility of laboratory testing in surveillance and investigation of patients with MoM hip implants, there is a need to synthesise the evidence for the measurement of blood cobalt and chromium ion concentrations. In this paper we report a systematic review and meta-analysis of the screening and diagnostic value of metal ion testing for ARMDs.

Materials and Methods

Results

Our literature search identified 575 unique references (Fig. 1). Six met the selection criteria defined a priori (see Appendix 1).^9,11,36-39 We contacted the authors of two studies^9,11 reporting logistic regression with a > 5 µg/L blood ion cut-point for additional data.

Fig. 1

Flow of studies through selection process.

These six studies included a total of 898 hips, of which 376 had an ARMD. The prevalence of ARMDs ranged from 29% to 69%, and the prevalence of symptoms ranged from 23.6% to 100% (Figs 2f and 2g, Appendices 3, 4, 5, 6). Studies differed in the blood fraction tested, ion measured and cut-point used. Only 50% of studies^36,38,39 used MARS MRI as the benchmark for diagnosis. Only one study³⁶ used blood ion levels in a diagnostic context (symptomatic patients) whereas the remaining four studies^9,11,37,39 used blood ion levels in a screening (undifferentiated patients) context. The three studies^9,11,37 not using MARS MRI were deemed at risk of bias (Table I, Fig. 2c, Appendix 3). Two studies^36,37 used plasma, rather than serum, for ion testing and were therefore deemed to have concerns regarding applicability (Table I, see also Appendix 3). No study described the time interval between ion testing and imaging, and therefore all were deemed to have concerns regarding applicability (Table I, see also Appendix 3). All studies were of either Level I or Level II quality.⁴⁰ Prior to proceeding with meta-analysis, we investigated whether clinical, methodological and quality variability manifested in heterogeneity in the estimates of diagnostic accuracy obtained from each study.

Fig. 2

Individual reports of sensitivity and specificity plotted in receiver operating characteristic space with (a) study, (b) ion threshold, (c) benchmark test, (d) index test characteristics, (e) number of hips, (f) adverse reactions to metal debris prevalence, and (g) prevalence of symptomatic patients highlighted. Three studies reported more than one estimate (a). Circle size is proportional to sample size in (e) and prevalence in (f) and (g).

Study-specific estimates of specificity and sensitivity all appear to lie close to a common smooth ROC curve (Fig. 2a), however, there appeared to be variability in the performance of specific cut-points across different studies (Fig. 2b). This suggests that the discriminative capacity of the ion test is homogeneous across studies but that there is substantial cut-point heterogeneity. The cut-point heterogeneity may be due to heterogeneity in the benchmark modality used: the studies reporting unexpectedly high specificity and low sensitivity at 5 µg/L and 7 µg/L cut-points did not exclusively use MARS MRI as the benchmark (Fig. 2c). Sample size, ARMD prevalence, prevalence of symptomatic patients, and ion test characteristics were not associated with cut-point heterogeneity (Figs 2d, 2e, 2f and 2g). Given the homogeneity in discrimination capacity but implicit cut-point heterogeneity we pursued SROC meta-analysis without meta-analysing cut-points (see Appendix 2).

Our best estimate of the “true” ROC curve for metal ion test is the mean SROC curve plotted in Figure 3. However, due to random variability the “truth” may not be the same in all studies. Accounting for this random variability, we have 95% confidence that the study-specific “truth” will lie within the 95% prediction region. The prediction and credible regions have similar widths, which further supported minimal heterogeneity. The area under the curve (AUC) for the SROC curve was 0.615 (95% CI 0.480 to 0.735), thus we can state that the probability that metal ion testing is actually clinically useful with AUC ≥ 0.75 is 1.7% (see Appendix 2).⁴¹

Fig. 3

Summary receiver operating characteristic curve for meta-analysis. Mean curve (—), 95% credible region (---), and 95% prediction region (…) are shown. Sensitivity and specificity reported by individual studies (•).

Due to implicit cut-point heterogeneity, we did not perform meta-analysis of cut-point performance (see Appendix 2). Therefore, the SROC curve in Figure 3 does not relate cut-points to a particular specificity and sensitivity. However, diagnostic performance at any given cut-point will lie somewhere on the SROC curve in Figure 3 – we just do not know where. Hence, our meta-analysis can be used to evaluate the overall performance of ion tests, without reference to a particular cut point.

Discussion

This systematic review and meta-analysis is the first synthesis of evidence for the use of blood ion measurements as a test for ARMDs in patients with MoM hip implants. We identified minimal heterogeneity in the inherent discrimination capacity of ion tests used in each study. Our meta-analysis indicates that blood ion levels are a poor test for classifying patients as having or not having an ARMD.

All but one study included in our review evaluated blood ion levels in a screening context. Estimates of diagnostic accuracy obtained from high prevalence/symptomatic samples can be biased upwards due to the spectrum effect.²² The prevalence of ARMDs and symptomatic patients in the included studies spanned a wide range (29% to 69%), therefore, we could graphically evaluate for a spectrum effect. Since Figures 2f and 2g demonstrated that symptom prevalence was not associated with the operating point, we concluded that there was an absence of spectrum effect in our meta-analysis.

Based on a mean AUC of 0.615, blood ion levels are a poor, and not clinically useful, test for classifying patients as having or not having an ARMD.^41-44 It has been suggested that a clinically useful test has an AUC ≥ 0.75.⁴⁵ Considering the reconstructive consequences of delayed diagnosis, a false negative result could harm patients.^15-17 With the availability of non-invasive tests which definitively determine the presence of an ARMD, we see no role for using blood ion levels to diagnose symptomatic patients.¹⁸

Screening is the process of identifying high-risk patients in the general population. Since screen-positive patients will undergo further testing, screening tests need not be as accurate as diagnostic tests. Screening can use two different approaches: exclude patients with very low probability of disease from further testing by maximising the negative predictive value (NPV), or identify high-risk patients for further testing by maximising the positive predictive value (PPV).⁴⁶ The performance of ion testing using these two approaches is shown in Table II. Calculations were made using the SROC curve plotted in Figure 3 and using the mean prevalence of ARMDs in the studies included in this review (41%). Indeed, maximising NPV is burdensome because 99% of patients will test positive. Furthermore, test-positive patients have the same probability of disease as they did prior to undergoing the test. On the other hand, maximising PPV does not reassure test negative patients because they still have a 21% probability of having an ARMD. Test-positive patients are hardly “high risk” because the risk of an ARMD is marginally different from the pre-test probability (52% versus 41%). Aside from statistical concerns, screening for ARMDs is problematic on theoretical grounds. The World Health Organization recommends that screening only be performed if patients will be offered treatment.⁴⁷ We are unaware of any evidence supporting revision on asymptomatic patients with ARMDs and thus screening would serve no clinical purpose.

We have synthesised the totality of evidence for the diagnostic value of metal ion levels for ARMDs in patients with MoM hip implants. We conclude that blood ion levels have no role in the diagnostic algorithm for ARMDs. The probability that we have incorrectly calculated the AUC to be less than 0.75 is 1.7%. Given the strength and consistency of the findings of our meta-analysis, and the improbability that the results of our meta-analysis are incorrect, further study of metal ion testing for the diagnosis of ARMDs would be an inefficient use of research resources.

A perceived limitation of our study may be that conclusions are based on a small number of studies, half of which did not use MARS MRI as the benchmark modality. We therefore carefully assessed for, and controlled for, heterogeneity. We used a powerful meta-analytic technique that allowed us to partition results into a “cut-point effect” and “accuracy effect” (see Appendix 2). The methodological heterogeneity only manifested in heterogeneity in the cut-point effect, and not in the accuracy effect. Due to heterogeneity, our meta-analysis cannot be used to determine a useful cut-point. However, this is a moot point because the accuracy of the test is so poor. It was remarkable that these methodologically heterogeneous studies formed a smooth ROC curve (Fig. 2). Therefore, there was substantial homogeneity among these studies in the accuracy effect. This homogeneity is further reflected in the fact that the prediction intervals and confidence intervals were nearly equivalent in width (Fig. 3). In other words, our results are tantamount to those from a single study with 898 hips and 376 ARMDs.

We emphasise that our systematic review evaluated the use of blood metal ion levels for the diagnosis of ARMDs. Our findings do not apply to the investigation of the systemic consequences of metal ion exposure which are believed to occur at levels > 60 µg/L.⁴⁸ Further research should be directed to determining how blood ion measurements should be used to investigate cobaltism.⁴⁹

We conclude that the available evidence does not support existing guidelines, which recommend the use of blood ion measurements for both screening and diagnosis of ARMD.