Pitfalls of Using Performance Measures to Evaluate the Quality of Hip Fracture Care

Posted on November 18, 2009

By Nelson F. SooHoo, MD; Bryan Correa, MD; Rajeev Pandarinath, MD
ORTHOPEDICS 2009; 32:84

Abstract

The objective of this study was to determine feasibility of using RAND quality indicators to evaluate hip fracture care. Retrospective chart review was used to determine the adherence to quality indicators and the location of documentation of compliance. A chart abstraction tool was created for systematic extraction of data from multiple chart components. A total of 111 patients underwent operative treatment of a hip fracture and met inclusion criteria in either 1998 or 2003. The main outcome measure was the rate of compliance with quality measures.

Overall, compliance was 88% for the 7 performance measures. Physician notes were the most accurate chart component but, if examined alone, would have only resulted in a reported rate of 81% adherence to indicators. Review of the nursing notes, ancillary service notes, results, and orders was required to fully document quality of care. Ceiling effects were noted for 4 of the 7 quality indicators as noncompliance was rare for these measures.

The results of this study highlight the need for a thorough method of abstracting multiple chart components to accurately report quality of care. This is an important consideration for any pay-for-performance program. Specifically, the failure to review all chart components may lead to incorrect conclusions about the quality of care delivered by individual providers. In addition, the selection of quality measures subject to ceiling effects may limit the usefulness of quality reporting initiatives.

Hip fracture represents one of the most common, severe, and costly injuries suffered by Americans. This public health impact has led hip fractures to be included in attempts to develop performance measures that can be used to objectively evaluate the quality of care delivered to patients.1 A study from RAND indicated that the quality of care for patients undergoing treatment for hip fractures did not meet minimum standards as measured by compliance with a set of evidence-based performance measures.2 Specifically, the patients in their national sample received only 22.8% of the care recommended for hip fracture patients.2 This low compliance included failure to adequately adhere to well-accepted practices including preoperative antibiotics, thromboprophylaxis, and preoperative laboratory testing. These results have been used support the need to improve quality of care with implementation of pay-for-performance programs.3,4 The implementation of pay-for-performance programs has been initiated despite the lack of performance measures that are validated for surgical procedures.5

The purpose of this study was to use chart abstraction methodology to determine the feasibility of using the original RAND performance measures to evaluate quality of care, as well as to document optimal methods of abstracting data from patient medical records.1 We also compared the compliance with the RAND quality indicators for patients undergoing hip fracture care in 1998 and 2003. Our hypothesis was that accurate documentation of quality of care would be found in a variety of chart components. In addition, we sought to determine if the year of surgery would be a statistically significant predictor of compliance, indicating either differences in the quality of care or method of documentation over different time periods.

Materials and Methods

Inclusion and Exclusion Criteria

Institutional Review Board approval was obtained for a review of patients undergoing hip fracture surgery at our institution for the calendar years of 1998 and 2003. International Classification of Disease-Ninth edition diagnosis codes (820.00-820.99) for hip fracture were used to identify 111 patients in our Healthcare Financial Services database with 55 eligible cases in 1998 and 56 eligible cases in 2003 (Table 1). Exclusion criteria included age younger than 50 years, high-impact trauma, multiple fractures, pathologic fracture, no surgical intervention performed, or treatment at another institution.6 We identified 8 patients from 2003 with incomplete access to inpatient medical records, while 28 patients had incomplete records from 1998.6 Patients suffering inhospital mortality during the index admission were also excluded because these patients would not be eligible for the indicator that requires determining the rate of risk factor screening during the initial two months after surgery.

Definition of Compliance

The original set of RAND performance measures for hip fracture included seven measures of the processes of care delivered to patients undergoing treatment of a hip fracture. These are listed in Table 2 and include the need for patients to receive imaging within one day of admission, adequate preoperative evaluation, use of prophylactic thromboembolics on admission to the hospital, use of prophylactic antibiotics on the same day as surgery, initiation of rehabilitation on postoperative day 1, pressure ulcer treatment for at-risk patients, and evaluation for modifiable risk factors for hip fracture.1 For the purpose of this study, a performance measure was considered feasible to measure if it was explicitly documented in at least one data source in one patient. If a given measure was feasible, then compliance was recorded.7 Compliance was defined as the proportion of patients with documented evidence of management consistent with the performance measures. The chart component was the type of document within the electronic or paper medical record that contained the data. For each feasible measure, we determined the rate of compliance accounted for by a given data chart component.

The 7 RAND performance measures include 2 related to diagnosis (D1-2), 4 related to treatment (T3-6), and 1 related to follow-up (F7) (Table 2). Diagnostic imaging (D1) included radiograph, technetium-99m bone scan, magnetic resonance imaging, or dual-energy x-ray absorptiometry scan on the day of admission.

Preoperative medical evaluation (D2) included 1 of the following: full medical history, physical examination, lab evaluation including complete blood cell count, chemistry panel, coagulation studies (either prothrombin time, partial thromboplastin time, or international normalized ratio [INR]), urinalysis, and electrocardiogram. Prophylactic antibiotics (T3) included any antibiotic given on the same day as surgery. Rehabilitation on postoperative day 1 (T4) included evaluation and treatment by physical and/or occupational therapy. Prophylactic thromboembolics (T5) included low-dose heparin (5000 units subcutaneous every 8 to 12 hrs), aspirin (325 mg daily), or low-dose warfarin (to achieve INR of 2.0-2.7 starting immediately after surgery) begun on admission. Pressure ulcer treatment (T6) included both provision of a pressure-reducing mattress (foam, air, gel, or water), and repositioning every 2 hours for those patients either currently with a pressure ulcer, or those that are at risk. Patients were considered at risk if they were nonambulatory prior to the fracture, had current or past pressure ulcers, were malnourished, or had urinary incontinence. Risk factor analysis (F7) included evaluation of modifiable risk factors for subsequent hip fracture within 2 months (before or after) the current surgery. Risk factors included use of long-acting sedatives or anti-convulsants, impaired vision, high caffeine intake (>3 cups/day), inactivity, smoking, and osteoporosis.1

Chart Abstraction

A combination of both electronic and paper chart review was performed for each of the eligible patients. The chart components were classified in the broad categories of physician notes, nursing notes, ancillary service notes, results, and orders. The electronic medical records included inpatient notes during the patient’s stay; follow-up visits were unavailable. Outside office charts and nonhospital documents were not available to review. All chart reviews were performed by 1 of the co-investigators not directly involved with the care of the patients.

A chart abstraction tool was created in Microsoft Access software to record all relevant data. The database was designed to record compliance or noncompliance with each of the RAND performance measures for each type of document found in the chart. As a result, the database records both the overall level of compliance as well as the proportion of compliance documented in each unique element of the patient’s medical record.

Statistical Analysis

All statistical analyses were performed using Stata/SE 8.0. (StataCorp, College Station, Texas). The patient sample is described with the use of bivariate descriptive statistics. Results reported with the use of descriptive statistics include patient demographics and rates of compliance with each of the 7 indicators of quality. Differences between the 1998 and 2003 cohorts of patients were determined with use of the t test for continuous variables and Fisher exact test for categorical variables. Bivariate analysis was used to estimate separately the impact of the year of surgery on the rate of compliance with each of the 7 indicators. For each indicator, bivariate comparison of proportions between the two years, 1998 and 2003 was performed using the Fisher exact test.

Results

Patient Sample

A total of 111 patients were included in the study with 55 patients undergoing treatment in 1998 and 56 patients in 2003 (Table 2). Mean patient age was 79 years with a range of 50 to 99 years. Females made up 76% (84) of the patient sample; 24% (27) were male. The interval between medical clearance and surgery was <36 hours in 95% of cases. The length of stay ranged between 3 and 15 days, with a mean of 6 days. Thirty-four of the patients had a history of osteoporosis and were receiving treatment. There were no statistically significant differences in age, gender, race/ethnicity, time to surgery, length of stay, or preoperative prevalence of osteoporosis when comparing patients undergoing surgery in 1998 to those undergoing surgery in 2003 (Table 1).

Compliance With Performance Measures

The overall rate of compliance for each performance measure is listed in Table 2. There were a total of 691 possible opportunities to comply with recommended care. This includes the entire sample of 111 patients eligible for 6 of the 7 measures. For the seventh measure, 25 of the 111 patients met the definition of patients at-risk for pressure ulcer treatment.

The overall level of compliance was 88% (608 out of a total 691 episodes of care eligible for compliance). There was a ceiling effect for 4 of the indicators, with noncompliance rare for radiographic evaluation (D1-99%), pre-operative evaluation (D2-100%), antibiotic prophylaxis (T3-99%), and rehabilitation on postoperative day 1 (T4-99%). An intermediate level of compliance was found for recommendation for assessment of underlying risks for hip fracture (F7-91%). Low compliance was noted for the assessment and treatment of pressure ulcers among at-risk patients (T6-20%). Compliance was also low for deep vein thrombosis prophylaxis on admission to the hospital (T5-55%).

Documentation of Compliance

The overall compliance using all chart components and the rate of compliance that would have been observed if only 1 chart component was examined are shown in Table 3. The overall compliance of 88% resulted from adherence to 608 of a total 691 opportunities to meet performance measure criteria. The isolated use of physician notes would have indicated an overall compliance rate of 81% as opposed to the 88% seen with use of all the chart components. Lower overall rates of compliance would also have been seen with isolated use of the nursing notes (52%), ancillary service notes (21%), results (31%), or orders (53%). For example, when all data sources were combined, a total of 61 patients (overall compliance 55%) were found to have received thromboprophylaxis on admission. Physician notes documented prophylaxis for 50 of these 61 patients, and nursing notes documented 44 of these 61 patients. If the physician notes had been the only chart component abstracted, it would have appeared that only 50 of 111 patients had received this treatment, and thus compliance would have been inaccurately reported as low with only 45% overall rate of compliance as opposed to the actual rate of 55% when data from all chart elements was combined.

Comparison of Compliance between 1998 and 2003

The compliance rate for patients is listed separately for the group of patients undergoing surgery in 1998 and for those undergoing surgery in 2003 in Table 2. From the bivariate analysis, there was a statistically significant association between the year of surgery and the likelihood of compliance with the quality indicator recommending osteoporosis screening and treatment (P=.02) (Table 2).

Compliance with the remaining 6 quality indicators was examined using bivariate analysis and did not demonstrate any statistically significant associations between compliance and the year of surgery for these other indicators. Bivariate analysis also was unable to demonstrate a statistically significant association between the year of surgery and the rate of overall compliance for the entire set of 7 indicators.

Discussion

The findings of this study identify some of the pitfalls of using quality indicators to evaluate the quality of care related to hip fracture surgery. Specifically, our results highlight the need for rigorous record abstraction to accurately document quality of care using performance measures. An Institute of Medicine report has raised concerns about the added morbidity and mortality that results from errors in medical care.8 This report and others have led to efforts to both measure quality of care and use these results in pay-for-performance programs.3,4,9,10 Recent studies have also noted potential variation in the quality of care delivered by different providers to patients suffering from hip fracture.11,12 Despite these concerns, several authors have noted that the use of pay-for-performance may be premature given the lack of validated quality measures for surgical procedures.5 This study confirms the need for caution in pay-for-performance and other quality reporting programs, as accurate documentation of quality even using a comparatively small set of 7 quality indicators developed for hip fractures requires examination of multiple chart components. We also observed a ceiling effect for 4 of the 7 indicators, suggesting that their usefulness in differentiating between providers may be limited. However, these indicators may be useful in defining a baseline standard of care.

We observed a wide variation in the location in which different aspects of compliance with performance measures was documented. From the perspective of a pay-for-performance program attempting to determine the level of compliance at a given institution, a system should already be established to make this information readily available. Alternatively, a thorough chart review is necessary to determine which chart component most reliably reports the information. Once a thorough review is performed it will be possible to create an algorithm to more efficiently find the information desired. For example, given our data, it is clear that the most reliable place to find documentation of compliance with the RAND hip fracture measures is in the physician notes where we documented an 81% compliance rate compared to 88% when combining data from all the chart components. The remaining chart components were unreliable when examined alone.

Another potential impediment to determination of compliance with indicators is an agreement upon what would be the gold standard of documentation. For example, a progress note may indicate that thromboprophylaxis was to be initiated on admission, but may not actually have been ordered on the same day. Why this occurs is usually not documented. Additionally, it is possible that a patient declines a treatment or intervention. For example, some patients were visited by physical therapy on postoperative day one but declined therapy for a variety of reasons. This would lower the reported compliance unless it is strictly documented, and thus lead to potentially unwarranted adverse judgments of the provider. In addition, we noted a higher rate of incomplete documentation when searching for the older data from 1998 compared to 2003. These issues highlight the importance of well-designed documentation systems to accurately report and maintain a record of the quality of care delivered to patients.

Limitations of this study include the small sample size and restriction to a single institution. Given the high level of overall compliance at 88%, the small sample results in limited power to determine the association of year of surgery to rates of compliance or noncompliance for many of the indicators. Specifically, noncompliance was rare for 4 of the 7 indicators regardless of the year of surgery, including imaging, preoperative evaluation, preoperative antibiotics, and postoperative rehabilitation. This ceiling effect may indicate that many proposed quality measures have limited ability to accomplish the goal of differentiating the quality of care delivered by different providers. However, they may still be useful in programs designed to insure providers meet a basic standard of care. We were able to detect a statistically significant difference between 1998 and 2003 for the proportion of patients undergoing risk assessment for osteoporosis. This preliminary finding indicates the need for further study with a larger sample size from a variety of institutions to determine which indicators are subject to ceiling effects and also to determine the patient and provider factors that predict noncompliance. Our examination of the RAND hip fracture quality indicators suggest that their use in an effective pay-for-performance program would require further study to determine which indicators are both valid and responsive measures of quality care.

Conclusion

This study documents the pitfalls of reporting data on quality of care and identifies specifically the need for rigorous abstraction of multiple chart components when using evidence-based indicators to measure the quality of care related to the treatment of hip fractures. Our findings also highlight ceiling effects for four of the seven RAND quality measures for hip fracture and the need for caution in the implementation and interpretation of pay-for-performance programs that rely on untested measures of quality.5

References

  1. Kerr EA, Asch SM, Hamilton EG, et al. Quality of Care for General Medical Conditions. Santa Monica, CA: RAND; 2000.
  2. McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003; 348:2635-2645.
  3. Demonstration project claims 1 billion dollars in potential savings. Healthcare Benchmarks Qual Improv. 2006; 13:97-99.
  4. Lindenauer PK, Remus D, Roman S, et al. Public reporting and pay for performance in hospital quality improvement. N Engl J Med. 2007; 356:486-496.
  5. Birkmeyer NJ and Birkmeyer JD. Strategies for improving surgical quality–should payers reward excellence or effort? N Engl J Med. 2006; 354:864-870.
  6. March LM, Chamberlain AC, Cameron ID, et al. How best to fix a broken hip. Fractured Neck of Femur Health Outcomes Project Team. Med J Aust. 1999; 170:489-494.
  7. Miller DC, Litwin MS, Sanda MG, et al. Use of quality indicators to evaluate the care of patients with localized prostate carcinoma. Cancer. 2003; 97:1428-1435.
  8. Institute of Medicine (US). Committee on Quality of Health Care in America. Crossing the quality chasm: a new health system for the 21st century. Washington, DC: National Academy Press; 2001:xx, 337.
  9. Sharan DA, Genuario J, Mehta S, et al. Current issues in health policy: a primer for the orthopaedic surgeon. J Am Acad Orthop Surg. 2007; 15:76-86.
  10. Herndon JH. Pay for performance: coming your way soon. AAOS Bulletin. 2005; 53:42-44.
  11. Siu AL, Boockvar KS, Penrod JD, et al. Effect of inpatient quality of care on functional outcomes in patients with hip fracture. Med Care. 2006; 44:862-869.
  12. Roy A, Heckman MG, Roy V. Associations between the hospitalist model of care and quality-of-care-related outcomes in patients undergoing hip fracture surgery. Mayo Clin Proc. 2006; 81:28-31.

Authors

Drs SooHoo, Correa, and Pandarinath are from the UCLA Department of Orthopedic Surgery, Los Angeles, California.

Drs SooHoo, Correa, and Pandarinath have no relevant financial relationships to disclose.

Correspondence should be addressed to: Nelson F. SooHoo, MD, UCLA Department of Orthopedic Surgery, 10945 Le Conte Ave, PVUB #3355, Los Angeles, CA 90095.

Posted in HIP