Press Release

Expert Analysis Finds Bias in STAR Rating Methodology

Study calls into question usefulness of overall CMS star ratings

Contact: Jennifer Schleman, (202) 638-5491, jschleman@aha.org; Marie Watteau, (202) 626-2351, mwatteau@aha.org

WASHINGTON (July 6, 2016) – A new analysis commissioned by the American Hospital Association (AHA) examines the fundamental design flaws of star ratings, and whether it is an accurate tool to compare hospital performance. The Centers for Medicare & Medicaid Services (CMS) star ratings system was designed to help consumers, their families and caregivers compare facilities.

Dr. Francis Vella, chair of the Department of Economics at Georgetown University, conducted an independent assessment of the CMS Overall Hospital Quality Star Ratings methodology. Dr. Vella is a highly-regarded expert in econometrics and has extensive knowledge of the statistical modeling approach CMS used to create overall star ratings.

“As currently designed, CMS’s star hospital ratings program is not up to the task of providing the public with meaningful and accurate assessments of hospital performance,” said Rick Pollack, AHA president and CEO. “Patients need reliable information to make important choices regarding their health care. And hospitals and health systems need reliable information so that they can continue to improve the quality of the care delivered. CMS star ratings misses the mark on both accounts.”

According to the analysis, the approach appears to have several shortcomings:

• There is a wide variation in the number of measures and categories used to identify the star ratings across hospitals. Different hospitals are rated using a number of different measures, which can bias the results, irrespective of their actual performance.
• The methodology is not well constructed. Dr. Vella asserts that “while it appears to give the impression of being rigorous and objective, the estimation aspect is highly dependent on choice of measures and the weighting scheme is entirely subjective and highly determinant of the final outcomes.”
• Ignoring other social determinants of quality outcomes (such as location of hospital, race, income and patient composition) potentially biases the results. As noted by Dr. Vella, “Two (or more) identical hospitals could have very different outcomes depending on the type of patient they have, where they are located, the type of health issues they typically face and multiple other factors.”
• The use of a star system implies that substantial differences in quality may exist across hospitals when they do not.

AHA believes CMS should make substantial changes in the star ratings system before it contemplates making it public. Recently, over half of the U.S. Senate and half of the U.S. House agreed that more work needed to be done.

To review Dr. Vella’s comments, visit http://www.aha.org/content/16/16georgetownmeas.pdf

About the AHA

The AHA is a not-for-profit association of health care provider organizations and individuals that are committed to the health improvement of their communities. The AHA is the national advocate for its members, which include nearly 5,000 hospitals, health care systems, networks, other providers of care and 43,000 individual members. Founded in 1898, the AHA provides education for health care leaders and is a source of information on health care issues and trends. For more information, visit the AHA website at www.aha.org.