Understanding CMS' Changes to Hospital Overall Star Ratings

The Centers for Medicare & Medicaid Services (CMS) in 2020 overhauled the Hospital Overall Star Ratings methodology to make ratings more transparent, equitable, stable and predictable. To improve transparency the agency moved away from the statistically heavy and difficult to interpret latent variable modeling approach and instead employed a simple average of measure to calculate measure group scores. As a way to attempt to achieve more equitable comparisons, CMS also created peer groupings by number of reported measures. To improve the stability and predictability of star ratings the weights applied to the different measures would be set in advance by CMS. On the whole, while CMS expected the new methodology could change the overall ratings distribution, it believed period-to-period changes would remain fairly stable.

AHA commissioned KNG Health to assess whether these methodological changes have fully achieved CMS' goals, while also estimating expected performance variation and drivers of performance and variation under the new methodology. The KNG analysis entailed:

• Comparing 2020 (old methodology) to 2021 (new methodology) hospital star ratings performance reported by CMS, assessing differences in ratings by hospital characteristics.
• Assessing the stability of CMS’ new methodology by comparing hypothetical 2020 performance under the new methodology to 2021 actual performance; and
• Assessing the equitability of the ratings through analyses of peer groupings.

Overall the new methodology preserves some year-to-year stability. When applying the new methodology to both 2020 and 2021 the distribution of ratings are similar (see Figure 1). Sixty percent of hospitals could have expected to receive the same score in 2020, with 36% seeing an increase or decrease of one star rating. 

View the detailed issue brief below.