Proposed Changes to Dialysis Facility Compare Tool May Make It More Useful

Dialysis Facility Compare (DFC) is an online tool that allows users to search and compare dialysis facilities based on star ratings and certain measures. DPC has previously submitted suggestions on how to improve the proposed measures. Recently, the Centers for Medicare and Medicaid Services (CMS) announced in a Planned Changes document that the star ratings included in the DFC would be reassigned based on a 2014 threshold, meaning that the distribution of ratings would look more like it does on CMS’ other websites, and apply standards that are more familiar and intuitive to consumers. DPC submitted another comment letter (published below) to CMS, applauding the apparent openness to a new star rating methodology.

A key feature of CMS’ current star ratings structure is a bell-curve grading that limits 4- and 5-star ratings to facilities ranked in the top 30 percent, and forces the bottom 30 percent into 1- and 2-star ratings. No other federal website that uses star ratings issues as many 1- and 2-star ratings, and some do not bestow such low ratings at all. It appears that CMS has retreated from insistence on treating dialysis clinics differently from hospitals, nursing home, and health plans.

DPC’s letter emphasizes that “maintaining a degree of stability in the star ratings must be a key consideration” in order to keep consumers’ confidence in the DFC’s accuracy. This year 2,500 facilities’ ratings were changed, and for every facility that increased in ratings another facility’s rating was lowered, regardless of its performance. DPC argues that downgrading ratings should only apply to facilities that have experienced an actual drop in performance quality; a feature that is included in CMS’ Planned Changes. DPC will continue to monitor CMS’ activities on this patient tool and report any updates in the future.

Read DPC’s letter below:

Kate Goodrich, M.D.
Chief Medical Officer
Centers for Medicare & Medicaid Services
7500 Security Boulevard
Baltimore, MD 21244

Re: Planned Changes to the DFC Star Rating Methodology

Dear Dr. Goodrich:

We are pleased to see that, after much back and forth, progress is being made in incorporating stakeholder concerns into the design of the DFC star ratings program. The immediate outcome proposed in the Planned Changes—using the bell-curve thresholds for 2014 as a fixed baseline and reassigning star ratings to encompass improved performance since that time—is acceptable to us as a next step. However, given the need to periodically re-set the baseline, we are concerned that the Planned Changes document is silent on how standards will be set upon re-basing. Many in the kidney community have interpreted the silence as an indication that DFC would return to the stack-ranking method when re-setting the cut-offs. We are perhaps more optimistic that accord can be reached on this question, but because the document refers to possibly re-basing upon the addition of new measures, and potential new measures will be ready for inclusion in the very near future, we must regrettably temper our approbation with a sense of urgency that a new system be ready to supplant the bell curve should that option be chosen.

Maintaining a degree of stability in the star ratings must be a key consideration. There were changes in star rating awards to 2500 dialysis facilities in the latest iteration of DFC. Per research by Mark Stephens reported in NNI, it appears that many facilities whose scores were near the previous version’s cutpoints were repositioned. We have two concerns about such changes. First, year-to-year volatility of the ratings—with 45 percent of facilities changing places—may undermine consumer confidence in their authoritativeness. Second, for patients in the 1250 facilities whose ratings were downgraded, the change may be alarming, conveying a sense that their facility is in some way troubled. We believe downgrading of facilities should be limited to those that actually experienced an absolute drop in their quality performance that reflects poorer quality and we interpret the Planned Changes document to mean that CMS agrees with us on this principle.

As the Planned Changes document recognizes, another important consideration is avoiding a lopsided distribution, as where more than half of facilities are rated as “above average.” We agree that new baselines must be set in such circumstances in order for the star ratings to retain legitimacy.

CMS must appropriately balance these competing considerations. In the recent case of Nursing Home Compare, star ratings were re-set after about five years. This is not, perhaps, the best case to cite, given that the rebasing occurred after advocates and journalists called into question the appropriateness and usefulness of the prior system. DFC is different in that we don’t believe the gaming that nursing homes engaged in is possible in DFC. But a four-to-five year run before rebasing would seem to strike the proper balance between consumer preferences for stability and the need to periodically raise the bar on quality expectations. HHS’ Healthy People 2020 initiative sets 2020 as the target year for achieving certain health outcomes so it seems fitting to us that 2020 be the target year for re-setting star rating cut-offs.

We have expressed a number of concerns about the Dialysis Facility Compare star ratings program, but at present our policy priority is aligning its rating criteria with those used in other CMS programs. We are able to support the next step described in the Planned Changes document because it reallocates DFC’s star awards so that its distribution will resemble those on the other CMS sites that beneficiaries and caregivers are likely to visit. In the new distribution, fewer patients will be alarmed by one- and two-star ratings that are identified in the public’s mind with substandard quality.

The table below compares the current distributions of stars across CMS’ consumer-facing sites. On Dialysis Facility Compare, thirty percent of facilities are awarded one or two stars; by contrast, fewer than one percent of Medicare Advantage plans and twelve percent of home health agencies receive such ratings. If one considers three stars to represent “average,” then 49 percent of nursing homes and home health agencies, and 60 percent of MA plans are portrayed as “above average,” while only thirty percent of dialysis facilities are so portrayed.

* denotes detailed breakdown could not be found.

We realize that this can seem somewhat abstract, so we have prepared two additional charts that graphically depict the star ratings that a consumer is likely to see while searching for providers and plans in two localities. Imagine an ESRD patient’s caregiver, who may be a spouse over 65 or a son or daughter, who has also searched for or assisted a beneficiary in search of a Medicare Advantage plan. We expect that, owing to the complex needs of ESRD patients, this caregiver may also have searched for hospital and post-acute care provider ratings prior to or contemporaneous with a search for dialysis facility ratings.

The first graph depicts the distribution of star ratings a consumer will find in Dayton Ohio. Because of the greater weight given to health outcomes on DFC relative to other CMS rating systems, Ohio has disproportionate number of 1- and 2-star dialysis facilities; but this is not the case with other rated entities. The graph depicts the ratings of the first 12 entities that are listed on five CMS sites. (Home health ratings on our graphs are for patient experience—Home Health Compare issues separate star ratings for clinical quality and patient experience.)

Of the first 60 entities listed, 12 are below 3 stars and 29 are above 3 stars. Of the 12 entities with 2 stars or below, eight are dialysis facilities. Of the 29 entities with 3 ½ stars or above, none are dialysis facilities. To the beneficiary who has been exposed to the other CMS star systems (or for that matter, systems like safercars.gov or Yelp) the sight of solely 2- and 3-star facilities nearby will be jarring. While 4- and 5-star hospitals, post-acute providers and health plans are readily accessible in Dayton, the nearest 4-star dialysis facility is 21 miles north in Troy, Ohio.

The second graph depicts the distribution of star ratings in Springfield, Missouri. Because fewer than twelve rated home health agencies and MA plans are available in Springfield, this graph shows ten entities in each provider category and nine in the health plan category.

Missouri is a state in which dialysis facilities are symmetrically distributed, and this pattern is seen in Springfield. However, the distributions of other entities skew to the high end as in Dayton: overall, 5 entities are below 3 stars while 28 are above. The effect of the distribution is to make 3-star entities look mediocre or laggard. This impression would be accurate with regard to most categories, since those star systems assign ratings above 3 stars to 41, 49, 49, and 60 percent of rated entities. But the impression is inaccurate with regard to dialysis facilities since 3 stars purports to mean “average” and only 30 percent of facilities can earn ratings above that.

To be clear, we expect such disparities in star rating systems to be a problem only in two thirds of the country. In the northwest, mountain west, upper Midwest and New England, the superior health of local populations skew DFC star ratings upwards so as to roughly match the skewing of the other rating systems.

We believe CMS’ other star rating systems appropriately convey a note of caution in identifying a small number of truly poor performers with one or two stars. They follow the conventions of movie reviews, Yelp, airline ratings, and other star systems familiar to the average consumer. It remains unclear to us why DFC was given a unique design nor why CMS would see a benefit to retaining its unusual design since dialysis patients and their caregivers are arguably the demographic group most likely to have reason to search multiple “Compare” websites and therefore experience confusion. We are therefore relieved that the Planned Changes document indicates that CMS will, at least for the immediate future, step away from this system.

We want to emphasize that we do not dispute the normative considerations behind DFC’s bell-curve distribution. We agree that ranking providers among their peers harnesses positive competitive dynamics and can provoke self-critical analysis among practitioners who all too typically believe themselves to be “above average.” We believe, and we suspect that the DFC team agrees, that awarding 4- and 5-star ratings to 40 percent of facilities, and a greater percentage in the future, is a less-than-ideal outcome.

Our concern has always been the disconnect between these idealized notions of rankings and the public’s experience with and understanding of star ratings. Since we raised that initial objection 18 months ago, CMS has rolled out new star ratings or modified old star ratings that adhere to the common-understanding paradigm rather than to the idealized paradigm manifested in DFC’s first iterations. As such, awarding four or five stars to 40 percent or more of dialysis facilities can be justified for purposes of harmonizing the various CMS programs, as well as conveying to consumers a sense, which we believe is accurate, that overall care is improving.

Thank you for your consideration of our comments. As you’ll recall, we had requested a meeting with you or Dr. Conway to discuss this matter in detail. We no longer feel an immediate need to hold such a discussion. However, if there are plans to revert to the bell-curve distribution in the near future we believe it would be helpful to gather and confer with the CMS personnel overseeing the various star rating systems to go over the disparities illustrated on the attached charts and seek a more uniform experience for consumers who visit CMS’s website.

Respectfully submitted,

Jackson Williams
Director of Government Affairs