Do Financial Benchmarks Truly Measure ACO Savings, Spending?

June 21, 2017 - Healthcare stakeholders should not use financial benchmarks developed by CMS to evaluate actual accountable care organization (ACO) savings, three Harvard Medical School experts in a recent Health Affairs blogpost.

As Medicare ACOs grow in popularity, the healthcare industry continues to evaluate how the alternative payment model and care delivery system has decreased costs. However, stakeholders have met challenges with determining how to evaluate actual ACO savings.

For example, CMS reported in 2014 that the 20 ACOs in the Pioneer model and the 333 in the Medicare Shared Savings Program (MSSP) saved a total of $411 million. But the ACOs actually generated a net loss of $2.6 million for Medicare after the federal healthcare program paid out incentive payments to high-performing ACOs that year, Kaiser Health News reported in 2015.

CMS and other research indicated that ACO savings to the industry as a whole may take more time, while other stakeholders have argued that the financial losses or limited savings suggest ACO payment reform is needed.

However, Michael Chernew, PhD, Christopher Barbey, and J. Michael McWilliams, MD, PhD, claimed in the Health Affairs blogpost that the industry needs an appropriate counterfactual scenario to accurately assess ACO savings and spending. Without an agreed-upon scenario of what ACOs would have spent if they were not under the ACO contract, stakeholders cannot calculate true ACO savings.

READ MORE: Best Practices for Value-Based Purchasing Implementation

“While we can observe actual spending for ACO beneficiaries, it is impossible to observe the counterfactual,” they wrote. “Therefore, it’s very hard to know what would have been spent if ACOs did not exist.”

They noted that some stakeholders have pointed to ACO financial benchmarks as a way to identify ACO savings. The financial data are publicly available and used by CMS to determine ACO shared savings. CMS also uses ACO financial benchmarks to report savings.

However, Chernew et al. warned stakeholders against using the benchmarks.

“It is important to avoid using the Centers for Medicare and Medicaid Services’ (CMS’s) benchmarks to evaluate ACO savings because benchmarks are constructed with policy goals in mind,” they contended. “Benchmarks are not designed to reflect the counterfactual scenario in which the providers were not under the ACO contract.”

Financial benchmarks for ACOs aim to accomplish policy goals, such as promoting ACO participation, developing incentives to reduce healthcare spending, and achieving program fiscal objectives. The benchmarks are not designed to function as a counterfactual scenario.

READ MORE: How Pioneer ACOs Earn Shared Savings, Improve Care Quality

The financial benchmarks are also invalid counterfactuals, the experts argued. Under the MSSP, ACO financial benchmarks during the first contract period are based on three years of historical spending on assigned beneficiaries. Then, CMS trends forward the weighted and risk-adjusted average spending for those three years to account for the national average Medicare spending growth.

CMS uses the same national average Medicare spending growth rate to adjust ACO financial benchmarks for organizations in all regions. Therefore, the ACO savings estimates are not accurate because Medicare spending growth significantly varies across regions.

“So, judging savings by comparing spending to the benchmark arbitrarily favors ACOs in areas where health care spending grew slower than the national average,” the blogpost stated. “ACOs in low-spending growth rate areas could ‘save’ simply by maintaining the status quo trend in the region because benchmark growth will outpace spending growth in that region by definition.”

“Similarly, ACOs in high-spending growth regions will appear to fail, even if they saved money relative to their non-ACO neighbors,” it continued.

Additionally, the experts found that Medicare ACO financial benchmarks actually underestimate true ACO savings. Using Medicare claims data from 2012 to 2014, they uncovered that benchmarks systematically underestimated savings because MSSP participation has been disproportionately greater in locations with faster Medicare spending growth.

READ MORE: The Future of Accountable Care Organizations Involves Risk

For every 0.5 percentage point higher rate of spending growth among non-ACO providers in a hospital referral region, MSSP ACO penetration was 25 percentage points higher.

The national Medicare spending growth rate also contributed to underestimated ACO savings. For example, if ACOs reduced Medicare reimbursements by 3 percent as the MSSP 2012 entry cohort did in 2014 and if one in three Medicare fee-for-service beneficiaries belonged to an ACO as is the current case, then national Medicare spending growth would be 1 percent slower because of ACOs.

Therefore, benchmarks would be 1 percent lower than the actual counterfactual scenario since CMS uses the growth rate to calculate financial targets.

To resolve the absence of appropriate counterfactual scenarios to evaluate true ACO savings, the experts suggested that the counterfactual be “set at an ACO’s expected level of spending based on its baseline spending and the concurrent spending trend in its service area among a control group of non-ACO providers.”

Although the approach does not account for Medicare patients treated at the ACO, but not assigned to the alternative payment model and care delivery system, the experts pointed out. If strong patient spillover exists, using the regional spending trends for counterfactuals could underestimate true ACO savings.

“Constructing a valid counterfactual is difficult, but it is necessary, and analysts will continue to disagree about the best approach,” the experts concluded. “However, thoughtful analysis of the effects of any program, including the ACO program, must avoid the temptation to use easily available figures if they are not designed to capture the true program effects.”