Printer icon Print Page
Hot Topics in Health Innovation

Benchmarking Provincial Health System Performance—Difficult and Controversial but Worth it!

Jan 22, 2013
Gabriela Prada Gabriela Prada
Health, Innovation, Policy and Evaluation

Provincial health performance benchmarking was in the news last week. As is typically the case with this type of assessment, some people are supportive of the findings while others are skeptical of the methodology used and the results. Benchmarking is a powerful tool that can provide insights and stimulus for change. From the eight key steps in any benchmarking process (as outlined in our recent report Measuring Success: A Framework for Benchmarking Health Care System Performance), perhaps the most important step is the first one: Setting the goal of the assessment. In other words, what is the goal of the analysis? What exactly are we trying to measure?

Selecting the goal is not a trivial matter. If the benchmarking process is to inform policy making, the goal needs to inform policy priorities, directions, and strategies. And this goal needs to be grounded in evidence. Once the goal is clear, then it will drive the selection of the evaluation framework and the indicators. The provincial health performance benchmarking assessment that has been in the news recently, which was published by the Fraser Institute, attempted to assess "value for money." This is laudable as "value for money" has become a major objective of governments at all levels. However, measuring it is very difficult.

The selection of the framework and the analysis provided in last week’s provincial health benchmarking report appears to have been driven by a belief that if you pay more, you should get more. This seems like a rational approach. However, I see two issues:

  1. First, ideal or optimal levels of performance are not identified or included as a benchmark. In other words, what is the optimal performance that should/could be attained by dollar invested? More services, when they are not needed, are not necessarily better. The health care system in the U.S. is a good example of wasted resources with no improvement in overall health outcomes.
  2. Second, cost of health care services vary across the country for a number of reasons including variation in population age, urban/rural mix, provider compensation, and other demographic differences. Therefore, if an assessment of this type is to be made, it should recognize the impact of such differences on the overall results. For example, it was concluded that Quebec delivers value for money because its ratio "overall value divided by cost" was among the best when compared to other provinces. This result is probably influenced by the fact that physicians’ salaries in Quebec are the lowest in the country. This has negative impacts on many Quebecers, including great difficulty in getting access to a family physician, but such impacts are not captured because assessment of performance focused very heavily on input (resources available in the system) and volume indicators (transactions). Only a few indicators in the framework selected focused on patient outcomes, and all of these assessed acute care performance only, leaving a huge part of the health care system uncovered.

At The Conference Board of Canada, we are currently in the last stage of our own provincial health performance benchmarking project. Our analysis will be different from the report published last week. Our goal is not to determine value for money, but to assess the overall health of the Canadian population and the state of provincial health systems. Our benchmarking framework takes into consideration that health care system performance is responsible for about 25 per cent of overall health outcomes; other key determinants of health are responsible for about 60 per cent of overall health outcomes, so understanding the differences among the provinces in this regard is essential to set and assess policy directions. (Biology and genetics account for the remaining 15 per cent of health outcomes.) Also, because we acknowledge that more is not necessarily better and that there are not good benchmarks of “value for money,” we will avoid including health care costs in our assessment of provincial performance. This data remains important, and will be included in provincial profiles that will accompany the report to provide better context to the results.

Our analysis includes almost twice as many indicators as the report published last week. It incorporates key measurements of health status, lifestyle factors, and health system performance that go beyond acute care and include screening and prevention, patient-centeredness, accessibility, effectiveness, appropriateness, safety, and continuity. Interestingly, our preliminary results are showing some similarities with the report released recently but also major differences in the overall results.

Benchmarking projects are always challenging and controversial. But when well designed, they provide valuable insights that can lead to performance improvements over time. They also have the power to raise awareness, generate discussion, and enhance accountability, which has been top of mind for Canadians. I think you will find our results insightful. Stay tuned for our report!

Related research

Measuring Success: A Framework for Benchmarking Health Care System Performance

If you enjoyed this article, get regular updates by signing up to our monthly newsletter.

The Hot Topics in Health Innovation blog invites contributions from diverse voices in the fields of health innovation and health policy. If you have news, research, or insights you'd like to share, please contact Jessica Forster-McKewen.