This is Part 3 of a series of articles on ROI for Quality Improvement (QI). Part 1 provided some background for ROI and some basic definitions and processes; Part 2 explored how to quantify costs and benefits. In this article, we will discuss base case analysis versus sensitivity analysis, and what should be included in either.
At this point in your ROI analysis, you have already determined many of the details of your project: you’ve defined the scope and perspective, identified the sources of the relevant costs and benefits, and established monetary equivalents for the direct benefits you expect to observe (and maybe even for some of the indirect benefits). The next step is to put it all together, lay out all the costs and benefits, and then calculate – and interpret – the resulting ROI. However, as I have been eluding to throughout this series, this can be complex and often involves several assumptions, especially if you are including an ROI analysis as part of a proposal to get funding for an initiative that has not yet taken place. Therefore, an ROI analysis with only a single calculation based on a single set of assumptions is woefully incomplete. The ROI analysis is intended to help demonstrate the merit of your initiative, so it is critical that you demonstrate the robustness of the potential return that can be expected by those who fund it. In this article we will discuss not only how to perform base-case and sensitivity analyses, but also the roles they play when crafting the message for the merits of your QI initiative. We’ll also talk about how best to display ROI inputs to make it as easy as possible for your intended audience to understand (and follow) your analysis, and touch on some “tips and tricks” for ROI analyses and reporting.
The Base-Case Analysis
The base-case analysis simply represents the most likely or most reasonable result of your QI initiative. The base-case may not necessarily represent the lowest or most conservative outcome, but should reflect one that is credible and defendable. Here are some ways to ensure a base-case analysis that is rock solid:
When presenting the analytics of your base-case, you will need to clearly describe: the inputs used, their estimates, and the supporting information. Although the details will need to be written out in the text, an effective way to summarize this information is in a table, as in the example below:
This type of summary provides all the necessary information and allows the reader to not only assess the quality of the supporting evidence, but also to clearly see the variety included in the sensitivity analyses.
In addition to this detail, you will want to present the calculation itself: sum up the costs and benefits and calculate the base-case ROI. Whether you include an interpretation at this point or whether you wait until after you present the sensitivity analyses is a matter of personal preference, but I believe that a full interpretation of a base-case ROI requires the context provided by the sensitivity analyses. Therefore, I usually suggest providing a very short interpretation at this point (a sentence or two, perhaps in conjunction with some other metrics like those described in Part 2 of this series) and then providing a more complete interpretation after the sensitivity analysis results are presented.
Typically, a sensitivity analysis includes (at a minimum) best-case and worse-case scenarios, although there may be circumstances that warrant an exploration of even more possible outcomes, as we will see below. Constructing the best- and worse-case scenarios is as much an art as it is a science. You want to be sure to accurately reflect the potential for variability, but you also want to remain securely within the realm of reason. Therefore, I suggest doing a few things:
And, just as additional metrics may help demonstrate the merit of your intervention in the base-case, it is reasonable to include those additional metrics discussed last time (e.g., net benefit per patient, payback period) as part of your sensitivity analyses as well.
Interpreting ROI and Practical Considerations
What level of ROI should I seek to achieve?
During the interpretation of the ROIs from your base-case and best- and worst-cases, you will attempt to make the case that your intervention has merit. To do this, you will need to argue that the expected ROI is adequate (if not exceptional)…but that requires that you somehow establish a threshold for what an adequate ROI would be. That can be more difficult that you might expect, and unfortunately, there is not a lot of specific guidance out there. Buzachero and colleagues (citation at the end of this article) suggest four strategies to establish thresholds for an adequate ROI:
Those are somewhat general, and may not always be helpful (for example, perhaps the funder has no idea what an acceptable ROI would be). One option would be to use ROI results from other interventions as a benchmark. There may be several studies that report results of an intervention similar to the one you are proposing, but they may not have performed an ROI analysis. However, they often provide enough data for you to calculate what the ROI would have been for their intervention, even if only as an approximate value or a range of values. Whatever you use as a benchmark or comparison, the key is to help the reader understand why your estimated ROI is (at least) sufficient to render the initiative worth doing.
How to select a QI project
You already know that when you’re selecting a QI topic area or project, it’s not enough to simply choose an area of need. You also need to ensure that it is an area where it is possible to accurately measure quality so that you can actually quantify an improvement. The same concept applies to ROI: you need to select an area where you can validly monetize the benefits so that you can demonstrate a monetary return. As an example, let’s consider the topic area of patient falls. Typically, falls are classified into two categories: those that result in injury and those that do not. The frequency of either type can be reliably measured to demonstrate improvements in care quality, but only one (injurious falls) may result in expenditures which you can credibly monetize for use in an ROI analysis. That doesn’t mean that you should only consider a QI project that focuses on injurious falls, but it is something to consider when selecting your topic area. At a minimum, you shouldn’t assume you can lump all falls (injurious and non-injurious) together and assign a single monetary benefit to reducing either type.
Tips and Tricks
In addition to the ideas presented above, there are a few tips and tricks that can help you during all phases of your ROI analysis. The first is that if you have a topic area that is of ongoing interest or importance to you and your organization, get in the habit of keeping a folder of information, data sources, and published articles which could inform future ROI analyses. Any time you see a new prevalence estimate or learn about a successful intervention in that topic area, add it to that folder. Then, when it’s time to plan your QI initiative, you have a wealth of information to help guide the design of the project and its associated ROI analysis.
Another tip is to keep a folder of ROI analyses others have done. Whether they are copies of CMS proposals or published studies of ROIs, they not only offer a blueprint for how to go about your own ROI analysis, but they can provide the relevant sources of estimates and comparisons for your own project.
Finally, take the time to create your own templates. If over the years you perform multiple interventions (and therefore multiple ROI analyses), you would like to be able to directly compare their results, which is only possible if they measure costs and benefits the same way and monetize them using the same methodology.
It takes some practice to effectively interweave results from your base-case and sensitivity analysis to clearly convey the potential ROI for a particular QI initiative. However, when considered together they should paint a picture so that the reader feels they have a solid grasp on all of the possibilities and their implications for the ROI.
In the fourth and final article in this series, I’ll explore the notion of ROI in a value-based healthcare environment, and the unique considerations that are necessary within that framework. If you have questions or comments, please feel free to reach out either on LinkedIn or directly at: csolid@SolidResearchGroup.com.
Note: In addition to my own experience, information from multiple sources was used in the development of this series of articles, including: “Measuring ROI in Healthcare: Tools and Techniques to Measure the Impact and ROI in Healthcare Improvement Projects and Programs” by Victor V. Buzachero, Jack Phillips, Patrician Pulliam Phillips, and Zack L. Phillips (McGraw Hill, 2013); AHRQ’s toolkit (https://www.ahrq.gov/professionals/systems/hospital/qitoolkit/index.html), and a white paper by IHI (Sadler BL, Joseph A, Keller A, Rostenberg B. “Using Evidence-Based Environmental Design to Enhance Safety and Quality” Innovation Series white paper. Cambridge, MA: Institute for Healthcare Improvement; 2009).