Return on Investment (ROI) for Quality Improvement Initiatives, Part 3: Base Case and Sensitivity Analysis
This is Part 3 of a series of articles on ROI for Quality Improvement (QI). Part 1 provided some background for ROI and some basic definitions and processes; Part 2 explored how to quantify costs and benefits. In this article, we will discuss base case analysis versus sensitivity analysis, and what should be included in either.
At this point in your ROI analysis, you have already determined many of the details of your project: you’ve defined the scope and perspective, identified the sources of the relevant costs and benefits, and established monetary equivalents for the direct benefits you expect to observe (and maybe even for some of the indirect benefits). The next step is to put it all together, lay out all the costs and benefits, and then calculate – and interpret – the resulting ROI. However, as I have been eluding to throughout this series, this can be complex and often involves several assumptions, especially if you are including an ROI analysis as part of a proposal to get funding for an initiative that has not yet taken place. Therefore, an ROI analysis with only a single calculation based on a single set of assumptions is woefully incomplete. The ROI analysis is intended to help demonstrate the merit of your initiative, so it is critical that you demonstrate the robustness of the potential return that can be expected by those who fund it. In this article we will discuss not only how to perform base-case and sensitivity analyses, but also the roles they play when crafting the message for the merits of your QI initiative. We’ll also talk about how best to display ROI inputs to make it as easy as possible for your intended audience to understand (and follow) your analysis, and touch on some “tips and tricks” for ROI analyses and reporting.
The Base-Case Analysis
The base-case analysis simply represents the most likely or most reasonable result of your QI initiative. The base-case may not necessarily represent the lowest or most conservative outcome, but should reflect one that is credible and defendable. Here are some ways to ensure a base-case analysis that is rock solid:
- Stick to direct benefits. For the base-case, using only direct benefits when estimating the monetary return will result in an ROI estimate that represents the tangible monetary/fiscal return and will provide the strongest “business case” for your initiative. When interpreting the results (either in written form or when presenting your results in person), you will certainly discuss the indirect benefits, but you will leave the monetary translation of those indirect benefits to the sensitivity analysis.
- Use estimates of prevalence, incidence, effectiveness, etc. that are supported by evidence and for which you can explain how and why they apply to your project. Again, these may not be the most conservative estimates. For example: you may find a variety of published literature that demonstrates that educational interventions on infection prevention reduce hospital-acquired infections anywhere from 10% to 40%, depending on the characteristics of the intervention, the setting, and the timeframe. Perhaps there’s one example in particular that you feel closely reflects your intervention and which resulted in an 18% reduction in infections. It may be the case that for the base-case scenario, an 18% reduction is the most appropriate estimate to use, even though it is not the most conservative one. That’s not to say you ignore the range of estimates from the published literature, however. To be clear, you would need to demonstrate to the reader why you feel the 18% is reasonable. You would also be wise to address the large spread in the published rates of success of education-based interventions (10% to 40%) since variability means uncertainty, which can reduce the credibility of your ROI analysis. In our example, perhaps the study that observed a 40% reduction is unique (unusual setting, uncommon staffing situation, high baseline value providing a large improvement opportunity, etc.) or not credible in your opinion (e.g., small sample size, unclear methods, etc.); and maybe you can argue that the study or studies that produced only a 10% reduction are not applicable to your project (different care settings, different circumstances, etc.). This not only provides reasoning for the estimate you chose, but also demonstrates your knowledge of the subject area and that you have considered all possibilities. In the end, you are best positioned to determine which estimates are the most applicable to your situation, but you should be able to defend the estimates you have chosen.
- Try to “poke holes” in your base-case assumptions regarding costs, time-frame, staff effort, and the effectiveness of your intervention. It can be difficult to be completely objective when compiling the base-case scenario because you have a vested interest in its success. Therefore, you and your team should attempt to identify potential problem areas and either secure the proper evidence/support for the choices you have made or make more conservative choices. You may also want to have an outside party weigh in to see what jumps out at them.
When presenting the analytics of your base-case, you will need to clearly describe: the inputs used, their estimates, and the supporting information. Although the details will need to be written out in the text, an effective way to summarize this information is in a table, as in the example below:
This type of summary provides all the necessary information and allows the reader to not only assess the quality of the supporting evidence, but also to clearly see the variety included in the sensitivity analyses.
In addition to this detail, you will want to present the calculation itself: sum up the costs and benefits and calculate the base-case ROI. Whether you include an interpretation at this point or whether you wait until after you present the sensitivity analyses is a matter of personal preference, but I believe that a full interpretation of a base-case ROI requires the context provided by the sensitivity analyses. Therefore, I usually suggest providing a very short interpretation at this point (a sentence or two, perhaps in conjunction with some other metrics like those described in Part 2 of this series) and then providing a more complete interpretation after the sensitivity analysis results are presented.
Typically, a sensitivity analysis includes (at a minimum) best-case and worse-case scenarios, although there may be circumstances that warrant an exploration of even more possible outcomes, as we will see below. Constructing the best- and worse-case scenarios is as much an art as it is a science. You want to be sure to accurately reflect the potential for variability, but you also want to remain securely within the realm of reason. Therefore, I suggest doing a few things:
- Consider the assumptions you are making, the risks involved, and the possible alternatives. What are you assuming about patient participation, your ability to implement the intervention, the validity of your measurements, or the impact of your intervention? Perhaps your ROI is driven mostly by one estimate that reflects the effectiveness you can achieve. It may be the case that part of your sensitivity analysis will be to explore multiple scenarios while varying only this one estimate to demonstrate the impact it can have on the ROI. There may also be assumptions that are contingent upon circumstances outside of your control; the recruitment of participants (either facilities or patients) and the time required to collect data, for example, are notoriously difficult to predict with accuracy. Be honest about what happens if one or more key aspects completely fall through. This is as much an exercise for you to understand where you might encounter challenges as it is a justification to those providing the funding. If you can identify all the potential pitfalls and still provide for how the intervention can remain effective, you will have gone a long way towards ensuring (some level of) success and in turn demonstrating the merit of the initiative or intervention.
- Examine the range of estimates you are considering in light of your specific situation. In the example presented previously, there was noticeable variability in the achieved reduction in infections (10% to 40% reduction) from the published literature of similar interventions. An appropriate sensitivity analysis would explore reasons for this large gap and assess what portion of this range is reasonable to apply to the best-case and worse-case scenarios of your project. As mentioned above, perhaps it is not reasonable to apply the 40% estimate to your intervention; you may instead select one that is slightly lower. You may wonder at this point whether it is reasonable to apply estimates that lay outside of the range that have been previously observed (either from published data or from your experience). There may be cases where this is reasonable, but I caution against including estimates that are better than those previously observed anywhere – you want your best-case scenario to still be feasible and credible. Finally, your best-case scenarios are where you can include monetary estimates of some indirect benefits, if applicable and reasonable – but, you may want to present them as a separate category.
- Vary the time-frame. How does the ROI change if you consider the short-term or the long-term? Are there certain outcomes that become more or less common or impactful over time? Maybe there is a cumulative effect of your intervention that you want to demonstrate. That is, maybe doubling the length of the time-frame during which you evaluate costs and benefits more than doubles the ROI. That certainly would be of interest to the reader.
And, just as additional metrics may help demonstrate the merit of your intervention in the base-case, it is reasonable to include those additional metrics discussed last time (e.g., net benefit per patient, payback period) as part of your sensitivity analyses as well.
Interpreting ROI and Practical Considerations
What level of ROI should I seek to achieve?
During the interpretation of the ROIs from your base-case and best- and worst-cases, you will attempt to make the case that your intervention has merit. To do this, you will need to argue that the expected ROI is adequate (if not exceptional)…but that requires that you somehow establish a threshold for what an adequate ROI would be. That can be more difficult that you might expect, and unfortunately, there is not a lot of specific guidance out there. Buzachero and colleagues (citation at the end of this article) suggest four strategies to establish thresholds for an adequate ROI:
- Use what other industries try to achieve: 15% to 20%
- Use a value that is above that expected for other types of investments (i.e., alternatives that the funding body could spend the money on instead)
- Use the break-even value (ROI = 0%)
- Let the funder set the minimum acceptable ROI
Those are somewhat general, and may not always be helpful (for example, perhaps the funder has no idea what an acceptable ROI would be). One option would be to use ROI results from other interventions as a benchmark. There may be several studies that report results of an intervention similar to the one you are proposing, but they may not have performed an ROI analysis. However, they often provide enough data for you to calculate what the ROI would have been for their intervention, even if only as an approximate value or a range of values. Whatever you use as a benchmark or comparison, the key is to help the reader understand why your estimated ROI is (at least) sufficient to render the initiative worth doing.
How to select a QI project
You already know that when you’re selecting a QI topic area or project, it’s not enough to simply choose an area of need. You also need to ensure that it is an area where it is possible to accurately measure quality so that you can actually quantify an improvement. The same concept applies to ROI: you need to select an area where you can validly monetize the benefits so that you can demonstrate a monetary return. As an example, let’s consider the topic area of patient falls. Typically, falls are classified into two categories: those that result in injury and those that do not. The frequency of either type can be reliably measured to demonstrate improvements in care quality, but only one (injurious falls) may result in expenditures which you can credibly monetize for use in an ROI analysis. That doesn’t mean that you should only consider a QI project that focuses on injurious falls, but it is something to consider when selecting your topic area. At a minimum, you shouldn’t assume you can lump all falls (injurious and non-injurious) together and assign a single monetary benefit to reducing either type.
Tips and Tricks
In addition to the ideas presented above, there are a few tips and tricks that can help you during all phases of your ROI analysis. The first is that if you have a topic area that is of ongoing interest or importance to you and your organization, get in the habit of keeping a folder of information, data sources, and published articles which could inform future ROI analyses. Any time you see a new prevalence estimate or learn about a successful intervention in that topic area, add it to that folder. Then, when it’s time to plan your QI initiative, you have a wealth of information to help guide the design of the project and its associated ROI analysis.
Another tip is to keep a folder of ROI analyses others have done. Whether they are copies of CMS proposals or published studies of ROIs, they not only offer a blueprint for how to go about your own ROI analysis, but they can provide the relevant sources of estimates and comparisons for your own project.
Finally, take the time to create your own templates. If over the years you perform multiple interventions (and therefore multiple ROI analyses), you would like to be able to directly compare their results, which is only possible if they measure costs and benefits the same way and monetize them using the same methodology.
It takes some practice to effectively interweave results from your base-case and sensitivity analysis to clearly convey the potential ROI for a particular QI initiative. However, when considered together they should paint a picture so that the reader feels they have a solid grasp on all of the possibilities and their implications for the ROI.
In the fourth and final article in this series, I’ll explore the notion of ROI in a value-based healthcare environment, and the unique considerations that are necessary within that framework. If you have questions or comments, please feel free to reach out either on LinkedIn or directly at: csolid@SolidResearchGroup.com.
Note: In addition to my own experience, information from multiple sources was used in the development of this series of articles, including: “Measuring ROI in Healthcare: Tools and Techniques to Measure the Impact and ROI in Healthcare Improvement Projects and Programs” by Victor V. Buzachero, Jack Phillips, Patrician Pulliam Phillips, and Zack L. Phillips (McGraw Hill, 2013); AHRQ’s toolkit (https://www.ahrq.gov/professionals/systems/hospital/qitoolkit/index.html), and a white paper by IHI (Sadler BL, Joseph A, Keller A, Rostenberg B. “Using Evidence-Based Environmental Design to Enhance Safety and Quality” Innovation Series white paper. Cambridge, MA: Institute for Healthcare Improvement; 2009).