Return on Investment (ROI) for Quality Improvement Initiatives - Part 1: ROI Basics

The notion of a return on investment (ROI) is one long associated with manufacturing or banking as a way to justify the purchase of new equipment or facilities, or to obtain external funding (think: Sharktank). Certainly, within the healthcare industry ROI has played a similar role for administrators for years; but more recently it has gained favor as a way to motivate the funding of Quality Improvement (QI) initiatives. That is, instead of discussing only the decrease in utilization and patient burden of successfully reducing infections, for example, many have started to also ask: “How much money will that ultimately save payors?” However, as ROI migrates from boardrooms to the front lines of healthcare delivery and QI organizations, there is a need to understand how to appropriately apply its techniques and accurately perform its calculations.

Over the next several weeks, I will be presenting several articles on ROI for QI. These articles will explore some of the basics of estimating and calculating ROI, and will also provide some guidance for those who need to perform ROI analyses either to obtain buy-in for a proposed QI initiative or to demonstrate the effectiveness of an initiative already in place. The goal is to educate, but also motivate those involved in healthcare QI to embrace ROI as a valuable tool to not only justify a specific initiative, but as a way to help clarify, focus, and plan interventions for maximum effectiveness. At the end of the series I’ll explore how the shifting reimbursement policies towards value-based care may impact the ability to effectively measure and quantify ROI in the future.

In this first article, I present the basic ideas of ROI, explore some of the available resources, and provide some initial aspects one should think about before designing or proposing a QI intervention.

 

Definitions and Terms

ROI is most commonly defined as the following:

(Net benefits from intervention – Costs of intervention)/Costs of intervention x 100%

So, for example, if a QI intervention that cost $25,000 to implement produced $35,000 in net benefits (we’ll get to how to quantify this later on), then we would calculate ROI as follows:

ROI = ($35,000 - $25,000)/$25,000 x 100% = 40%

Whether that’s an acceptable ROI depends on a lot of things which we’ll explore in detail going forward, but the basic interpretation is that the intervention produced a 40% return on the investment made to implement the intervention. Now, it is important not to confuse ROI with another frequently used metric: the benefit-to-cost ratio (BCR). The BCR is exactly what it sounds like: the ratio of benefits to costs, which is similar to the ROI calculation except that the cost of the intervention is NOT subtracted from the benefits in the numerator. So, the BCR for the same example described above would be:

BCR = $35,000/$25,000 = 1.4

The BCR is appealing because it represents the benefit per dollar spent, as in: “For every $1 spent, there was a benefit of $1.40.” And you can see very quickly how the ROI and BCR are related; but, it is important to understand the difference between ROI and BCR because their interpretations for certain values are very different. Specifically, values greater than zero can have very different meanings. For ROI, any value greater than 0% indicates that the benefits covered the cost of the intervention and some additional benefit was realized. However, for BCR, a value between 0 and 1 reflects positive benefits, but not enough to cover the cost of the intervention. That is, a BCR of 0.6 would indicate that the benefits covered only 60% of the cost of the intervention, meaning that financially one would have been better off not having done the intervention (the corresponding ROI for that example would be negative). Only when the BCR is at least 1.0 is it the case that the cost of the intervention was fully covered by realized benefits. For our purposes, we’ll stick to using the ROI, but we’ll come back to the BCR when we discuss alternative metrics one can use to justify the financial viability of QI interventions.

 

Basic ROI Design

The concepts presented below on ROI design reflect information gathered from multiple sources, which I’ll repeatedly refer to throughout this series. The first is a text by Victor Vuzachero and colleagues entitled “Measuring ROI in Healthcare: Tools and Techniques to Measure the Impact and ROI in Healthcare Improvement Projects and Programs” (McGraw Hill, 2013). It is a thorough examination of ROI in the healthcare setting, but lacks much practical guidance for real-world application, in my opinion. There are also references and toolkits available online, including those by AHRQ (https://www.ahrq.gov/professionals/systems/hospital/qitoolkit/index.html) and IHI (Sadler BL, Joseph A, Keller A, Rostenberg B. “Using Evidence-Based Environmental Design to Enhance Safety and Quality” Innovation Series white paper. Cambridge, MA: Institute for Healthcare Improvement; 2009). Incidentally, the resources from AHRQ discuss ROI in theory and definition, but the calculations they suggest using actually reflect those for the BCR, which just underscores the importance of understanding the difference.

A valid ROI design requires several considerations, including defining the scope and perspective, identifying costs and benefits, performing sensitivity analyses, and assessing risks and alternatives. Below I’ll walk through several things to consider when defining the scope and perspective, and we’ll leave details of the other considerations for later articles. The scope and perspective will follow from the underlying design of the intervention itself, and therefore it is appropriate to consider them at the same time that you are designing the intervention. In general, the scope represents the patients, setting(s), and time-frame(s) of interest, and the perspective refers to the group or entity for whom the ROI is realized. To help define scope and perspective, it can be helpful to address several questions:

Who incurs the costs and receives the benefits?

Are you asking CMS to fund this initiative? If so, then the ROI analysis will quantify costs to CMS in the form of reimbursement, utilization, etc. There may be other costs incurred by other parties, but if you’re calculating the ROI from the perspective of CMS, then the costs (and benefits) experienced by an individual hospital or to patients’ caregivers are not likely to be included in the formal calculation. Alternatively, if the intervention is funded at the local level by the facility itself, then you’ll end up using different cost and benefit metrics, and measure different things (e.g., impact on staff turnover, bed availability, bonuses or penalties incurred as a result of performance in quality programs, etc.).

Who is included in the analysis and over what timeframe?

This involves not only the number of patients, providers, or facilities, but also their characteristics: perhaps you are targeting a specific subset of patients (or perhaps only a specific subset is available to you). Or maybe you’re only able to implement this at one or two facilities. Whatever the case may be, this information related to the scope of the project has major implications for the generalizability of your analysis and whether you can “scale up” your ROI estimates to other patient populations or care environments. Even the intensity of the intervention (that is, the level of involvement and attention required by staff to implement the intervention) is relevant when you’re trying to argue whether your observed results are generalizable to other settings and situations. Basically, addressing this question allows you to understand how strongly you can contend that the results of your intervention (and ROI) are applicable to other settings and patient populations.

As for the timeframe, realized returns may vary significantly by time period. Some interventions may have an inherent delay in realized returns (e.g., reducing 30-day readmissions won’t be realized for at least…wait for it…30 days), while others may see an immediate benefit that diminishes over time. Perhaps the real benefits of your intervention will not be realized until after the implementation is completed. In each case, the period of time during which you measure associated costs and benefits can have significant implications for the resulting ROI calculation(s). Selecting the appropriate time period is a function of several things, and we’ll talk about the importance of varying the time period when performing sensitivity analyses in a later article.

Who or what is the comparison group?

Is this a “before vs. after” type of intervention? Or a “case vs. control” intervention? The details of your comparison group(s) speak to your ability to isolate the effects of your intervention and thereby attribute observed benefits (and monetary gains) to that intervention. For example, in a “before vs. after” design you will have to consider any changes in other aspects of care, facilities, staff, health policy, or outside factors which may have also changed between periods, and determine how (if at all) they could have effected your results. At the same time, in case-control designs it can be difficult to mitigate potential spill-over effects that can dilute the observed benefits for the treatment group. Whatever the case, it’s important to consider advantages and disadvantages and think about the potential implications for the resulting ROI analysis.

What are the risks and alternatives?

If you are performing an ROI analysis as part of a proposal, you’ll also need to discuss the potential risks and alternatives. Basically, this becomes an exercise where you explore the assumptions you’re making about things like: facility/patient participation, your ability to implement the intervention, your proposed time-frame, the assumed impact of the intervention, your ability to measure, quantify, and monetize the results, etc. Not only should you clearly lay out all of these assumptions, but you should also address what will happen in each case if an assumption turns out to be WRONG. Here, too, we’ll dive into more detail when we discuss sensitivity analyses.

Once you have specific answers to these questions, your scope and perspective can be clearly defined and articulated. This is a crucial step in the ROI process, and when reporting ROI results, the description of the scope and perspective provide a backdrop for everything that follows: the costs and benefits you identify, the alternatives you consider in your sensitivity analyses, and the generalizability to other settings.

 

Implications for choosing QI interventions

You can quickly see that planning ROI analyses go hand-in-hand with planning the intervention itself. Just as designing an intervention requires that you consider how you will measure changes in care quality or effectiveness and that you can ensure the validity and reliability of those measurements, you’ll also need to consider what data you’ll need for the ROI analyses and how you’ll leverage the results to justify the cost of the intervention to the funding source. The good news is that thinking about ROI can help improve the design of your intervention and potentially focus your efforts on the most appropriate aspects of improving care while pursuing efficiency and a reasonable return on investment.

 

Next Steps

In the next article, we’ll dive into the details of quantifying costs and benefits, discuss direct versus indirect benefits, and explore the different methods for identifying credible sources for the monetary values you assign to both costs and benefits.