Breaking Down the Arguments for Process vs Outcome Measures, Including the Role of Risk Adjustment
As quality measures are developed and quality monitoring programs seek to refine their measure list to those that are the most impactful and least burdensome, there will continue to be discussions about the appropriateness of different measures for different programs, clinical areas, and provider types. There will also be a lot of opinions and assessments of current measures, as individuals and groups advocate for certain measures and discourage the use of others. Although many of the examinations of measure appropriateness typically focus on the specifics of an individual measure, you have probably noticed that from time to time there are arguments for or against “types” of measures, including those comparing and contrasting process measures versus outcome measures. Therefore, a closer examination of these types of measures is warranted, including the strengths and weaknesses of each.
These are, in general, measures regarding the steps providers take (or don’t take) during a patient encounter and/or in the course of providing care. Process measures are intended to encourage providers to do things such as screen for a condition or risk-factor, take the time to go over a surgical safety checklist, provide discharge instructions, etc. Proponents of process measures laud several of their attributes:
- Process measures often reflect or are intended to promote evidence-based best practices that have been proven to improve outcomes or the patient experience of care;
- Process measures are (usually) directly measurable and immediately available; this makes them trackable and easily targeted in a Quality Improvement (QI) intervention because they are actions one can encourage or discharge, (e.g., EHR alerts or reminders to screen high-risk patients for potential harms such as infections);
- Often the “culture” of a facility or department is cited as a primary catalyst for effective change, and process measures that encourage certain behavior may help to facilitate changes in culture (e.g., an OR time-out that requires certain staff to speak up who may not have done so previously).
When people talk about the drawbacks of process measures, they usually discuss the following:
- Sometimes it can be difficult to demonstrate a direct correlation between improvements in a process measure performance and a corresponding improvement in patient outcomes;
- Certain process measures can be difficult to operationalize if there are complex criteria regarding which patients or situations to which the process applies (e.g., maybe there are viable reasons why door-to-needle time may be delayed for certain patients);
- In some cases, process measures may unintentionally introduce surveillance bias or increase the likelihood of false-positives (e.g., higher rates of VTE-related diagnostic imaging may lead to higher rates of VTE).
There is such a variety of process measures that both the pros and cons listed above are more applicable to some measures than to others. And, certainly these are not exhaustive lists. But in general process measures can be effective in encouraging certain types of behaviors and facilitating certain steps or actions which can improve care delivery and the patient’s experience. However, directly linking improvements in process to the subsequent health and outcomes of patients may be challenging.
Ultimately, some would argue, the purpose of measuring aspects of healthcare delivery is to improve patient outcomes, whether it means reducing utilization (hospitalization, readmissions, ED visits) or adverse events (infections, complications, death). So, when it comes to promoting the use of outcome measures, proponents usually note the following:
- Outcomes are “what matters” to patients and represent the most important aspects of care, namely the resulting health of those treated;
- These measures often reflect the biggest financial implications and therefore allow for a more straightforward calculation of saved costs and increased value.
However, just as with process measures, outcome measures have potential limitations that should be noted:
- Some outcomes are likely a function of several aspects of care and it can be difficult to attribute the outcome to a particular provider or facility (e.g., the likelihood of 30-day readmission is not just a function of the discharge hospital’s quality, but also patient behavior including medication adherence, comorbid conditions including frailty and cognitive impairment, and social determinants of health);
- Using patient outcomes as a measure of quality can sometimes foster a punitive culture within a facility;
- Attempts to avoid adverse events can potentially change provider behavior for the worse (e.g., providers who are hesitant to insert catheters because they don’t want to be at risk for a CLABSI, or intentionally avoiding certain high-risk patients, etc.)
- These types of measures are more likely to require risk-adjustment to (try to) account for differences between facility characteristics, geography, and patient populations.
Not all of these are applicable in all situations - but these are the kinds of potential drawbacks referenced when there are concerns regarding some outcome measures. Just as was the case with process measures, the limitations of outcome measures do not mean that these types of measures are not or cannot be effective for tracking and comparing quality. They are simply things to be considered when selecting appropriate measures and interpreting results.
So, which is better to measure, a process or an outcome? It depends; each type of measure has its strengths and weaknesses, and different situations, goals, resources, and time-frames call for different measures - or sometimes even a combination of the two. Be careful when someone universally praises one type over the other, because they each have their place if they are appropriately defined, created, implemented, and interpreted. Also be aware of the strengths and weaknesses of each as you determine what is best for your organization and situation. Consider how a chosen measure will allow you to provide continuous feedback to your team and facility, so that you can track progress, celebrate successes, and quickly address areas of need. Understanding how each type of measure may facilitate change or encourage innovation can help you design and implement the most effective intervention possible.
The Role of Risk-Adjustment
When using outcome measures (like mortality or readmission rates) to compare quality between providers or facilities, the notion of risk-adjustment is usually part of the conversation: should it be done, how best to do it, how well it performs, etc. There is a lot of literature out there devoted to the methods and effectiveness of risk-adjustment, and for good reason: there is enormous variation in provider and facility characteristics (size, capabilities, available resources, staffing, location, case-mix, etc.), and it is well-known that some factors can significantly impact access to care, timing of care, and ultimately patient outcomes. This variety and these differences should be acknowledged, studied, understood, and when possible, used to adjust measures or stratify those being measured. When done well, risk-adjustment methods can do an excellent job of mitigating the impact of uncontrollable factors and make comparisons far more fair. However, there are a couple of things to keep in mind when it comes to risk-adjustment:
- Only factors that are known, measurable, and for which there are data can be included in risk-adjustment, meaning that unknown or unmeasurable factors cannot be accounted for. For example, patient behaviors (such as diet, medication adherence, etc.) can greatly affect outcomes but are typically not measurable.
- Simply including the factor when developing the risk-adjustment model doesn't necessarily mean that its impact just disappears. Models are approximations that are useful tools, but their functional ability is dependent on the data and factors included (and those omitted), and can potentially be influenced by things like co-linearity, selection bias, and how well surrogate factors reflect the underlying concept they are trying to capture (e.g., median income of zip code as a surrogate for SES).
This doesn’t invalidate risk-adjustment as a meaningful tool to aid in making meaningful comparisons of quality, but it is certainly not a panacea, especially in a complex system like healthcare. Additionally, we are continuously learning about new things that can impact the quality of care delivered and patient outcomes; there are studies that claim that stroke outcomes can be impacted by EMS care before the patient even reaches the hospital, and the quality of SNFs near a hospital may influence the likelihood of hospital readmission. This underscores the notion that there are very likely unknown factors that could be influencing patient outcomes, and no amount of risk-adjustment can completely level the playing field. That is why as users and interpreters of measures (unadjusted and risk-adjusted alike) we need to understand the background and the factors at work, and make judgments that take into consideration more than just the single piece of information of a risk-adjusted rate. Gaining an understanding of risk adjustment methods - as well as basic concepts of measure development - are crucial for all of us working to improve care and outcomes of patients.
Those of us who work in quality improvement understand these concepts, including risk-adjustment, to some degree. Developing a deeper understanding of the tools commonly used to identify, develop, and adjust measures can only improve our collective ability to assess and evaluate the quality of care delivery. By seeking ways to further our aptitude and proficiency with data-related aspects of quality measurement through continuous learning and frequent discussions, we will improve improving patients’ health and experience of care.