MMHA 6520 Week 9 Analytics Program Evaluation Example

Fewer Emergency Visits After Enrollment, but Compared With What? Evaluating Riverline's High-Utilizer Outreach Program

Student Name

Walden University

MMHA 6520: Health Informatics and Population Health Analytics

Instructor Name

Month Day, Year

Fewer Emergency Visits After Enrollment, but Compared With What? Evaluating Riverline's High-Utilizer Outreach Program

The Program's Claim

Riverline, the invented Medicaid plan this course follows, runs an outreach program for members with frequent emergency department use. Members are enrolled when they reach four emergency visits within six months; a care coordinator then calls, helps them schedule primary care, reviews medications and connects them to community resources. The program's annual report states that enrolled members had substantially fewer emergency visits in the six months after enrollment than in the six months before. This evaluation examines whether that comparison shows what it appears to show, and proposes a design that could answer the question. The figures in the report are taken as given; the issue is what they mean.

Limitation 1: Regression to the Mean

Members were selected at the moment their emergency use was highest, when they crossed the four-visit threshold. Use that is unusually high at one point in time tends to be lower at the next measurement, even with no intervention, because some of the high use reflects temporary circumstances, such as an acute illness, a housing crisis or a lapse in medications, that resolve on their own. This statistical pattern is called regression to the mean (Barnett et al., 2005). A before-and-after comparison that selects members at a peak will always show a decline, and part or all of that decline would have happened anyway.

The concern is not hypothetical. When the Camden hotspotting program was finally tested by lottery, its patients' readmissions dropped steeply after enrollment, and so did the readmissions of patients the lottery had left with ordinary care, by about the same amount (Finkelstein et al., 2020). Before-and-after comparisons of that program had suggested large effects that the trial did not confirm. Riverline's program enrolls members by a similar peak-based rule, so its before-and-after decline cannot be attributed to the program without a comparison group.

Limitation 2: Attrition

Medicaid members frequently lose coverage, and the six months after enrollment are no exception (Sommers et al., 2016). Members who lose coverage disappear from the plan's claims and therefore from the after period. If those who leave differ from those who stay, for example if members with the most unstable housing are both the heaviest emergency users and the most likely to lose coverage, the after period contains a healthier, more stable group than the before period did. The comparison would then show a decline that reflects who remained rather than what the program did. The annual report does not state how many enrolled members were still covered six months later or whether the comparison was limited to them.

Limitation 3: Claims Lag

The report was prepared shortly after the end of the program year, using claims available at the time. Emergency department claims can take weeks or months to arrive and be finalized. Members enrolled late in the year had their after period measured before all of their claims were in, so their after-period visits are undercounted. This makes the program look more effective than it may be. The fix is simple, though it requires patience from leaders who want results quickly: wait until claims for the after period are substantially complete, typically at least three months after the period ends, or limit the analysis to members whose after period ended early enough.

What the Program Demonstrably Did

The limitations above concern the program's claimed effect on emergency use. They do not erase what the program clearly accomplished. Its records show that coordinators reached most enrolled members by phone, that a substantial share of reached members completed a primary care visit within 60 days, and that coordinators made referrals for food, housing and transportation needs that members described. These are outputs, not outcomes, but they are real and valuable. They also give the plan something it can build on: a working outreach process, trained coordinators and relationships with community partners, which the staggered design below can test properly. Members who were connected to primary care and community resources received help they might not otherwise have had. The program's achievements should be reported accurately, as reach and connection, rather than as a reduction in emergency use that the data cannot support.

A Staggered Design

A staggered rollout would allow the plan to estimate the program's effect without denying help to anyone indefinitely. Members who meet the enrollment threshold in a given month would be randomly assigned to start outreach either immediately or three months later. During the delay, the later group serves as a comparison for the earlier group: both were selected at the same peak, so both will regress toward the mean, and the difference between them reflects the program. After three months, the later group receives outreach. Designs that roll an intervention out to groups in sequence, with timing randomized, are well established for evaluating programs that an organization intends to offer to everyone eventually (Hemming et al., 2015).

The design has costs. A three-month delay means some members wait longer for help, which raises an ethical question the care management director and medical director should weigh. One option is to exempt members with very recent hospital admissions from randomization and offer them immediate outreach, accepting a smaller comparison. The analysis should also count all enrolled members, including those who lose coverage, using enrollment files to track who left, and should wait for claims to complete before measuring outcomes.

Recommendation

Riverline should revise the annual report to describe the program's reach and connections, and to state that the before-and-after decline in emergency visits cannot be attributed to the program. It should adopt the staggered design for the next program year, with the analysis plan written in advance, the comparison measured at three months and claims allowed to mature before results are reported. If the program does reduce emergency use, this design will show it convincingly; if it does not, the plan will learn that before expanding a program on the strength of a misleading comparison.

References

Barnett, A. G., van der Pols, J. C., & Dobson, A. J. (2005). Regression to the mean: What it is and how to deal with it. International Journal of Epidemiology, 34(1), 215-220. https://doi.org/10.1093/ije/dyh299

Finkelstein, A., Zhou, A., Taubman, S., & Doyle, J. (2020). Health care hotspotting: A randomized, controlled trial. New England Journal of Medicine, 382(2), 152-162. https://doi.org/10.1056/NEJMsa1906848

Hemming, K., Haines, T. P., Chilton, P. J., Girling, A. J., & Lilford, R. J. (2015). The stepped wedge cluster randomised trial: Rationale, design, analysis, and reporting. BMJ, 350, Article h391. https://doi.org/10.1136/bmj.h391

Sommers, B. D., Gourevitch, R., Maylone, B., Blendon, R. J., & Epstein, A. M. (2016). Insurance churning rates for low-income adults under health reform: Lower than expected but still harmful for many. Health Affairs, 35(10), 1816-1824. https://doi.org/10.1377/hlthaff.2016.0455