DDHA 8601 Week 10 Benefits Measurement Example

Did the Promise Arrive? Measuring Routine In-Clinic Device Checks One Year After a Device Monitoring Platform Was Bought, and Admitting What the Data Cannot Say

Student Name

Walden University

DDHA 8601: Technology and Innovation in Healthcare

Instructor Name

Month Day, Year

Did the Promise Arrive? Measuring Routine In-Clinic Device Checks One Year After a Device Monitoring Platform Was Bought, and Admitting What the Data Cannot Say

The Claim Being Measured

When Linden Ford Community Hospital's leaders approved the vendor-neutral device monitoring platform for the cardiac device clinic, the business case stated its expected benefit in these words: "Within twelve months of go-live, the platform will reduce routine in-clinic device checks by 30 percent and cut nurse review time per transmission by half." This paper takes that sentence as the claim, because a benefit cannot be measured until someone has said exactly what it is. The hospital and its figures are composite.

The claim contains two promises. This paper measures the first. The second, nurse review time per transmission, cannot be measured fairly, for a reason explained below.

The Measure

The measure is routine in-clinic device checks per 100 actively followed patients per quarter.

The numerator counts in-person device interrogations in the clinic that were scheduled as routine follow-up. It excludes checks within 90 days after an implant, which are required regardless of monitoring; checks prompted by symptoms or by a remote alert, which remote monitoring is meant to trigger, not prevent; and checks done before a surgical procedure at another department's request.

The denominator counts patients with an active device enrolled in the clinic's follow-up on the middle day of the quarter, including those not enrolled in remote monitoring.

The comparison is between the four quarters before go-live and the four quarters after it, with the first quarter after go-live reported separately because the clinic was still adjusting.

The definition matters because a loose version would mislead. Counting all in-clinic checks would mix routine follow-up, which remote monitoring can replace, with alert-driven checks, which it should increase. Berwick et al. (2003) argue that measurement intended to support improvement must be specific to the process being changed, and this definition isolates the visits the platform was supposed to reduce.

The Baseline

Before the platform, the clinic already used each manufacturer's remote monitoring, so routine in-clinic checks were already less frequent than in a clinic without it. The baseline comes from the clinic's scheduling system, where each in-person device appointment carries a visit reason. The device nurse manager checked a sample of 60 appointments against the visit notes and found the visit reason accurate in 56, which is acceptable for this purpose but not perfect. Without the platform, this baseline would likely have continued to drift down slowly as more patients enrolled in remote monitoring; that drift is one reason the comparison below cannot be read as a clean effect.

Two Threats to the Comparison

The first threat is a change in coding practice during the same period. Six months after go-live, following an internal billing audit, the clinic changed how it coded some in-person visits, recording a combined programming evaluation in place of a separate interrogation for some patients seen by electrophysiologists. Because the clinic's reports identify device checks partly from billing codes, some routine checks after the change may not have been counted as interrogations. The paper therefore counts checks from scheduling visit reasons, not from billing codes, but the change still affects any comparison with the hospital's financial reports.

The second threat is a change in the panel. During the year after go-live, the clinic's share of patients with implantable loop recorders rose, because the hospital began a stroke follow-up program. Loop recorders rarely need routine in-clinic checks. A panel with more of them will show fewer routine checks per 100 patients even if nothing else changes.

A smaller threat is that remote monitoring enrollment itself rose during the period. Varma et al. (2010) showed that remote monitoring reduces in-clinic evaluations, so a rise in enrollment, which the platform did not cause, would lower routine checks on its own.

What the Measure Showed

According to the clinic's own scheduling reports, routine in-clinic checks per 100 followed patients per quarter fell by roughly a fifth between the year before go-live and the last three quarters of the year after. The fall was largest among patients with pacemakers and smallest among those with defibrillators, whose routine checks were already infrequent. Patients followed for more than five years showed a larger fall than recent implants, perhaps because the clinic was readier to space out checks for stable, long-followed devices. The first quarter after go-live showed almost no change.

The reduction is real but smaller than the promised 30 percent, and part of it is explained by the threats above. Adjusting for the rise in loop recorders, which the nurse manager did by recalculating the measure without loop recorder patients, reduces the fall to roughly one seventh. The paper does not attempt to adjust for rising enrollment, because the platform may have contributed to it by making enrollment easier to manage.

What the Data Cannot Answer

The data cannot say how much of the reduction the platform caused, as opposed to remote monitoring itself, which the clinic already used and which the consensus statement on cardiac device monitoring recommends as standard care (Slotwiner et al., 2015). A comparison clinic without the platform would be needed for that, and Linden Ford has none.

The data also cannot answer the second promise. The business case promised to halve nurse review time per transmission, but no one recorded review time before go-live, and nurses began logging time only in the fifth month. Any comparison would set a measured present against a remembered past. The paper reports this as a measurement failure that occurred when the purchase was approved, not as evidence that the time saving did not happen.

What Should Happen Next

The hospital should report the routine check measure quarterly, with loop recorder patients shown separately, and should ask the vendor for comparison data from similar clinics before the subscription renews. For future purchases, the business case should define each promised benefit as a measure, name the person who will collect it with a baseline recorded before go-live, so that the question this paper asks can be answered in full.

References

Berwick, D. M., James, B., & Coye, M. J. (2003). Connections between quality measurement and improvement. Medical Care, 41(1 Suppl.), I30-I38. https://doi.org/10.1097/00005650-200301001-00004

Slotwiner, D., Varma, N., Akar, J. G., Annas, G., Beardsall, M., Fogel, R. I., Galizio, N. O., Glotzer, T. V., Leahy, R. A., Love, C. J., McLean, R. C., Mittal, S., Morichelli, L., Patton, K. K., Raitt, M. H., Ricci, R. P., Rickard, J., Schoenfeld, M. H., Serwer, G. A., . . . Yu, C.-M. (2015). HRS expert consensus statement on remote interrogation and monitoring for cardiovascular implantable electronic devices. Heart Rhythm, 12(7), e69-e100. https://doi.org/10.1016/j.hrthm.2015.05.008

Varma, N., Epstein, A. E., Irimpen, A., Schweikert, R., & Love, C. (2010). Efficacy and safety of automatic remote monitoring for implantable cardioverter-defibrillator follow-up: The Lumos-T Safely Reduces Routine Office Device Follow-up (TRUST) trial. Circulation, 122(4), 325-332. https://doi.org/10.1161/CIRCULATIONAHA.110.937409