DDHA 8303 Week 5 Design Justification Example

Why a Staggered Difference-in-Differences Design Beats a Snapshot and a Single Time Series for Studying Remote Coders' Accuracy

Student Name

Walden University

DDHA 8303: Research Methods for Healthcare Administrators

Instructor Name

Month Day, Year

Why a Staggered Difference-in-Differences Design Beats a Snapshot and a Single Time Series for Studying Remote Coders' Accuracy

The Question

The question this design must answer was set in Week 4 and keeps every element here: it concerns Halvorsen Health's inpatient coders, its outcome is the monthly rate at which their DRG matches the auditor's, its exposure is the shift to fully remote work in January or September 2024, and its comparison is with coders who kept working onsite or hybrid through those months. The question has three structural features that the design must respect. It asks about change over time, not about a difference at one moment. It compares coders who moved with coders who did not. And it involves two move dates, so some coders were exposed earlier than others.

The Chosen Design

The study will use a difference-in-differences design with staggered treatment timing, drawing on existing audit records. Halvorsen's internal audit team reviews a random sample of about ten inpatient accounts per coder each month and records whether the auditor's DRG agrees with the coder's. For each of the system's 64 inpatient coders, the study will collect the number of audited accounts and the number with DRG agreement for every month from January 2023 through December 2025, along with each coder's work arrangement in each month, tenure and credential. That gives twelve months before the first move and sixteen from the second move onward.

The analysis will compare how agreement rates changed for coders after they moved with how rates changed over the same months for coders who had not yet moved or never moved. Wing et al. (2018) describe this design as a way to estimate the effect of a policy when randomization is not possible, by using a comparison group to account for changes that would have happened anyway. Because Halvorsen's coders moved at two different times, coders in the September wave serve as a comparison for the January wave during the months between, and coders who stayed onsite serve as a comparison throughout.

First Rival: A Cross-Sectional Comparison

The simplest alternative is to compare the agreement rates of remote and onsite coders during a recent period, for example the first half of 2025. This design has real strengths. The data are ready, the analysis is easy to explain to leaders and the comparison is immediate: remote coders either agree with auditors more often or less often than onsite coders do.

What it would miss is the history of the two groups, since nobody drew lots to decide who went home. At Halvorsen, the January wave was offered first to coders with at least three years of tenure and an audit record above the department's target, so the remote group began with higher accuracy than the onsite group. A snapshot taken in 2025 that showed remote coders slightly ahead would therefore be consistent with remote work having lowered their accuracy from a higher starting point. A cross-sectional comparison cannot separate who moved from what moving did, and in this case the selection runs in a direction that would hide exactly the effect the director suspects.

Second Rival: A Single-Group Time Series

A stronger alternative follows only the coders who moved, tracking their agreement rates month by month before and after the move and looking for a jump or a bend in the series when the move happened. An interrupted time series of this kind uses each coder as their own comparison, which removes the selection problem that defeats the snapshot. It is also feasible with the same data.

What it would miss is everything else that changed at the same time. In October 2024, as every year, the ICD-10 code set was updated, and in the same year Halvorsen adopted new computer-assisted coding software across all five hospitals. Either change could alter agreement rates for every coder, remote or not. A single-group series would attribute such a system-wide shift to remote work, because it has no group that experienced the same months without moving. The comparison group in the chosen design is what absorbs those shared shocks.

A randomized design is not a realistic third option. The moves have already happened, and reassigning coders to the office at random to test the question in reverse would answer a different question at considerable cost to the people involved.

What the Chosen Design Protects and What It Leaves Exposed

The design protects two inferences that matter to the director. It guards against stable gaps between movers and stayers, such as experience or skill, because it compares changes rather than levels. And it protects against changes that affected all coders at once, such as the code set update and the new software, because both groups lived through them.

It leaves two inferences exposed. The first is the assumption that, without the move, agreement rates in the two groups would have moved in step. Nobody can prove that, but it can be examined: if the groups' rates were already diverging during 2023, the design's estimate would be suspect. The second exposure comes from the staggered timing itself. Goodman-Bacon (2021) showed that the standard regression used for difference-in-differences can give misleading estimates when treatment begins at different times and its effects change over time, partly because earlier-treated units end up serving as comparisons for later ones. The analysis will therefore use an estimator built for staggered timing, one that compares each wave only with coders not yet treated (Callaway & Sant'Anna, 2021).

A further exposure is attrition. If coders who struggled after moving were more likely to leave, the remaining remote coders would look more accurate than the group that moved. The design will report results for coders present throughout as well as for all coders.

Feasibility

Access depends on the audit team and the compliance office, both of which already use the audit data and have agreed in principle to provide a de-identified extract. The data cover all inpatient coders, so no recruitment is needed for this branch, and extraction should take about two weeks. The main time cost is in cleaning work arrangement histories, which are held in human resources records rather than in the audit system. No participant cooperation is required for the quantitative branch, which reduces both burden and the risk that coders act differently out of awareness of being observed.

Conclusion

A staggered difference-in-differences design fits the question's structure and uses records Halvorsen already keeps. It answers the question a snapshot would bias and a single series would confound, while its remaining weaknesses can be examined openly through pre-move trends, a timing-robust estimator and attrition checks.

References

Callaway, B., & Sant'Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200-230. https://doi.org/10.1016/j.jeconom.2020.12.001

Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254-277. https://doi.org/10.1016/j.jeconom.2021.03.014

Wing, C., Simon, K., & Bello-Gomez, R. A. (2018). Designing difference in difference studies: Best practices for public health policy research. Annual Review of Public Health, 39, 453-469. https://doi.org/10.1146/annurev-publhealth-040617-013507