DBAX 9103 Week 5 Analysis Walkthrough Paper Example

The Forks the Finished Analysis Hides: An Analysis Walkthrough for a Caregiver Retention Study, Showing Each Point Where the Analysis Could Have Gone Another Way and What the Result Would Have Been

Student Name

Walden University

DBAX 9103: Business Research Project Stage 3

Instructor Name

Month Day, Year

The Forks the Finished Analysis Hides: An Analysis Walkthrough for a Caregiver Retention Study, Showing Each Point Where the Analysis Could Have Gone Another Way and What the Result Would Have Been

Why Show the Forks

A finished analysis seldom shows the points where it could have gone another way. The reader sees one model, one sample and one estimate, as if no other choice was available. Gelman and Loken (2014) call these hidden decision points a garden of forking paths and warn that results can depend on them even when no one is fishing for significance. Steegen et al. (2016) recommend showing how results change across reasonable alternatives. This walkthrough traces the main analysis of the home care retention data, stopping at each fork to show the road not taken.

The Main Analysis

The main analysis was a logistic regression predicting employment at day 90 from the number of supervisor check-ins in the first 14 days, completion of all onboarding modules within 14 days and their interaction, controlling for starting pay, prior caregiving experience and log travel miles. Its sample was the 531 caregivers who were still on staff two weeks after hire.

The Forks

Table 1

Analysis Forks and Their Consequences for the Check-In Estimate

On the path taken, the check-in estimate settled at 1.33, and its 95 percent interval spanned 1.10 to 1.61.

Reading the Forks

Across every reasonable path, the check-in estimate stayed above 1, ranging from 1.24 to 1.41 for the count. The conclusion that more check-ins go with higher retention does not depend on any single choice. Its size does: the 30-day window inflates it, and counting scheduling calls dilutes it. That pattern is what one would expect if the 30-day window partly measures tenure and if scheduling calls add noise.

The onboarding estimate behaved differently. It ranged from an odds ratio of 0.96 to 1.18 across the same forks, never with an interval excluding 1. The interaction estimate was unstable in sign, which reinforces the decision to report it as inconclusive.

Forks Not Taken and Not Tried

Some alternatives were not run because they could not be justified before seeing results, such as dropping caregivers with very long travel distances. Running them would only add paths, which Simmons et al. (2011) warn increases the chance of a spurious finding. They are listed in an appendix as forks considered and set aside, each with the reason it was not pursued and the date it was set aside.

The Supervisor Fork in Detail

Clustering deserves a closer look because it changes the interval rather than the estimate. The 531 caregivers were supervised by 22 field supervisors, and caregivers who share a supervisor share that supervisor's habits, including how often and how carefully check-ins are logged. Treating them as independent overstates how much information the data hold. With clustering, the interval for the check-in estimate widened from 1.15 to 1.54 unclustered to 1.10 to 1.61 clustered. The conclusion held, but the clustered interval is the honest one, and it is the one the results chapter reports.

What the Walkthrough Adds to the Report

The results chapter reports the main estimate. The walkthrough lets a reader see that the direction of the check-in association survives every reasonable alternative while its size depends on the window, and that the onboarding and interaction results are fragile. That is more useful than a single number presented as if no other number were possible.

How the Forks Were Chosen

The forks listed are those where I had a genuine choice before the analysis, recorded in the analysis plan. I did not search for forks after seeing results. That distinction, between forks planned in advance and forks discovered afterward, is what keeps a walkthrough from becoming a search for the most favorable path. A reader who doubts this can compare the forks listed here with the dated analysis plan held by my chair.

References

Gelman, A., & Loken, E. (2014). The statistical crisis in science. American Scientist, 102(6), 460-465. https://doi.org/10.1511/2014.111.460

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632

Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702-712. https://doi.org/10.1177/1745691616658637