DBAX 9103 Week 10 Reproducibility Peer Read Example

Week 10 Discussion: Peer Read for Reproducibility

Could a Stranger Rebuild This Result: Sharing a Methods and Results Excerpt From a Caregiver Retention Study for Classmates to Reconstruct, and Reading Two Classmates' Excerpts the Same Way

My Excerpt

Only a stranger can say whether a report could be rebuilt, so I am sharing an excerpt from my results chapter and asking you to try. The excerpt comes from my study of whether early supervisor contact keeps newly hired home care aides past their third month. The excerpt below is the paragraph that defines the sample and the first result.

"Of 612 caregivers hired over 24 months, 22 rehires and 14 caregivers without check-in logs were excluded, leaving 576. Because check-ins were counted over the first 14 days, the 45 caregivers who left before day 14 were also excluded, leaving 531, of whom 369 were employed at day 90. A check-in was any logged supervisor contact about the caregiver's work or wellbeing, excluding contacts solely about scheduling. In a logistic model adjusting for starting wage, previous caregiving work and log travel miles, with supervisor-clustered standard errors, more check-ins went with greater odds of remaining on staff at the 90-day mark (odds ratio 1.33, with a 95% interval running from 1.10 up to 1.61)."

What I Am Asking

If you had my raw files, could you produce that 1.33 from this paragraph alone? Where would you have to guess? Appelbaum et al. (2018) set reproducibility from the written report as the standard for methods reporting, and I suspect I have left gaps I can no longer see.

Where I Think the Gaps Are

I suspect two. First, "excluding contacts solely about scheduling" depends on reading note text, and I have not said how ambiguous notes were classified. Second, I have not said how retention was coded for caregivers who transferred to a sister company. Either gap could lead a careful reader to a slightly different number, and neither would be visible to them as a choice.

Reply to Fatima

Fatima, I tried to rebuild your average order value figure from your excerpt. I could not tell whether returned orders were removed before or after averaging. That changes the number, sometimes by several dollars. Could you add one sentence saying when returns were removed?

Reply to Jonah

Jonah, your excerpt reports a regression with "standard controls." I could not rebuild it without knowing which controls. Simmons et al. (2011) recommend listing every variable in a model, because unlisted choices are where flexibility hides. Could you name them in the text or a table note? Even a short list would let me try the rebuild again this week.

What Your Reads Changed

Classmates' reads of my excerpt found both gaps I suspected and one I did not: I never said whether the 14-day window began on the hire date or the first shift. It began on the hire date, and that sentence is now in the chapter. Cumming (2014) argues that transparent reporting is what allows results to accumulate across studies; a sentence that lets a stranger rebuild a number is a small part of that.

References

Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73(1), 3-25. https://doi.org/10.1037/amp0000191

Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7-29. https://doi.org/10.1177/0956797613504966

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632