DDHA 8303 Week 6 Single-Site Claims Post Example

Week 6 Discussion: What a Single-Site Study Can Claim

A Mechanism Travels, a Percentage Does Not: What the Patent Office's Work-From-Anywhere Result Can and Cannot Tell a Health System About Its Remote Coders

A study at one site can tell other organizations which mechanism to look for, but it cannot tell them how large the effect will be in their own setting. The finding I will use to argue this comes from Choudhury et al. (2021), who studied patent examiners at the United States Patent and Trademark Office as they moved from working from home, tied to commuting distance from the office, to a program that let them live anywhere. Because the timing of each examiner's move depended largely on administrative and union scheduling, the authors could treat it as a natural experiment, and they found that output rose by about 4.4% after the move without a measurable loss in quality. For anyone studying remote coders, the result is tempting, because examining patents and coding charts are both rule-bound document review done alone.

Several features make that site particular. Patent examiners worked under a long-established production system with explicit output credits, so output was measured precisely and had been for years. Only examiners with enough tenure and a record of working from home were eligible, which means the people who moved had already adapted to remote work. The employer was a federal agency with a union contract, stable pay and an unusual degree of job security, so examiners who moved far away were not risking their jobs. And the change was from one remote arrangement to a freer one, not from the office to home. Each feature is a reason the 4.4% might not appear at Halvorsen Health, where coders went straight from the office to full remote work, under a productivity standard introduced only two years ago, in a labor market where coders change employers easily.

What does transfer is the mechanism. The authors linked the gain partly to examiners being able to choose where they lived, for example near family or in places with lower costs, which changed their circumstances outside work. Polit and Beck (2010) distinguish between statistical generalization, from a sample to a population, and the more modest step of judging whether a finding's conditions resemble one's own. On that view, a Halvorsen administrator can reasonably ask whether its coders gained similar flexibility and whether flexibility matters for their accuracy. The patent office result supports asking that question. It does not supply the answer, and quoting the percentage in a board presentation would invite a decision the study cannot support.

Reply to Lindiwe

Lindiwe, you argued that a single-site study can be generalized if the site is "typical," and you described your hospital as a typical 300-bed community hospital. I would test that boundary. Typical in size and ownership is not the same as typical in the conditions that produced the result. If your falls study succeeded partly because a particularly committed nurse manager led it, as you mentioned, then the condition that matters is the leader, not the bed count. Polit and Beck (2010) would suggest describing those conditions so that readers can judge similarity for themselves. Would you be willing to claim the result only for units with a manager in that role?

Reply to Ewan

Ewan, you went further than I did and said single-site findings should not inform decisions elsewhere at all. I think that sets the bar too high. Bloom et al. (2015) ran their experiment at one firm, and their mechanism, that much of the gain came from more working time, has been useful well beyond it. A single site cannot tell you the size of an effect, but it can tell you where to look, and that is worth having when the alternative is guessing. At what point, in your view, does a finding stop informing and start deciding?

References

Bloom, N., Liang, J., Roberts, J., & Ying, Z. J. (2015). Does working from home work? Evidence from a Chinese experiment. The Quarterly Journal of Economics, 130(1), 165-218. https://doi.org/10.1093/qje/qju032

Choudhury, P., Foroughi, C., & Larson, B. (2021). Work-from-anywhere: The productivity effects of geographic flexibility. Strategic Management Journal, 42(4), 655-683. https://doi.org/10.1002/smj.3251

Polit, D. F., & Beck, C. T. (2010). Generalization in quantitative and qualitative research: Myths and strategies. International Journal of Nursing Studies, 47(11), 1451-1458. https://doi.org/10.1016/j.ijnurstu.2010.06.004