DBAX 9102 Week 10 Design Peer Challenge Example

Week 10 Discussion: Peer Challenge of a Stage 2 Design

A Design That Invites Someone to Break It: Laying Out a Coaching and Quota Study for Classmates to Attack, With the Three Weakest Points Named in Advance and Two Challenges Offered in Return

My Design in Brief

My study asks whether coaching by sales managers relates to quota attainment among outside salespeople at an industrial supply distributor where a policy sets a target of two coaching sessions a month. The design is multilevel and correlational: sixty-four field sellers grouped under eleven managers, with coaching frequency taken from logs, coaching skill from a salesperson survey based on Heslin et al. (2006), and quota attainment from four quarters of sales records, controlling for territory potential and tenure.

Where I Expect It to Break

I see three weak points and would like you to push on them or find others. First, reverse causation: a manager who spends extra sessions on a weak seller could make coaching look harmful when it is actually helping. My answer is a one-quarter lag between coaching and results, plus a control for earlier results, but that may not be enough. Second, the frequency measure may be hollow if managers log brief check-ins to meet the target. Third, eleven managers is a small number for a multilevel design, which limits what I can say about differences between managers.

What I Hope You Find

The most useful challenge would be one that shows a result I could not interpret. For example, if frequency and attainment turn out unrelated, is that because coaching does not matter, because the logs are hollow or because struggling salespeople get more coaching? If I cannot tell those three apart, I need another measure, and I would rather learn that now than after data collection.

Question for Classmates

Which of these would you attack first, and is there a weakness I have not named?

Challenge to Priya

Priya, your quasi-experiment compares two clinics before and after one adopted a new scheduling system. My challenge is history: the adopting clinic also hired a new office manager that year, as your context section mentions. Shadish et al. (2002) describe this as a history threat, since any concurrent change could explain a difference. Could you add a second comparison clinic, or measure an outcome the office manager would affect but the scheduling system would not, to separate the two?

Challenge to Marcus

Marcus, your survey links employees' trust in leadership to their intention to stay. My challenge is that both are self-reported by the same employees at the same time. Podsakoff et al. (2003) show how common method variance can inflate exactly that kind of relationship. Could you use actual turnover from human resources records six months later as a second outcome?

What I Will Change After This Thread

I will read your challenges with one rule: if a challenge points to a weakness I cannot answer with the current design, I will either add a control, add a measure or state it in the limitations, and I will record which. A design that changes nothing after a challenge thread has not really been challenged.

References

Heslin, P. A., Vandewalle, D., & Latham, G. P. (2006). Keen to help? Managers' implicit person theories and their subsequent employee coaching. Personnel Psychology, 59(4), 871-902. https://doi.org/10.1111/j.1744-6570.2006.00057.x

Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879-903. https://doi.org/10.1037/0021-9010.88.5.879

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.