MGMT 8705M Week 11 Final Evaluation Paper Example

Collaboration Ratings Fit for Development, Not for Pay: A Final Evaluation Paper Connecting a Construct, an Indicator, a Rating Design and a Predicted Response, and Arguing Where the Evaluation Stops Meaning What It Claims

Student Name

Walden University

MGMT 8705M: Organizational Behavior Performance and Evaluation

Instructor Name

Month Day, Year

Collaboration Ratings Fit for Development, Not for Pay: A Final Evaluation Paper Connecting a Construct, an Indicator, a Rating Design and a Predicted Response, and Arguing Where the Evaluation Stops Meaning What It Claims

The Claim

Peer and lead ratings of collaboration at Larchmont Software, gathered with behavioral frequency items and checked against an agreement threshold, can support quarterly development decisions but not pay decisions. The reason is that the response the ratings provoke, already present when they inform development, would grow once money depended on them, and the evaluation's meaning would erode as it grew.

The Construct, Carried Forward

Earlier in the term, collaboration was defined behaviorally, as work on a common task in which people alter one another's output, with the evidence lying in documents, decisions and code that bear several contributors' marks. Instructor comments led to one revision: code review comments that are offered but not incorporated now count as an attempted contribution, recorded separately, rather than being left undecided. The revision matters because, if ratings become consequential, people may offer comments chiefly to be seen offering them.

The Indicator and the Design

The indicator is a behavioral observation scale of eight frequency items, following the approach Latham and Wexley (1977) developed, rated by two peers who shared at least two tasks with the person and by the team lead. Self-ratings inform the development conversation but stay out of the score, because self-ratings agree with others far less than peers and supervisors agree with each other (Harris & Schaubroeck, 1988). Raters complete frame-of-reference training, which Woehr and Huffcutt (1994) found produced the largest gains in rating accuracy among the training approaches they compared, and scores are released only when raters' agreement on the total reaches a set threshold.

The Rater as Part of the Measure

Even a well-designed rating reflects the rater as well as the ratee. Scullen et al. (2000), analyzing ratings of more than 2,000 managers each rated by several bosses, peers and subordinates, found that idiosyncratic rater effects accounted for more than half of the variance in ratings, while the ratees' actual performance accounted for less than half as much. Training and multiple raters reduce this problem but do not eliminate it. Murphy and Cleveland (1995) added that raters pursue goals other than accuracy, such as maintaining working relationships, avoiding conflict or motivating a colleague, and that these goals shape ratings systematically. Peer raters on a small product team have strong reasons to rate one another generously, because they will work together next quarter.

What Ratees Will Do Next

When collaboration ratings inform only development, the predicted response is modest: some people will make their contributions more visible, for example by commenting on more drafts. That behavior is partly the intended effect, since visible contribution is part of the construct, and partly noise. Holmstrom and Milgrom (1991) showed that strong incentives on measured tasks draw effort away from unmeasured ones. If pay depended on collaboration ratings, the incentive to be seen collaborating would rise sharply, and effort would shift from deep solo work, which the scale does not measure but the product needs, toward visible but shallow contributions. Raters, knowing that their ratings affect a colleague's pay, would face stronger pressure to be lenient, increasing the rater effects Scullen and colleagues measured.

Why the Evaluation Means Something for Development

For development, the ratings retain their meaning. The stakes are low, the conversation focuses on specific behaviors and items rather than a total, and errors can be corrected next quarter. The agreement check ensures that scores are shared only when raters see the same behavior. The response the ratings provoke is small and partly desirable.

Why It Stops Meaning It for Pay

For pay, the same ratings would be asked to carry a decision that is hard to reverse, while the conditions that support their meaning, low stakes and candid raters, would disappear. The evaluation would increasingly measure skill at appearing to collaborate and raters' willingness to be generous.

Verbs the Evidence Allows

The paper's claims are bounded accordingly. The design is likely to produce ratings that reflect collaboration as defined, under developmental use. The research predicts, but this paper has not observed, that pay-linked use would degrade them. A pilot that linked a small bonus to ratings in two teams, with independent measures of collaboration from the code repository, would test the prediction.

Limitations

Larchmont is a composite, and the agreement and rater-effect figures come from other settings. The construct's restriction to recorded work leaves out collaboration that leaves no trace.

References

Harris, M. M., & Schaubroeck, J. (1988). A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. Personnel Psychology, 41(1), 43-62. https://doi.org/10.1111/j.1744-6570.1988.tb00631.x

Holmstrom, B., & Milgrom, P. (1991). Multitask principal-agent analyses: Incentive contracts, asset ownership, and job design. Journal of Law, Economics, and Organization, 7, 24-52. https://doi.org/10.1093/jleo/7.special_issue.24

Latham, G. P., & Wexley, K. N. (1977). Behavioral observation scales for performance appraisal purposes. Personnel Psychology, 30(2), 255-268. https://doi.org/10.1111/j.1744-6570.1977.tb02092.x

Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Sage.

Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85(6), 956-970. https://doi.org/10.1037/0021-9010.85.6.956

Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67(3), 189-205. https://doi.org/10.1111/j.2044-8325.1994.tb00562.x