DDBA 8524 Week 3 Instrument Critique Example

An Appraisal Form Is a Theory of Work: Critiquing the Performance Review Used Across a Multinational Logistics Firm's Units, Naming the Assumptions in Its Items and Asking Whether Scores Can Be Compared Across Settings

Student Name

Walden University

DDBA 8524: Seminar in Multicultural Management

Instructor Name

Month Day, Year

An Appraisal Form Is a Theory of Work: Critiquing the Performance Review Used Across a Multinational Logistics Firm's Units, Naming the Assumptions in Its Items and Asking Whether Scores Can Be Compared Across Settings

The Instrument

A composite logistics firm headquartered in Chicago operates warehouses and freight offices in the United States, Brazil, Poland and Vietnam, with about 6,500 employees. Since 2023 it has used a single annual performance review form for all salaried staff. The form rates employees on eight items on a five-point scale, and the ratings feed a global talent calibration in which managers from all four countries compare their people. Among the items are: "Speaks up in meetings and challenges ideas, including those of senior leaders"; "Takes initiative without waiting for direction"; "Sets ambitious personal goals and tracks progress against them"; and "Builds a personal network across the company." Headquarters describes the form as neutral and objective.

The Theory of Work in the Items

Read as claims about what a good employee does, the items describe a particular theory of work. They value individual voice, self-direction, personal ambition and individually built relationships. They say little about loyalty to a team, care in following procedures, harmony with colleagues or deference to experience, which may be valued highly in some units. Hofstede (1980) argued that American management theories embed assumptions about individualism and low power distance that are not shared everywhere, and the form's items look like such theories turned into measurement. An employee who rarely challenges a senior leader in a meeting, but raises concerns privately afterward, scores low on the first item even if the concern is heard and acted on.

Can Scores Be Compared Across Units?

The calibration process assumes that a score of four on "speaks up in meetings" means the same thing in Chicago and in Ho Chi Minh City. Vandenberg and Lance (2000) reviewed the methods for testing measurement invariance, that is, whether a scale captures an identical concept, scored identically, in each group it is given to, and emphasized that comparing group scores is meaningful only if invariance holds. Without such testing, a lower average in one country could reflect a real difference in behavior, a difference in how managers interpret the item or a difference in how the behavior shows up locally. Tsui et al. (2007) noted that cross-national organizational research often assumes equivalence of measures rather than testing it.

The firm's own data hint at a problem. Average scores on the "speaks up" item are 3.9 in the United States and Poland, 3.5 in Brazil and 2.8 in Vietnam, yet the Vietnam unit has the firm's lowest error rates and highest customer satisfaction. Either the Vietnamese staff are poorer performers on this dimension who somehow produce better results, or the item does not capture how good performance shows itself there.

What This Means for the Calibration

If the items do not measure the same thing across units, the global calibration ranks employees partly by how closely their local workplace culture matches the form's assumptions. That could steer promotions and development spending toward units whose norms resemble headquarters'. Hofstede (1980) warned that exporting management practices without examining their assumptions could produce exactly this kind of mismatch.

What the Firm Could Do

Three steps would help. Test the form for measurement invariance across units, using the firm's existing ratings, before using scores in global calibration. Add items that capture contributions the current form misses, such as raising concerns through appropriate channels or strengthening team performance, written with input from each unit. And until invariance is established, calibrate within units rather than across them.

A Possible Objection

Headquarters might answer that a global firm needs one standard, and that the form describes the behavior it wants everywhere, whatever local habits are. That answer is coherent if the firm has decided that speaking up in meetings is a required behavior for every salaried job, in which case the form should say so and the firm should train staff in it rather than mark them down for not already doing it. But the firm has not argued that the behavior produces results. The Vietnam unit's error rates and customer scores suggest that good performance can show itself in other ways, and a form meant to identify good performers should capture those ways too. A standard that ranks a strong unit last is measuring conformity to a style, not contribution to the firm.

A Small Test

Before redesigning the form, the firm could pilot two added items in one quarter in all four countries and compare their link to error rates and customer scores with the current items' link.

Limits

The critique relies on the form's wording, average scores and the author's interviews with four managers; it has not tested invariance statistically.

References

Hofstede, G. (1980). Motivation, leadership, and organization: Do American theories apply abroad? Organizational Dynamics, 9(1), 42-63. https://doi.org/10.1016/0090-2616(80)90013-3

Tsui, A. S., Nifadkar, S. S., & Ou, A. Y. (2007). Cross-national, cross-cultural organizational behavior research: Advances, gaps, and recommendations. Journal of Management, 33(3), 426-478. https://doi.org/10.1177/0149206307300818

Vandenberg, R. J., & Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4-70. https://doi.org/10.1177/109442810031002