MGMT 8525M Week 3 Measurement Critique Example

The Churned Flag Counts Unpaid Accounts, Not Customers Who Chose to Leave: A Measurement Critique of One Telecom Field, the Records Its Rule Gets Wrong and Why the Error Is Larger in Some Regions Than Others

Student Name

Walden University

MGMT 8525M: Big Data Decision Making and Management

Instructor Name

Month Day, Year

The Churned Flag Counts Unpaid Accounts, Not Customers Who Chose to Leave: A Measurement Critique of One Telecom Field, the Records Its Rule Gets Wrong and Why the Error Is Larger in Some Regions Than Others

The Rule Beside the Label

In the customer file of Teton Valley Wireless, a composite regional mobile carrier with about 1.2 million accounts in five states, the field churned equals 1 when an account goes sixty days without a payment and is then closed. The retention team reads the field as customers who chose to leave, and it trains models on it to predict which customers will defect to a competitor so that it can send them retention offers. The rule and the label are not the same thing. The rule counts accounts that stopped paying. The label promises customers who decided to go.

Records the Rule Gets Wrong

Reviewing a random sample of 600 closed accounts from 2025 against the billing notes, collections records and fraud team logs, this critique sorted the closures by what actually happened. About 52 percent were customers who ported their number to another carrier, the event the retention team means by churn. About 21 percent were customers in billing disputes who stopped paying while contesting charges. About 17 percent showed signs of financial hardship, such as a payment plan request or a series of partial payments before the account lapsed. About 7 percent were closed by the fraud team for subscription fraud, where the account was opened with stolen identity details and never intended to be paid. The remaining 3 percent could not be classified.

So nearly half of the accounts the field calls churned did not leave in the sense the retention team intends. A model trained on the field learns to predict nonpayment, which overlaps with defection only partly.

Construct Validity as the Frame

Cronbach and Meehl (1955) introduced construct validity to describe the question of whether a measure captures the theoretical attribute it is meant to represent, as opposed to whether it predicts some criterion. The churned field has a construct, voluntary defection, and an operationalization, sixty days unpaid. Jacobs and Wallach (2021) argued that many problems described as unfairness in computational systems are failures of measurement: a construct is operationalized through a proxy, and the proxy's mismatch with the construct, and with different groups, carries into every decision built on it. The churned field is a clear example.

Uneven Error Across Groups

The mix of closure types differs by region. In the carrier's two metropolitan markets, number porting accounts for about 64 percent of closures. In its three rural regions, porting accounts for about 38 percent, and hardship accounts for about 29 percent, nearly twice the metropolitan share. Rural customers have fewer competing carriers to port to, and incomes in those regions are lower. A churn model trained on the field will therefore treat rural hardship as if it were rural defection, and the retention team will send discount offers designed to beat competitors' prices to customers whose problem is ability to pay.

A Published Mechanism, Used Carefully

Obermeyer et al. (2019) studied an algorithm widely used in American health systems that used health-care cost as a proxy for health need. Because spending on Black patients with equal illness ran lower, the tool read their need as smaller than it was, and fewer were referred to programs that would have helped. The parallel here is the mechanism, not the domain: a proxy that tracks its target unevenly across groups produces decisions that are wrong unevenly. Nothing in this critique suggests the telecom case involves race or carries harms comparable to those in health care.

What the Field Counts and What It Is Taken to Mean

[after]

The table summarizes the critique for the retention team, whose model decisions depend on the field.

Table 1

The Churned Field: Counts, Meaning and Evidence of the Gap

Keep, Correct or Replace

The critique recommends replacing the field for the retention model with a new field built from porting records, which directly capture defection to a competitor. The existing field should be kept and relabeled as unpaid closure, since collections and credit teams have legitimate uses for it. Hardship cases should be routed to a payment assistance team rather than to retention offers.

References

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302. https://doi.org/10.1037/h0040957

Jacobs, A. Z., & Wallach, H. (2021). Measurement and fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 375-385). Association for Computing Machinery. https://doi.org/10.1145/3442188.3445901

Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453. https://doi.org/10.1126/science.aax2342