MGMT 8825M Week 1 Discussion Post Example

Week 1 Discussion: Who Answers for the Machine's Decision?

A Selfie Rejected, an Account Closed and No One to Ask: A Discussion Post Mapping the Many Hands Behind an Automated Courier Deactivation and Arguing, With Nissenbaum Against Matthias, That the Responsibility Gap Here Was Built, Not Inevitable

The Case

Last spring, a courier for Parcelrun, an invented same-day courier platform active in eleven US cities, opened the app before a Saturday shift and was asked for a selfie. The platform's automated identity check compared the photo with the one on file from her sign-up two years earlier and flagged a mismatch. Her account was deactivated within the minute. Her appeal, submitted through the app, was answered by a second automated screen asking her to resubmit the photo, which was again rejected. She had recently cut her hair short and started wearing glasses. It took eleven days, and a complaint to a city council member, before a person looked at her case and reinstated her.

The Hands Involved

At least four parties shaped that outcome. A vendor trained the face-matching model and sold it with a recommended similarity threshold. Parcelrun's trust and safety team set its own, stricter threshold to reduce account sharing. A product manager approved automatic deactivation on a single failed match. And an operations director removed the step in which a human reviewed flagged accounts, after calculating that review cost about $4 per case across roughly 30,000 flags a month.

Two Readings

Matthias (2004) offers one reading. He argued that learning systems create a responsibility gap: because their behavior emerges from training rather than from explicit rules, neither the maker nor the operator can predict or control particular outcomes, and so neither can fairly be blamed for them. On that view, no one at Parcelrun or the vendor did anything wrong in the ordinary sense; the system did something no one chose.

Nissenbaum (1996) offers the other. She identified many hands as one of several barriers to accountability in computing, along with treating bugs as inevitable, blaming the computer and claiming ownership without liability. She treated these as barriers that deliberate practice can overcome, not as facts of nature.

Which Reading Fits

I adopt Nissenbaum's reading for this case. The match model may have been opaque, but every decision that turned its output into a deactivation was made by people who could explain it: the threshold, the single-strike rule and the removal of review. The gap Matthias describes exists, but here it was widened deliberately by taking a human out of the appeal path. Danaher (2016) argued that automation can produce gaps in our ability to assign retributive blame even when people bear some responsibility; that concern applies more to the vendor than to Parcelrun.

What the Choice Costs

Accepting responsibility has a price. Restoring human review of flagged accounts would cost Parcelrun about $120,000 a month at current volumes, or a little under $1.5 million a year. A cheaper middle path, human review only of appeals, would cost perhaps a tenth of that. Either is a choice Parcelrun can make. Calling the deactivation the machine's doing would be cheaper still, but only because the cost would stay with the couriers.

References

Danaher, J. (2016). Robots, law and the retribution gap. Ethics and Information Technology, 18(4), 299-309. https://doi.org/10.1007/s10676-016-9403-3

Matthias, A. (2004). The responsibility gap: Ascribing responsibility for the actions of learning automata. Ethics and Information Technology, 6(3), 175-183. https://doi.org/10.1007/s10676-004-3422-1

Nissenbaum, H. (1996). Accountability in a computerized society. Science and Engineering Ethics, 2(1), 25-42. https://doi.org/10.1007/BF02639315