AI fired a San Francisco cashier and California lawmakers noticed

The story broke last week: a national retail chain let an algorithm terminate a part-time employee at a Market Street location after the system flagged "productivity anomalies" across three consecutive shifts. No human manager reviewed the decision before the termination notice hit the worker's phone. The employee — six months on the job, no prior write-ups — found out via push notification at 6:47 a.m. on a Tuesday.

California's Labor Commissioner opened an inquiry within 48 hours. Assembly Bill 2930, already in committee, suddenly picked up three new co-sponsors. The bill would require any employer using automated decision systems for hiring, firing, scheduling, or discipline to disclose the logic, allow human appeal, and audit for disparate impact quarterly. Violations carry a $5,000-per-incident penalty.

Here's what makes this technically significant: the system wasn't some black-box LLM. It was a gradient-boosted ensemble trained on badge-in/badge-out timestamps, POS transaction rates, customer satisfaction scores, and inventory shrink deltas. Feature importance weights showed "transactions per labor hour" at 0.41, "register void rate" at 0.23, and "shift adherence variance" at 0.18. The model retrained weekly on rolling 90-day windows. When the employee's void rate spiked 12% over two weeks — later traced to a faulty barcode scanner on register 3 — the SHAP values pushed their risk score past the 0.87 termination threshold automatically.

The vendor's documentation explicitly warns: "Model outputs are advisory; final employment actions require human review." That clause sits in Section 4.2 of the SLA. The retailer's implementation skipped it.

Three technical takeaways worth your attention:

• Drift detection failed silently. The scanner malfunction created covariate shift in the void-rate feature. No monitoring alert fired because the feature distribution stayed within historical bounds — just the cause changed. This is classic label leakage in disguise: the model learned "high void rate = bad employee" but the causal pathway broke.

• No human-in-the-loop guardrail. The API response included {"action": "terminate", "confidence": 0.91, "review_required": false}. That review_required flag is configurable. Default is true. Someone flipped it.

• Explainability exists but wasn't surfaced. SHAP values for the termination decision were logged. The employee never saw them. The store manager never saw them. The district manager saw a PDF summary after the fact.

AB 2930's audit requirement would force quarterly SHAP reports by protected class. That's actually implementable — most ML platforms (Vertex, SageMaker, Databricks) already emit these artifacts. The hard part is organizational: who owns the review? Legal? HR? Engineering? Right now, nobody does.

The retailer's CTO put out a statement calling it "an isolated configuration error." Sure. But the architecture enabled the error. When you deploy a model that can fire people without a human signature, you've built a weapon. Configuration errors happen. The architecture chose to make them irreversible.

Expect more of this. Not just retail — logistics, call centers, gig platforms. Anywhere labor is instrumented, models will optimize it. The regulatory response is inevitable. The technical response — rigorous drift monitoring, mandatory human gates, explainability surfaced to affected workers — is optional. For now.