DeliverooFirat Yaman6 min readintermediate
Roonomics: How Economics Helped Us to Build a Churn Prediction Model
Summary
Deliveroo data scientists rebuilt their restaurant churn model to prioritize precision over recall, using rolling‑window cross‑validation, SHAP interpretability, and a ranking‑based output. Offline tests show a >90% drop in flagged restaurants and an 8× precision gain, with an A/B experiment underway to measure impact on actual churn.
- Previous high‑recall model flooded the retention team with false positives; economics dictated a precision‑first approach.
- Implemented rolling‑window CV to avoid leakage in time‑series forecasting.
- Optimized for precision on a rare‑event (≈1% churn) problem, using appropriate scoring metrics.
- Added SHAP values for feature‑level interpretability without sacrificing accuracy.
Accurate churn prediction lets a limited retention team focus on restaurants where outreach yields a positive ROI, turning a costly blanket‑alert system into a targeted, economically justified intervention.
6/10
