Home / Case studies / CASE 02 · CUSTOMER ANALYTICS
CASE 02 · CUSTOMER ANALYTICS
A better churn model. But a better business outcome?
On 3,000 fictional subscribers, gradient boosting barely improves model ranking. The stronger business question is who to contact, at what cost and against which holdout.
Evidence status. All subscriber records and performance measures in this case are synthetic. AUC and lift are properties of the illustrative held-out dataset; no retention campaign or customer saving was deployed or measured.
context
The situation
A subscription business wants to reduce preventable churn. A data science team can improve a model’s headline discrimination, but the retention team has a limited outreach budget and a finite number of call-center or digital impressions.
The decision is not simply which algorithm wins. It is whether a model-led targeting policy creates incremental retained contribution after discount cost, contact cost, customer value, operational capacity and the effect of contacting customers who would have stayed anyway.
diagnosis
The diagnosis
Two synthetic models are evaluated on the same held-out sample: a logistic regression baseline and gradient boosting. Their AUC scores are 0.788 and 0.797. The difference is 0.009 AUC units (0.9 percentage points).
Without uncertainty intervals, calibration, stability and treatment economics, that gain alone does not justify a more complex production system. The apparently modest model difference is not the same as a statistical conclusion of equivalence.
intervention
The analytical intervention
Separate three decisions that dashboards often collapse into one.
- Prediction: rank customers by likelihood to leave; verify holdout performance and calibration by segment.
- Actionability: estimate which departures can actually be influenced and avoid harmful or wasteful contacts.
- Value: allocate outreach by expected incremental gross margin net of incentive and contact costs.
findings
What the sample actually shows
Targeting concentration looks more material than the marginal model upgrade: the top 10% of ranked subscribers contain 35.5% of churners and the top 30% contain 68.2%. That supports using a ranking to focus scarce resources, subject to operational and economic checks.
SHAP ranks viewing trends, time since last login, tenure and contract structure among important model features in this synthetic example. These are associations explaining the score; they are not proof of causal drivers or a license to treat sensitive customer characteristics as action levers.
A model that finds risk is not necessarily a system that prevents loss. The experiment must measure the counterfactual.
decision
The management recommendation
Recommended: do not approve a complex-model replacement on AUC alone. Keep the simpler model as an operating baseline while piloting a bounded contact strategy. Randomize eligible high-risk customers into treatment and no-contact control groups and measure holdout-adjusted net contribution by decile and treatment.
| Gate | Evidence needed |
|---|---|
| Predictive | Out-of-time AUC, calibration, stability, bias checks |
| Economic | Incremental retention × contribution minus incentives, contact and servicing costs |
| Operational | Reach rate, contact capacity, offer eligibility, exclusions |
| Decision | Scale only if net contribution clears a precommitted threshold |
These are proposed experiment gates, not completed pilot results.
limits
What this case does not prove
- AUC and lift do not equal causal retention impact.
- No real acquisition, retention, revenue or margin outcomes are measured.
- Feature importance cannot establish why an individual churned.
- Calibration, variance estimates, offer responsiveness and control-group economics remain untested in the published toy sample.
sources
Evidence and further reading
ReadSigil: synthetic model exploration · Original fictional model report (PDF) · Churn playground
As a separate published real-world example of scaling analytics in banking, see Google Cloud’s Monzo customer story. It documents the data platform, not a causal churn experiment.