Keep an interpretable baseline for a randomized retention pilot. A 0.009 AUC improvement is not proof of higher saved contribution, and predictive ranking does not establish treatment uplift.
01 / THE MANAGEMENT PROBLEM
The wrong question was “which model scores higher?”
At fictional Contoso Play, a retention team wants to know whom to contact. The model team offers a logistic baseline and a gradient-boosting alternative. A buyer should care whether a contact saves more contribution than it costs—not whether the slide contains a more fashionable algorithm.
The management decision: which customers should receive which treatment, at what capacity and cost, based on demonstrable incremental retention?
Retention / lifecycle marketing (proposed).
Model validation, data privacy and customer-treatment controls.
3,000 held-out fictional subscribers; synthetic churn outcomes.
No controlled treatment uplift, incremental margin or contactability measurement.
02 / THE DIAGNOSIS
The uplift in predictive ranking is only 0.009 AUC.
The gradient-boosting model scores 0.797 AUC, compared with 0.788 for logistic regression. That 0.009 gain is less than one AUC percentage point. Without paired uncertainty estimates, the case cannot assert that this difference is statistically robust.
Exhibit 1 · Discrimination and concentration
SYNTHETIC HOLDOUT| Evidence | Observation | Implication |
|---|---|---|
| Model discrimination | 0.788 vs 0.797 AUC | Small offline performance difference |
| Top 10% of customers | 35.48% of churners; 3.55× lift | Ranking supports prioritization |
| Top 30% of customers | 68.2% of churners | Capacity constraint materially changes selection |
| Contact treatment outcome | Not present | No evidence of causal uplift or ROI |
| Calibration / drift | Not established in this case | Do not equate score with probability |
Crucial distinction: predicting who will churn is not predicting who can be retained. Some high-risk customers will leave regardless of the offer; some lower-risk customers may be especially persuadable.
03 / THE OPTIONS
Keep the simplest adequate model until value is proved.
| Option | Trade-off | Recommendation |
|---|---|---|
| Deploy the boosting model immediately | Marginal AUC gain; added governance cost; no uplift proof | No |
| Start with logistic ranking and controlled treatments | Interpretable; easier operational baseline | Yes, for pilot |
| Send an offer to all flagged users | High cost, incentive leakage, contact fatigue | Reject |
| Do nothing | Avoids costs but forfeits learning | Use as randomized control, not a strategy for everyone |
04 / THE INTERVENTION
Make treatment economics the unit of evaluation.
- 01 · SELECTDefine eligibility, contact capacity and channel permissions. Score eligible customers, then deduplicate and suppress ineligible contacts.
- 02 · RANDOMIZERandomly assign comparable customers to treatment and holdout. Keep the allocation rule and cohort fixed before launch.
- 03 · TREATProvide a documented offer through an approved channel; capture delivery, cost, acceptance, redemption and adverse experiences.
- 04 · MEASUREUse a predeclared retention window; compare treatment and holdout outcomes and contribution after incentive and servicing cost.
- 05 · ESCALATEMonitor complaints, fairness, consent, contact frequency, calibration and drift; let accountable humans pause campaigns.
05 / THE VALUE BRIDGE
Try the economics. Every assumption is editable.
This calculator uses the synthetic decile churn rates (the top three average 32.89%), extending into lower-ranked deciles when more people are selected. It is a scenario calculator, not a model result. The “prevented share” is an assumed treatment effect among otherwise-churning people reached, not an observed retention lift.
Exhibit 2 · Hypothetical campaign contribution
SCENARIO · NOT OBSERVEDBreak-even effect = campaign contact cost / (number reached × underlying risk × net contribution per saved customer). For scale approval, replace scenario assumptions with a randomized pilot estimate and confidence interval.
06 / DELIVERY PLAN
Six weeks to a falsifiable retention pilot.
- WEEK 01Business owner agrees retention definition, eligible population, channel budget, contribution assumptions and guardrails; model team verifies feature leakage.
- WEEK 02Risk and analytics validate holdout sampling, churn model stability, targeting fairness, contact permissions and treatment scripts.
- WEEKS 03–04Campaign operations runs the randomization, contact suppression, delivery and exception logging; independent monitoring tracks adverse outcomes.
- WEEKS 05–06Evaluate as much of the predeclared retention window as has elapsed. Do not call the pilot successful prematurely; continue measurement where required.
Kill / revise rule: stop scale-up if treatment-adjusted incremental contribution is nonpositive, complaints breach approved tolerances or confidence is insufficient to justify deployment. A positive AUC gap cannot override a failed business experiment.
07 / EVIDENCE LEDGER
Prediction is demonstrated. Retention impact is not.
| Claim | Classification |
|---|---|
| AUC and decile capture | Observed in synthetic holdout |
| Boosting model produces more net retention value | Unproved |
| Calculator contribution | Illustrative assumptions only |
| Randomized pilot with a simple model | Recommended, not executed |
Decision: do not buy model complexity until it clears the financial, statistical and operating thresholds. The first valuable capability is often a sound experiment.
Underlying synthetic churn report (PDF) ↗ · Original charts ↗