Why this matters
Marketing changes (new creative, discounts, targeting, attribution settings) often shift retention, revenue, and payback. Cohort comparison lets you see if the change actually improved customer value or just shifted timing.
- Real task: Compare pre vs. post-change cohorts to decide whether to scale a campaign.
- Real task: Check if a promotion pulled revenue forward but harmed margin and long-term LTV.
- Real task: Estimate payback period and breakeven point after targeting changes.
- Real task: Isolate the effect of a change while controlling for seasonality and channel mix.
Progress & saving notes
The quick test and exercises are available to everyone. Only logged-in users will have their progress saved.
Who this is for
- Marketing Analysts validating campaign changes.
- Growth/CRM Analysts monitoring promotions and retention.
- Product/Monetization Analysts assessing LTV impact.
Prerequisites
- Basic cohort concepts (grouping users by acquisition month/week/day).
- Comfort with metrics: CAC, ARPU, LTV, retention, gross margin.
- Ability to align date ranges and attribution windows.
Concept explained simply
A cohort is a group of users who started at the same time (e.g., acquired in March). To evaluate a campaign change, compare similar cohorts from before and after the change. Look at their revenue and retention over the same age (Day 7, Day 30, Month 2), not just calendar time.
Mental model
Think in two layers:
- Layer 1: The timeline split β define the exact cutover date/time of the campaign change. Build cohorts before and after this line.
- Layer 2: The cohort lenses β compare cumulative metrics by cohort age: retention, ARPU, LTV, payback, margin. If curves diverge early and stay apart, the change is likely impactful.
Key assumptions to keep consistent
- Attribution window and model.
- Price, margin, and refund policy.
- Currency and inflation treatment.
- Channel mix and major seasonality events.
Key metrics and formulas
- CAC = Total acquisition cost / New customers in that cohort
- ARPU(t) = Cumulative revenue up to time t / Customers in cohort
- LTV(t) β ARPU(t) Γ Gross margin%
- Retention rate(t) = Active customers at t / Original cohort size
- Churn rate(t) = 1 β Retention rate(t)
- Payback time β Smallest t where LTV(t) β₯ CAC
- Incremental LTV = LTV_post β LTV_pre (compare at same t)
For subscription, consider MRR and churn. For transactional, use order frequency and AOV by age.
Worked examples
Example 1 β New creative on Paid Social
Change date: April 10. Compare March cohorts (pre) vs. April 15β30 cohort (post). Assume gross margin 70%.
- Pre (Mar cohort): CAC $42; ARPU D30 $50; ARPU D60 $58; ARPU D90 $63.
- Post (Apr 15β30 cohort): CAC $48; ARPU D30 $56; ARPU D60 $67; ARPU D90 $74.
Interpretation
- LTV D90 (pre) = $63 Γ 70% = $44.10; Payback not reached by D90 (CAC $42? Actually payback reached near D60: $58 Γ 70% = $40.60 < 42, D90 $44.10 β₯ 42 β payback ~ between D60βD90).
- LTV D90 (post) = $74 Γ 70% = $51.80; Payback between D30 ($39.20) and D60 ($46.90) β ~around D55.
- Incremental LTV D90 = $51.80 β $44.10 = $7.70; CAC increased by $6 β Net +$1.70 per customer by D90.
- Decision: Likely positive but modest. Consider scaling if cash flow allows slightly longer payback vs. pre.
Example 2 β 20% discount campaign
Change date: November 1. Margin baseline 70% without discount; with discount effective margin drops to 55%.
- Pre: CAC $30; ARPU D30 $35; D60 $41; D90 $45.
- Post (discount): CAC $28; ARPU D30 $40; D60 $46; D90 $48.
Interpretation
- Pre LTV D90 = $45 Γ 70% = $31.50 β Payback not reached by D90 (CAC $30? It is reached near D90 because $31.50 β₯ $30).
- Post LTV D90 = $48 Γ 55% = $26.40 β Worse despite higher ARPU due to lower margin.
- Incremental LTV D90 = $26.40 β $31.50 = β$5.10; CAC improved by $2 β Still net negative (β$3.10).
- Decision: Discount boosted revenue timing but destroyed margin. Not good for long-term value.
Example 3 β Targeting shift to higher-intent audience
Change date: June 1. Margin 65%.
- Pre: CAC $55; ARPU D30 $52; D60 $64; D90 $70.
- Post: CAC $68; ARPU D30 $63; D60 $81; D90 $92.
Interpretation
- Pre LTV D90 = $70 Γ 65% = $45.50; Payback not reached by D90.
- Post LTV D90 = $92 Γ 65% = $59.80; Still below CAC ($68) by D90. However, the curve is steeper; projection suggests payback after D120.
- Decision: Good if cash runway and risk tolerance allow longer payback; otherwise, negotiate CAC down or improve monetization.
How to run a cohort comparison (step-by-step)
- Define the exact change: what changed, when (timestamp), and why.
- Select cohorts: choose acquisition cohorts strictly before and strictly after the cutover.
- Standardize: same attribution window, currency, refunds, and margin assumptions.
- Align by cohort age: compare D7 to D7, D30 to D30, etc., not calendar dates.
- Compute metrics: CAC, ARPU, LTV (with margin%), retention, and payback.
- Visualize: cumulative ARPU/LTV curves; retention curves by age.
- Decide: evaluate Incremental LTV vs. CAC change and cash flow impact.
- Document: write a short decision memo with data, assumptions, and next action.
Quick template for your memo
- Change description:
- Compared cohorts:
- Assumptions (attribution, margin):
- Key results (ΞCAC, ΞLTV@D90, Payback):
- Confounders considered:
- Decision and next steps:
Handling confounders
- Seasonality: Use matching months/holidays from last year if possible; or include control cohorts from adjacent periods.
- Channel mix shifts: Run per-channel cohorts or weight-adjust.
- Product/price changes: Keep margin assumptions updated; note feature launches.
- Inventory/stockouts: Flag affected cohorts; exclude or annotate.
- Attribution changes: Freeze model during test or reprocess both sides uniformly.
Visualization and interpretation
- Cumulative ARPU/LTV curves: If the post-change line is above and stays above at most ages, it is likely better.
- Retention curves: Detect early-life drop-offs or improved stickiness.
- Payback chart: The earlier the crossing point (LTV β₯ CAC), the better for cash efficiency.
Exercises
Complete these in any spreadsheet or notebook. Mirror of the tasks below in the Exercises section. Focus on method and interpretation.
Exercise 1 β Payback and incremental LTV
Pre-change: CAC $40; ARPU D30 $38; D60 $45; D90 $50. Post-change: CAC $46; ARPU D30 $44; D60 $54; D90 $60. Margin 60%.
- Tasks: Compute LTV at D30/D60/D90 for both; find payback month; compute incremental LTV at D90; recommend scale/hold.
Exercise 2 β Margin-aware discount check
Pre: CAC $28; ARPU D60 $42; margin 70%. Post (discount): CAC $26; ARPU D60 $46; margin 52%.
- Tasks: Compute LTV D60 both, net gain/loss per customer vs. CAC change, and a short decision note (3 lines).
Checklist before you conclude
- Compared cohorts at the same ages (D30/D60/D90).
- Used the same margin% for each scenario's reality.
- Noted any confounders and their likely direction.
- Stated payback window and cash implications.
Common mistakes and self-checks
- Mixing calendar time with cohort age. Self-check: Are you comparing D30 to D30?
- Ignoring margin. Self-check: Did you apply gross margin% before comparing to CAC?
- Changing attribution mid-comparison. Self-check: Same model/window used?
- Overfitting to one cohort. Self-check: Do results hold for 2β3 adjacent cohorts?
- Stopping at ARPU only. Self-check: Did you compute payback and incremental LTV?
Practical projects
- Build a cohort comparison dashboard: inputs (cutover date, margin%), outputs (ARPU/LTV curves, payback).
- Create a decision template: one-pager auto-filled from your dashboard to standardize approvals.
- Run a 4-week monitoring loop: weekly add a new post-change cohort and track if effects persist.
Learning path
- Start: Cohort basics β retention and ARPU by age.
- Next: Margin-adjusted LTV and payback.
- Then: Confounder control and sensitivity analysis.
- Finally: Experiment design and scaling decisions.
Next steps
- Apply the steps to your latest campaign change and document a decision memo.
- Share with stakeholders and agree on a monitoring cadence.
- Take the quick test below to reinforce concepts.
Mini challenge
In one paragraph: Explain to a non-analyst why a higher ARPU post-change can still be worse for the business. Use the words margin, payback, and cash flow.