Budget Split Testing Basics

Why this matters

As a Marketing Analyst, you often need to split limited budget across test variants while protecting current performance. Typical tasks include:

• Deciding between a 50/50 learning split vs. a safer 90/10 split for a risky creative.
• Estimating how much spend is needed to reach a reliable result this week.
• Setting guardrails (e.g., CPA, CTR) and ramping a winning variant.
• Communicating test timelines, risk, and expected ROI to stakeholders.

Concept explained simply

Budget split testing is how you divide spend between variants (A, B, …) during an experiment. Your split affects speed to learn, total cost, and risk to performance.

Simple rule of thumb:

• 50/50: fastest learning and lowest cost to reach significance.
• Weighted (e.g., 90/10): protects performance but needs more time or total traffic to learn.
• Staged ramp: start small (e.g., 10/90), check guardrails, then widen to 50/50 or promote a winner.

Mental model

Think of budget split as a speed vs. safety dial:

• Left (safety): small exposure to the risky variant; slower learning.
• Right (speed): equal exposure; fast and cheap learning.

Guardrails act like seatbelts—if a metric breaks thresholds, you slow or stop the test.

How to choose your split

1. State the goal: maximize learning speed or protect short-term performance.
2. Pick an initial split: 50/50 for low risk; 90/10 or 80/20 for riskier changes.
3. Define guardrails: e.g., CPA must not exceed baseline by 25%; CTR must not drop below X; spend caps per day.
4. Estimate sample size and budget using baseline p, target MDE, and CPC.
5. Plan a ramp: small exposure for 1–2 check cycles, then move to 50/50 if stable.
6. Set stopping rules: minimum sample size and minimum time (cover weekdays/weekend patterns) before deciding.

Worked examples

Example 1: 50/50 prospecting ads

Baseline CVR p = 3% (0.03). Target MDE = 0.6 percentage points (20% relative of 3% → 0.006 absolute). CPC = $0.80.

• Visitors per variant: 16 × 0.03 × 0.97 / 0.006^2 ≈ 12,934.
• Spend per variant: 12,934 × $0.80 ≈ $10,347.
• Total test spend (50/50): ≈ $20,694.

Takeaway: 50/50 gives the fastest path to this learning target.

Example 2: 90/10 safety split for a risky landing page

Baseline CVR p = 4% (0.04). Target MDE = 0.4 pp (10% relative). CPC = $1.50.

• Visitors per variant (for the planned MDE): 16 × 0.04 × 0.96 / 0.004^2 ≈ 38,400.
• At 90/10, the 10% variant needs 38,400 visitors → total traffic ≈ 384,000 visits.
• Budget at $1.50 CPC: ≈ $576,000 to hit that MDE at 10% exposure.

Takeaway: Heavy weighting protects revenue but learning becomes slow/expensive. Use a staged ramp or accept a larger MDE initially.

Example 3: Staged ramp with guardrails

Plan: Start 10/90 for 1 day; if guardrails pass, go 30/70 for 1–2 days; then 50/50. Guardrails: CPA within +25% of baseline, CTR within −10%, no policy flags.

• Day 1 (10/90): quick safety check.
• Days 2–3 (30/70): collect more signal; monitor daily.
• Days 4+: 50/50 until hitting minimum sample size and time window.

Takeaway: Ramps reduce downside risk while keeping a path to efficient learning.

Checklist: ready to run

• Clear goal (learn fast vs protect performance).
• Initial split chosen and ramp plan written.
• Guardrails set (CPA/CTR/CVR thresholds, spend caps).
• Sample size and budget estimated per variant.
• Minimum runtime covers seasonality (e.g., at least 7 days if daily patterns matter).
• Stopping rule defined (hit n and time; then decide).

Exercises

These mirror the interactive exercises below. Do them here first, then check your answers.

Exercise 1: Plan a 50/50 test budget

Context: Paid social test, 50/50 split. Baseline CVR p = 2.5% (0.025). Target MDE = 20% relative (absolute 0.5 pp = 0.005). CPC = $0.60.

• Q1: How many visitors per variant are needed (approx)?
• Q2: What is the spend per variant and total test spend?
• Q3: Roughly how many conversions per variant at baseline?

Exercise 2: Choose a safe split and ramp

Context: New landing page may hurt performance. Baseline CPA = $40; must keep blended CPA ≤ $45 during test. Traffic CPC = $0.90.

• Q1: Propose an initial split and a two-step ramp plan.
• Q2: Define 2–3 guardrails with numeric thresholds.
• Q3: Define a stopping rule that balances safety and learning.

Common mistakes and self-check

• Mistake: Always using 90/10. Self-check: Calculate time/cost to reach your MDE; if huge, switch to staged ramp toward 50/50.
• Mistake: No guardrails. Self-check: Do you have CPA/CTR/CVR thresholds and a daily review plan?
• Mistake: Peeking early. Self-check: Have you set a minimum sample size and time window before deciding?
• Mistake: Tiny MDE. Self-check: Would detecting this lift change budget decisions? If not, increase MDE.
• Mistake: Ignoring traffic quality mix. Self-check: Keep placement/audience mix consistent across variants.

Who this is for

• Marketing Analysts and performance marketers running paid tests.
• Product and growth analysts planning controlled experiments.

Prerequisites

• Basic understanding of A/B testing concepts (control, variant, conversion rate).
• Comfort with simple percent and cost calculations.

Learning path

1. Understand when to use 50/50 vs weighted splits.
2. Estimate sample size and translate to budget.
3. Set guardrails and ramp plans.
4. Run, monitor, and stop with confidence.

Practical projects

• Create a one-page test plan for a real campaign, including split, MDE, sample size, budget, guardrails, and ramp schedule.
• Build a simple calculator (spreadsheet) that takes p, MDE, CPC and outputs visitors and spend per variant.

Next steps

• Apply staged ramping on your next high-uncertainty creative test.
• Refine your MDE by aligning with business impact (e.g., CPA or revenue goals).

Mini challenge

You have 5 days to test two video ads. Baseline CVR 3.5%, target MDE 15% relative (0.525 pp), CPC $0.70. Propose a split and ramp that fits the timeline, with guardrails. Estimate total spend to reach your MDE. Make trade-offs explicit if you cannot fully reach the target in 5 days.