Playbook

    Email A/B Testing Guide

    Most A/B tests in email marketing produce no useful result. Not because testing is broken, but because the test was too small, too vague, or measuring the wrong thing. Here's how to run tests that actually change what you do next.

    This is part of the Supercharged Email playbook, built to help you get better results without guesswork.

    TL;DR

    • Only test things that will change what you do next.
    • One variable at a time. Always.
    • Subject tests need ~1,000 per variant. Click tests need ~5,000.
    • A 1% lift is almost always noise. Want 10%+ before calling a winner.
    • Log every result in a single spreadsheet.

    1. Only test things that change a decision

    • If the result won't change what you send next time, don't run the test.
    • "Should the button be green or blue?" rarely matters. "Should we send Tuesday or Thursday?" usually does.
    • Write down the decision before the test: "If A wins, we'll do X going forward."

    2. Test one variable at a time

    • Two changes at once means you don't know which one moved the number.
    • Subject A vs Subject B with identical bodies. Body A vs Body B with identical subjects. Never both.
    • The single exception: testing two completely different creative approaches when you've ruled out small tweaks.

    3. You need more sends than you think

    • Subject-line tests (open rate): aim for ~1,000 recipients per variant for a meaningful read.
    • Body/CTA tests (click rate): ~5,000 per variant, clicks are 10x rarer than opens.
    • If your list is smaller, run the same test across 3-4 sends and pool the results.

    4. The 5 highest-ROI tests

    1. Subject line angle, curiosity vs benefit vs urgency. Biggest single lever on opens.
    2. Send day/time, Tue 10am vs Thu 2pm, run across 4 sends.
    3. From name, "Brand" vs "Jane at Brand". Almost always wins for newsletters.
    4. CTA style, single button vs button + text link.
    5. Email length, short (under 100 words) vs medium (200-300).

    5. Pick the right metric

    • Subject line / from name / preview text → open rate (with caveat: Apple Mail inflates it).
    • Body / CTA / image / length → click-through rate.
    • Offer / urgency / pricing → conversion rate or revenue per email.
    • Don't celebrate an open-rate win if clicks went down.

    6. A 1% lift is usually noise

    • Small differences on small lists are almost always random.
    • Rule of thumb: a winner needs ~10% relative lift AND to feel obviously different in copy/design.
    • If the test is close, the right answer is "either is fine, pick the easier one to keep doing".

    7. Write down the result, every time

    • Keep a one-row-per-test spreadsheet: date, hypothesis, variants, winner, lift, decision.
    • Most teams re-run the same test every 6 months because nobody wrote down the result.
    • Pattern-matching across 20 tests beats any single result.

    Shortcut: test subject lines first

    Subject lines are the cheapest, highest-leverage thing to test. Draft 2-3 variants with the Subject Line Generator, score them with the Subject Line Tester, then A/B test the top two.

    Do this today

    • Pick one variable to test on your next send
    • Write down the decision the result will change
    • Confirm your sample size is large enough
    • Run the test and log the result
    • Plan the next test using what you learned

    Frequently asked

    How long should I run a test before calling it?+
    Until you hit the sample size, not a clock. Subject tests usually resolve in 2–4 hours; click tests can take 24–48 hours.
    Can I test on a small list?+
    Yes, but pool results across multiple sends. One test on 500 people is too noisy to act on alone.
    What's a winning threshold?+
    10%+ relative lift with a meaningful sample. Anything less, pick whichever is easier to keep doing.
    Should I A/B test images?+
    Rarely. Image differences are swamped by other variables. Test copy first.
    What if my test is inconclusive?+
    That's a result too, the change doesn't matter, move on to a bigger lever.

    Put these strategies into action

    Start free with Constant Contact
    Opens Constant Contact's free trial, where you'll see our partner name, Synergy.The account is yours and the price is the same. We may earn a commission.

    Get new email tools in your inbox

    Free tools and quick, practical tips for sending better emails. No spam, unsubscribe anytime.