Email Testing and Optimization Cheat Sheet >


One reason that email is an enduring, powerful channel for marketers: the ability to prove itself with numbers. If you aren’t using the phrase, “Let’s test into it!” on a daily basis, you’re doing it wrong.

Testing can be daunting when terms like “statistical significance,” “holdout group” and “hypothesis” are thrown around, but it is truly one of the most powerful tools for optimizing your email marketing program. Here are some of the basic steps and requirements to design and deploy a statistically sound test.

Set Goals 
Be specific with your optimization. Do you want to increase awareness of the campaign message? Do you want to improve engagement with a particular content area?  Are you trying to increase conversion or revenue generated from email? It is important to design your test with a specific goal in mind so you can be confident in the results of the test.

Develop a Hypothesis 
What do you think will make an impact? Do you think lifestyle imagery will convert better than an infographic or will emoji in the subject line drive a higher open rate?  Maybe you’ve been wondering if the use of animation in your email content actually increases engagement or if it’s just a waste of time. Determine your hypothesis and stick to it. The entire purpose of the test is to identify if your hypothesis is right or not, so make sure it is specific and measurable.

Design Your Test
Work with creative resources to determine the test versions. Don’t test too many areas at once, but be prepared to spend more time analyzing results if you perform multiple tests.

Determine Your Audience
You’ll need statistically sound test results so it is important that you have a large enough sample size (test audience) to determine the winner. If the campaign will be automated you can analyze volumes over time to determine how long you should let the test run. Use a test like statistical significance to help you understand if your test results are reliable and repeatable. Make sure that the test audiences are randomly selected so you can be sure of your results.

Use a Hold Out Group
Set aside a portion of the audience to suppress from the campaign. This will allow you to measure lift or the incremental increase in revenue generated (or not generated) from the campaign. Hold out tests can determine the lift generated by a campaign, and whether or not your audience would purchase without one.

Deploy Your Test 
You’ll want to deploy all versions of your test at the same time on the same day to ensure no other factors are impacting the test results.

Measure Your Test Results
Wait for all of the results to come in (anywhere from 48 – 72 hours for a simple A/B test to several months for an ongoing campaign with a small audience) and use a measurement like statistical significance to determine the winner and reliability of the results. Statistically significant results will confirm whether you can be certain that the test results are repeatable and can be applied to future campaigns. Don’t be upset if your hypothesis is disproven. This is why we test! I’ve seen tried and true “best practices” fail when tested against a version that wasn’t expected to win. These often lead to follow-up tests since they generate questions about the outcome.

Apply Your Learnings 
Now that you’ve proven or disproven your hypothesis you can apply what you learned to future campaigns. Periodically re-run your test to identify whether changes in your target audience, marketplace or other factors may result in different results.


Leave a Reply
(will not be published)