Reading Time: 15 minutes
What is an Email Marketing Split Testing?
Do you Split test your email marketing campaigns?
Email marketing Split Testing can be used by marketers to determine which variant of an email marketing campaign will perform best.
Also known as A/B testing and multi-variant testing, split testing is a technique marketers can use to improve email marketing performance.
The process involves sending two or more variants of an email to a segment of the email marketing list.
The one which performs the best wins the split test and can then be distributed fully.
In this post, we will talk you through split testing and the benefits you will reap
Why should you care?
Wait a minute.
Why should you bother with split testing? You are an email marketing professional with years of experience.
Your emails will be awesome anyway, won’t they?
Not so fast.
The truth is, nobody can get it right all the time.
Everybody makes mistakes.
Sometimes we all need to be steered in the right direction.
It’s the little changes that can make the biggest difference.
That’s where split testing comes in. One of the best ways to find which techniques work best is to test them to a segment of your marketing list.
Split testing eliminates doubt.
You know that the single change you have made led to the change in performance. There is no doubting whether coincidence was at play or whether causation was the real reason for your sudden increase in click through rates.
If you’d prefer to watch our split testing video, take a look below:
You can use this to fine tune details in your email campaigns.
You likely have a grasp of the bigger picture.
You know what message you want to get across.
However, choosing whether to use one subject line over another or finding the strongest call to action is an important detail. Don’t worry, there is no need to stress.
The best way to find out what works for you is to ask your contacts. Your contacts are at the centre of everything you do. Split testing gives you the opportunity to find out how they react to different stimuli in real time.
Sending a fully optimised campaign to your mailing list helps you achieve the best possible results for your business.
If you fail to split test, you will miss out on opportunities to make your campaign perform to its full potential.
Nobody wants that.
Split Testing Variables
Your next step towards a fully optimised email is to choose your split testing variable.
At this point, we want to give you the ultimate split testing advice.
Split test one element of your email campaign at a time
Sounds obvious, right?
Split testing email variables is only effective if you isolate each element in turn. If you choose to test a number of changes at one time, you will not be able to identify which change impacted your performance.
The truth about split testing is that you are limited only by your email campaign.
We will take a look at the different types of split test in turn.
Subject Line Testing
The email subject line is the first thing your contacts notice.
Your subject line is one of the most crucial elements of your campaign. Customers use the subject line to decide whether or not they will open your email.
Do you split test email subject lines to make sure your contacts have an incentive to open your emails?
For a subject line to be effective, you must show your contacts the value of opening your campaign.
Get the subject line wrong and your campaign will fail.
Sorry to be blunt!
As well as influencing whether your email finds its way into the spam folder, the subject line directly impacts open rates. The subject line is one of three elements of the email that recipients use to decide whether to open a particular email campaign. However, the impact of using an effective subject line is much wider reaching.
Consider this campaign from GNC:
Subject Line: LAST DAY/15% off your entire order
The GNC subject line gives rise to many split testing opportunities
What do you think?
Here in the Wired Plus office, we think this is a great example of a well optimised subject line.
GNC demonstrate the value of opening their email campaign straight away.
Should GNC want to split test a similar subject line in a future campaign, they could try including personalisation: LAST DAY/Get 15% off your entire order, [Name]
Or, emojis: LAST DAY/15% off your entire order
Each of these examples gives you an idea of how you can make slight adjustments to an already good email subject line.
Even with the current subject line, GNC give the reader a clear indication of what to expect inside the email. A recipient reading GNC’s subject line would likely open the email with the intent of making a purchase.
Subject Line Split Testing Ideas
Coming up with split testing ideas is great fun but we thought we’d help you out with a few testing ideas of our own:
-
Subject Line Length
According to Invescpro, the performance of subject lines vary according to subject line length. Subject lines of 6-10 words performed best with an open rate of 21%. Conversely, long subject lines of 21-25 words performed worst with an open rate of 9%.
Be concise. Shorter subject lines often work better because they get the message across more succinctly.
However, obsess about the length of your subject line at your own risk! We believe that the quality of what you say is more important than the quantity!
-
Preview Header Text
The second element the recipient can use to decide whether to open the email campaign is the preview text. Your recipient has very little information when they see the email in their inbox.
Split testing different preview text strings may influence the open rate of the email campaign if the value of the email is more clearly stated.
The preview text needs to supplement your subject line, working as a team to drive open rates upwards.
-
Using Emojis
Emojis increase email open rates in some cases.
Econsultancy found that emojis work around 60% of the time. However, when emojis fail, performance deteriorates.
You might wonder whether they would fit with your brand or for your specific audience. One way to find out is to trial the use of emojis in your subject lines in a split test.
Have you performed split tests to see whether emojis improve campaign performance?
Emoji or no emoji, you need to come up with a strong subject line for your email campaign.
Think of them as a turbo on your car. They will enhance, not revolutionise performance when used well.
We’d love to know how SMW New York’s emoji filled subject line performed in a split test with a text-only example.
This example from SMW New York demonstrates good use of emojis in the inbox. The most important thing to note about this example is that the emoji fits naturally into the context of the subject line.
A Quick Idea
We suggest looking at popular emojis on social media sites and blog posts before you start experimenting with your email campaigns. You could split test emojis in your blog posts and on your website before using them in email campaigns.
This will allow you to split test to a larger audience, making your results more reliable. When you then introduce emojis into your email campaigns you can be confident that they will work.
-
Including Numbers and Percentages
Ever received an email and felt inclined to believe it straight away? Citing numbers and percentages in your subject lines adds validity to the claims you are making. This adds authority to your subject line copy.
Overusing numbers and percentages to make your point may seem pushy or forced so be careful. A split test can reveal whether your numbers and percentages add enough value to the email subject line to make a difference.
If they don’t, get rid!
-
Personalisation Techniques
Addressing your contact by name in the inbox makes them sit up and take notice if done correctly.
A study by Oracle demonstrated the improved performance associated with email personalisation. Although use of the recipient’s name drove open rates up by 5%, custom personalisation techniques increased open rates by nearly 10%.
That doesn’t seem like much until you scale it up!
To use custom personalisation techniques you need to look deeper into your data, perhaps looking at your contact’s previous purchases, birthday or an upcoming event.
As with any other technique, it is worth trialing these techniques for yourself. A split test could be used to find the optimum level of personalisation for an email subject line.
-
Wording
‘It’s not what you say it’s the way that you say it’ is a popular phrase for a reason.
Subject lines need to stand out. Each subject line you write should be concise and original, differentiating it from the crowd.
Be aware of the perils of sounding boring or flat. You need your subject line to grab your readers’ attention and boost open rates.
An Invescpro study found that the words ‘Free’, ‘newsletter’ and ‘fw:’ caused a downturn in performance. They aren’t original, don’t give users true value and appear ill thought-out.
A poor email subject line split test example
Boring, isn’t it?
This example from SurveyMonkey uses one of the terms that Invescpro highlighted in their study. Using the subject line ‘Newsletter: June 2018’ doesn’t give the recipient a good reason to open the email campaign.
However, if SurveyMonkey had used a subject line such as ‘The ultimate checklist for your next survey’, this presents the value of the email in a much clearer way.
Subject lines are an opportunity to develop your brand. Once you have hit upon a winning formula, slight tweaks to your future subject lines will yield great results.
For some inspiration with creating your next email campaign, take a look at our Subject Line Swipe File.
Testing Campaign Content
Of course, subject lines aren’t the only thing you can test.
The content your recipients view when they open the email is just as important.
The Caffe Nero ‘Double Crackers’ email is clear, succinct and visually appealing
You can also look at any number of elements within your email campaign. Possible split tests include:
-
Format:
The way that you present information in an email can impact the way people interact with it. The F-shaped reading pattern shows how people scan information on a screen and is a good guide for email marketers to follow. Have you noticed yourself scanning content in this way?
Would a standardised design allow you to get your main message across with the level of creativity you need or do you need to take an alternative approach?
-
Image Positioning:
The positioning of your images can help your recipients scan your emails quickly. This may be as simple as moving your image from above to below a paragraph. We’ll discuss this idea in much more detail later in the post.
Have you carefully considered where your images are placed in your email campaigns? Does their positioning complement the copy you use?
-
Call To Action:
The position and wording of your call to action can also impact your click through and conversion rates. Writing different call to action copy for a split test will improve the overall click through rate of your email campaign when it is distributed to the full mailing list.
Do you compel your readers to click through from your email campaigns with your current call to action copy or could it be improved?
-
Length of Copy:
Limit the amount of copy you write. Present your copy in a short and concise way. The purpose of your email is to drive your recipients towards an action. The longer they have to spend reading your email campaign, the less likely they are to act.
Would you be able to get your message across in a few short paragraphs or does longer form content work better for your brand?
Each aspect of your email marketing campaign impacts performance in some way.
Building Your Split Test Hypothesis
Identifying the right metric to analyse your split test result is essential.
In other words, make sure you know what to look out for. The type of split test you perform will affect the performance metric you should monitor.
When you split test one element of your email campaign, a number of performance metrics may change. Without a clear idea of which metric you need to look at, this quickly becomes confusing.
Do you know what metrics you need to monitor?
Do you know the results you expect to find?
Before you perform the split test, you should have a rough idea of the result you expect to see. Your intuition is great to use when split testing as the results are often consistent with your initial hunch. Note down your ideas so that you don’t forget.
The next stage is to decide what success looks like.
Defining Your Split Test Metrics
Decide which metric is most important to your performance.
Sure, we know that everybody wants to make more money to help pay the bills, but what specifically, do you want to achieve with your email split test?
Before you split test any element of your email campaign, you must know what you expect to see.
Split testing a subject line is likely to change:
- Open Rates: The most obvious reason to split test your subject line is to check for changes with open rates.
- Spam Rates: Using particular words or phrases might influence whether your contacts mark your emails as spam. This could have a knock on effect to your email’s open and click through rates.
Split test your subject line to make sure your email is not flagged as ‘Spam’
However, you should bear the Halo Effect in mind, other changes may also take place that you can’t analyse so easily. We would question how literally these results can be interpreted.
How do you know that the increase in click through rates happened because you changed your subject line? The two elements are seemingly disconnected and any changes are down to correlation rather than causation.
Don’t jump to any hasty conclusions!
Similarly, you can’t measure the amount of emotion somebody feels when reading your subject line, or what first impression they may have gathered.
Choosing Who Receives Your Email Marketing Split Test
To perform an effective split test, the next step you need to take is to select a segment of your contact list.
The process of sending your email marketing split test doesn’t end there. Next, you need to decide who you will send your test to.
The larger the split test, the more reliable your results will be. However, increasing the size of your split test also means that more people will receive an unoptimised campaign.
It’s a balancing act.
For a split test to be effective, you need to choose a representative sample of your mailing list. We’re not talking about going to extremes you might have learned about in science.
Use your common sense to identify whether your split test takes into account a large enough cross section of your list.
A good email marketing platform will even allow you to test the cross section of the list you are using.
What do I do if I’m only sending to a handful of people?
I hear your cry! You have to make sure that your split test is statistically viable. As your email marketing list gets smaller, the value in the results of a split test decline. If you need to send an email to a very small segment of your list, take a look at previous campaign data and trust your instincts.
We suggest that, for a split test to work, your minimum list size should be at least 1,000. However, even then, your results will not be as reliable as with a much larger list.
Analysing Email Marketing Split Testing Results
It’s a marketers’ dream when the results of the split test are clear cut and the difference between variants is huge.
Unfortunately, that’s not always the case. If your results look weirdly skewed, take a look at your hypothesis. You may want to discard the results and test again. Here are some things to consider:
Confidence
Your split test should have a confidence interval of at least 95% to ensure that the result is reliable.
Investopedia define a confidence interval as:
“A confidence interval is an interval that will contain a population parameter a specified proportion of the time. The confidence interval can take any number of probabilities, with the most common being 95% or 99%.”
In plain English, confidence is based on your own intuition and reflects how much you can trust that your results reflect the model that you are trying to test.
If you have tested a particular element of your campaign on a small list segment, but you do not feel that the results will be applicable to the full marketing list, then the confidence level for the test will be low.
Take the following example:
A company performs a split test to check to see whether orange or blue call to action buttons will work best in their next email campaign.
The result of the split test suggests that the blue call to action button works best. However, the test was only performed with a small subsection of the email list.
Intuitively, a good marketer would be able to identify that the orange call to action button would work best.
If your confidence level falls below 95% and the results don’t seem to match your hypothesis, then the results should be discarded.
This is not always easily measured but is a good sense check to make sure that you are not blindly following the results of a poorly performed split test.
Statistical Significance
The result of the split test should be statistically significant.
Statistical significance determines whether the results of a test occur because of causation or if they are simply a correlation. When the result of a split test is statistically significant it can be assumed that the split test variable is responsible for the change.
Where the results are marginal and aren’t statistically significant, allow your experience, previous campaign data and gut feeling to guide you to the right selection.
Split tests need to be statistically significant before their results can be relied on for marketing campaigns
Split Test Duration
Think about the amount of time that you wait before analysing the results of your split test. If you are eager to send your email campaign out, it is tempting to perform a split test over a very short period of time.
The problem with this is obvious.
If your contacts are away from their desk, or don’t check their emails on their phone, for an hour or two then your results will be skewed. Make sure that you plan enough time in between sending the split test and making any decisions.
Don’t be tempted to declare your results early.
Once you have identified the best variant of your campaign, you can send the campaign to the rest of your mailing list.
If you are planning to perform several split tests, you need to have a longer amount of time and a larger list.
Top Tips for Split Testing Your Next Campaign
We thought we’d share some top tips with you to help you get started:
Test one element at once
We can’t stress this one enough.
We often encounter marketers who have tested their campaigns, only to find out that they don’t know why performance changed. This happens because they changed several email elements or even their whole email campaign.
The only way you will find out which element of your email campaign altered your performance is to isolate each one in turn.
Know what you are testing well in advance
Don’t decide to do a split test at the last minute.
You should plan your split test before you started the creation process. This will save you lots of arguments and confusion in the testing stage!
Identify what makes an email successful
Know what you are looking for.
If you are optimising your subject line, know that you should be looking for changes in open and spam rates. Any other improvements are a bonus but shouldn’t affect your final decision.
Make sure that your results are statistically significant and have a high enough confidence value before you take their results into account.
Plan Time For Split Testing
If you rush the split testing process, you are wasting your time.
You need to send your email split test to a segment of your contact list.
Then you must wait. Allow your contacts enough time to receive and react to your split test before judging the results.
How much time do you use when planning your email marketing split test?
More haste, less speed.
Don’t Expect Miracles
Sure, split testing will improve your performance.
However, you should have a clear idea of how similar email campaigns have performed before you review your split test results. Look at your historical data and measure the split test accordingly. Incremental improvements are better than standing still.
Your results will add up over time!
Choose The Right Email Marketing Platform
Not all email marketing platforms are created equal.
For example, you will not be able to split test your campaign if your company uses Outlook.
Instead, you need to invest in an email marketing platform with split testing functionality.
Consider Your List Segments
Make sure that you have enough contacts to make a split test worthwhile.
If you have tightly controlled email list segments and a refined email marketing approach, you may not need to split test your campaigns at all.
Small, targeted segments allow marketers to be extremely specific about their messaging.
Split testing campaigns would not be effective in this instance as small segments do not contain enough contacts to perform a split test.
Some Great Email Split Testing Examples
So, now we’ve discussed how you can split test your own campaigns, we thought we’d take a look at some email campaigns and how split testing could be used to enhance them.
We don’t know whether the emails have already been tested, these examples are purely suggestions of what you could do:
Map My Run
The Map My Run campaign leads to many split testing opportunities
In this example, we wonder whether the four elements could be named a little more inventively. Currently, each element is described as ‘Essential x’ with a generic ‘Read More’ call to action.
It’s a bit… beige.
This doesn’t give the reader the benefit of clicking through to that content piece. Providing the benefit of clicking through would make the purpose of the email clearer to the recipient.
One split test of this email would reveal whether mixing up the call to actions could drive more engagement. Being specific about the benefits of taking action would surely benefit this MapMyRun email campaign.
Alternatively, you could split test the description given to each of the email elements. In each case, the click through rate of the email would be monitored to determine which variant was successful.
TGI Fridays
We think the TGI Friday campaign could be improved with a couple of split tests…
The position of the call to action in the TGI Friday email is unusual. Normally, in an email of this type, you would expect the call to action to be positioned in the centre of the email body.
Positioning the call to action in the centre would help funnel the reader down from the header to the call to action. The success of the split test would be determined by the click through rate.
The TGI Friday campaign has a busy design, with three distinct elements:
- Tomahawk Steak Promotion
- Father’s Day Bookings
- Download Our App
For an email campaign to be successful, it should have one main message. Clustering these three call to actions in one email makes the email look overcrowded.
Using just one call to action per email campaign would give the email more focus.
Positioning the text over the top of the steak also detracts from its appeal. The imagery in the email body could be presented in a much more appealing way, driving higher engagement rates.
Summary
As you have seen, split testing your email marketing campaign is a great way to improve campaign performance.
Marketers have a huge number of decisions to make when creating a marketing campaign and it is easy to become overawed. Yet, by split testing your campaigns you can rest assured that you are presenting your brand in the very best way.
Once you have the right tools in place, a good strategy and hypothesis, this process will help you deliver fully optimised email marketing campaigns.
Share your split testing successes with us in the comments!
The post Email Marketing: Split Testing appeared first on Wired Plus Marketing Hub.
This post first appeared on Wired Plus Email Marketing Platform, please read the originial post: here
