We’ve all heard that if you’re in marketing, in particular email marketing, you should constantly be testing to maximize results. The most common test mentioned is the ubiquitous “A/B” split test, meaning a 50/50 list split to test one variable against another (graphics, copy, offer, layout, list, time of day, day of week, etc.).
But is an A/B test all you can or should do? If you have only a few thousand or fewer emails to work with, an A/B test may be all you can do to ensure statistically reliable results. However, if your list is too small, an A/B test might not make any sense. For example, if you only have a few hundred email addresses, splitting and conducting one test will literally tell you nothing (statistically) other than directionally relevant information. Instead you may need to try to replicate the test over time, to aggregate the results and to analyze your collective data over a longer period.
The first consideration is to quantify how many email addresses you need to test to ensure you have a representative sample and more importantly, to ensure the results are reliable. There is a lot of math and science behind this topic, and fortunately a lot of math/science/statistics sites have free online tools such as this one.
You must set up the test(s) correctly (with sufficient sample sizes and assumed response rates) on the front end to ensure that results on the back end are reliable, meaning with a confidence level that you’re comfortable with (we recommend a 95% confidence level if it’s possible). Again, there are resources online to assist such as this one. The key is to avoid the common mistake of merely looking at results and assuming winners/losers based on seemingly different response rates.
Before testing, you have to identify the goal or the question you’re trying to answer. We recommend that you actually write these down and then, as briefly and concisely as possible, describe the various yardsticks you will use to determine your winner. As form follows function, the goals/objectives of the test coupled with the means to measure results should help drive copy, graphics, and/or layout to ensure the messages are properly structured and focused on whatever question you’re trying to answer..
Let’s say your goal is a higher click rate and after an A/B test you find “A” has a 2.7% CTR and “B” has 2.85%. It is a common mistake to use subtraction and declare that “B” was the winner or that “B” was only 0.15% higher and that could lead you down the path of thinking it wasn’t a significant result (i.e. a virtual “tie”). Or maybe you routinely just pick the higher percentage as the winner and run with that. Using proper percent increase/decrease calculations, we find that this is actually a 5.56% increase from “A” to “B.”
That however may or may not be statistically significant, but as you can see it’s a much larger increase than originally assumed. In order to determine if the results are statistically significant, use one of the calculators, plug in each version’s list size and the click percentage (or open percentage, or conversion rate, etc. depending on the key metric you’re analyzing) and it will instantly tell you whether this difference is enough to be reliable (with a 95% confidence level).
In this example, let’s pretend I sent “A” and “B” to a random 2,000 people each. The calculations indicate that this would not be enough of a difference to be statistically reliable. In fact, the “B” cell’s click rate would have to have been at least 3.81% in order for the difference to be reliably significant. However, if you didn’t analyze the results properly you wouldn’t know this.
The other way to ensure you’re maximizing your results is to avoid doing a full scale A/B test. If your database for an email marketing campaign is large enough (again calculate minimum sample size), you can do a different kind of split test. First, split your list 10%/90% (ensuring it’s random). Then split the 10% group in half so you have two small splits and the remaining 90%.
Deploy your test to the 10% splits, give as much time as possible for activity to occur (twenty-four hours if possible), analyze the results and then deploy the winner to the remaining 90%. That way you’ve done your best to maximize the campaign’s results without going “all in” on a typical full file A/B split.
As with gambling, learn the rules, do the math, analyze the data and place your bets. Do it right, and the odds will swing in your favor.
Animation can make your message stand out from the email crowd. But without knowing how to use animation you can easily overwhelm your readers. Find out the do’s and dont’s of effective email animation in an all new Email Marketing Minute!
How easy is it to find an email opt-in on a homepage? SubscriberMail takes a look at three websites to determine how placement of an opt-in box can make or break the success of an email program.
Several months ago I wrote a post about Re-Touch Efforts and Seeing Positive Results. In it I noted some unexpected success with re-mailing the exact same offer to individuals who neither opened nor clicked the original email they received three to four days earlier. What was surprising to me at the time was the fact that the re-touch effort resulted in the same response rate as the original. And it was surprising because the rule of thumb in direct mail was always that you’d get about half the response rate that you received in the initial mailing everything being equal.
But when you think about it, when are all things ever equal? They never are. Today is not like yesterday; next Wednesday won’t be like this past Wednesday, July 15 this year won’t be like July 15 next year. And that may well be the reason for email re-touches being much more effective than direct mail re-touches. Despite the differences, yesterday is generally more like tomorrow than a day 28 months from now just as an individual’s recent past behavior is more like is/her near term future behavior than their behavior at some distant point in the future.
I don’t have any empirical evidence but I think the reason email re-touches are more successful than direct mail re-touches is that they generally occur so much faster. You don’t need to wait 30 days for results or even two weeks for projected-results to determine which segments are responding to which offers using which creative approaches. Nor do you have to worry about production schedules and print queues. With email your questions are answered in hours. You not only know which segment(s) responded at what rate to which offer using which creative, you know that you can deploy the re-touch email to the right target in minutes with the winning offer tucked neatly into the winning design.
Since April we’ve had the opportunity to use these findings in combination with one another with some even more impressive results. Not only have we managed to equal the response rate’s of the initial offering, in more than one instance we’ve managed to triple it by combining the most responsive list segments with the most appealing creative.
In each of these scenarios we’ve chosen to hold the offer constant for purposes of simplicity and I think we’ve made a mistake in doing so. Inasmuch as the “offer” generally carries more weight in the email success equation than does “creative”, the next effort will be to measure the effect of quickly re-touching winning segments with winning offers using winning creative to see if we can’t raise the response rates even further. The trick will be to do that without getting so narrow and so microscopic that we find the one guy in 100,000 who’s ready to buy and no one else. Will keep you posted on our progress.
When is the last time you reviewed your email opt-in form(s)? Are you still asking visitors for their birth date even though your birthday promotion was canceled months ago?
It is a good idea to periodically review/update your company’s opt-in strategy to align with current business objectives. Marketers love having access to gobs of data, which can sometimes work against them. The key is finding the right balance of what to request and what you actually use in your program. Planning for the future is not necessarily a bad idea but asking users to complete multiple fields for data that you currently do not use can lead to lower opt-in rates.
If you are in doubt, use the formula below as guide to determine how much is too much. Your PI, or Personalization Index, is determined by dividing the number of elements (data sources) you use in your email marketing strategy by the number of elements you collect (data fields on your opt-in form). The rule of thumb is that if your PI is less than .3 you are collecting too much. In other words, you are asking for far more information than you actually use in your email marketing efforts.
Let’s take a look at an example of this. Say an online retailer requests 14 fields on their opt-in form. In addition to name and email address they are requesting items like birth date, HH income, zip code, marital status, etc. These are all great demographic data points and can be used to segment and personalize messages, but is it too much? If this retailer only uses first name, email address, zip code and household income in their program then it is too much.
PI = 4/14 = .285
By eliminating just one or two of these fields the ratio would move above the .3 measurement. Understand that this is meant to serve as a rule of thumb and is more of an exercise to get you thinking about the data you collect and how you use it. So take a look at what your PI ratio is and remove any unused/unnecessary elements from your forms. You may soon see an increase in quality and volume.
We’ve all heard that if you’re in marketing, in particular email marketing, you should constantly be testing to maximize results. The most common test mentioned is the ubiquitous “A/B” split test, meaning a 50/50 list split to test one variable against another (graphics, copy, offer, layout, list, time of day, day of week, etc.).
Several months ago I wrote a post about 
When is the last time you reviewed your email opt-in form(s)? Are you still asking visitors for their birth date even though your birthday promotion was canceled months ago?
