Posts Tagged ‘Email Marketing’
Posted by Rob Ropars on August 26th, 2011
We’ve all heard that if you’re in marketing, in particular email marketing, you should constantly be testing to maximize results. The most common test mentioned is the ubiquitous “A/B” split test, meaning a 50/50 list split to test one variable against another (graphics, copy, offer, layout, list, time of day, day of week, etc.).
But is an A/B test all you can or should do? If you have only a few thousand or fewer emails to work with, an A/B test may be all you can do to ensure statistically reliable results. However, if your list is too small, an A/B test might not make any sense. For example, if you only have a few hundred email addresses, splitting and conducting one test will literally tell you nothing (statistically) other than directionally relevant information. Instead you may need to try to replicate the test over time, to aggregate the results and to analyze your collective data over a longer period.
The first consideration is to quantify how many email addresses you need to test to ensure you have a representative sample and more importantly, to ensure the results are reliable. There is a lot of math and science behind this topic, and fortunately a lot of math/science/statistics sites have free online tools such as this one.
You must set up the test(s) correctly (with sufficient sample sizes and assumed response rates) on the front end to ensure that results on the back end are reliable, meaning with a confidence level that you’re comfortable with (we recommend a 95% confidence level if it’s possible). Again, there are resources online to assist such as this one. The key is to avoid the common mistake of merely looking at results and assuming winners/losers based on seemingly different response rates.
Before testing, you have to identify the goal or the question you’re trying to answer. We recommend that you actually write these down and then, as briefly and concisely as possible, describe the various yardsticks you will use to determine your winner. As form follows function, the goals/objectives of the test coupled with the means to measure results should help drive copy, graphics, and/or layout to ensure the messages are properly structured and focused on whatever question you’re trying to answer..
Let’s say your goal is a higher click rate and after an A/B test you find “A” has a 2.7% CTR and “B” has 2.85%. It is a common mistake to use subtraction and declare that “B” was the winner or that “B” was only 0.15% higher and that could lead you down the path of thinking it wasn’t a significant result (i.e. a virtual “tie”). Or maybe you routinely just pick the higher percentage as the winner and run with that. Using proper percent increase/decrease calculations, we find that this is actually a 5.56% increase from “A” to “B.”
That however may or may not be statistically significant, but as you can see it’s a much larger increase than originally assumed. In order to determine if the results are statistically significant, use one of the calculators, plug in each version’s list size and the click percentage (or open percentage, or conversion rate, etc. depending on the key metric you’re analyzing) and it will instantly tell you whether this difference is enough to be reliable (with a 95% confidence level).
In this example, let’s pretend I sent “A” and “B” to a random 2,000 people each. The calculations indicate that this would not be enough of a difference to be statistically reliable. In fact, the “B” cell’s click rate would have to have been at least 3.81% in order for the difference to be reliably significant. However, if you didn’t analyze the results properly you wouldn’t know this.
The other way to ensure you’re maximizing your results is to avoid doing a full scale A/B test. If your database for an email marketing campaign is large enough (again calculate minimum sample size), you can do a different kind of split test. First, split your list 10%/90% (ensuring it’s random). Then split the 10% group in half so you have two small splits and the remaining 90%.
Deploy your test to the 10% splits, give as much time as possible for activity to occur (twenty-four hours if possible), analyze the results and then deploy the winner to the remaining 90%. That way you’ve done your best to maximize the campaign’s results without going “all in” on a typical full file A/B split.
As with gambling, learn the rules, do the math, analyze the data and place your bets. Do it right, and the odds will swing in your favor.
Posted by Nic Winters on June 10th, 2011
Are you a supervisor that likes to stay informed… or perhaps you have a boss that likes to stay informed? SubscriberMail features two tools that clients can utilize to allow them or their co-workers to keep informed regarding emails deployed through their SubscriberMail account.
Clients can find both features within the Admin tab of their accounts—Message Delivery Notifications and Seed List.
If activated, The Message Delivery Notifications tool will send an email alert after your message has deployed to any email addresses you specify. This alert will include the subject line and the size of the list the message was sent to, followed by a second alert prompting the recipient(s) to log into SubscriberMail and view the reporting metrics that have been gathered.
With the Seed List function you can automatically append email addresses to the deployment list of every message sent through your SubscriberMail account – allowing others to receive a copy of each message you deploy.
Both features can provide an easy method to keep yourself or your colleagues informed regarding all of your email campaign activity.
Contact the SubscriberMail Client Support team at support@subscribermail.com for more information about how you can utilize these features and keep informed!
Posted by Bill Leming on September 9th, 2010
In email marketing, there are always a lot of questions about how to judge the performance of an email campaign, and what will make it the most successful. The questions have been answered in any number of ways across the industry, but we have tended to see people with the same three questions since the birth of email marketing.
We’ve come to call them The Big Three:
1. What kind of a response rate should we expect from our list?
2. How often do you think we should send emails to our list?
3. What’s the best time of day/day of the week to send our emails?
This month we’ll tackle question one. Let’s begin by emphatically stating that there are no hard and fast rules regarding the answer to any one of these questions. The answer to question one depends upon how you define “response rate,” how the list was compiled, how it has been used /abused, how relevant the messages have been to the recipients, what performance baseline measurements exist, how many times a day/week/month/year the list has been mailed, what’s been the policy /practice re: subject lines and From addresses and about 100 other issues too numerous to list.
Currently there are no meaningful benchmarks that can be provided because there are simply too many variables at play. So unless you can definitively and accurately answer the question, “How long is a piece of string?” don’t expect anyone with any amount of integrity to answer what email response rate you should expect beyond, “It depends.” It’s simply not an answer that can easily be provided on a time-sensitive basis without performing due diligence and running a series of diagnostics.
In the future, measuring response rate will become a bit easier for those using an email service provider that has adopted the eec’s set of standardized metrics, known as the S.A.M.E Project. It will create a set of standardized email metrics that will create a common language and definition for metrics like response rate and make it easier to benchmark results. SubscriberMail will have complete adoption of the standardized metrics by December 2010. Stay tuned for part two!
Posted by Nic Winters on April 5th, 2010
In my most recent blog post I passed along some information related to splitting lists, a function usually performed in preparation for testing within your email marketing plan.
If the test you are hoping to perform is related to differences in subject lines, I would like to pass along a time saving tip. After creating your draft email message and saving it in your SubscriberMail account you then have the ability to quickly create and send the additional version of the message. Within the Message tab simply check off the box next to the draft message and hit the Copy button.
This will open a new page that allows you to specify your list selection and insert your updated subject line. This message can even be scheduled from that same screen (instead of walking through the step-by-step creation of another version of your original email).
And while you are thinking about subject lines – be sure to check out our white paper on words you shouldn’t include.
Posted by Bill Leming on June 22nd, 2009
When I think of “large companies” what comes to mind are multi-layered enterprises that typically consist of various organizational divisions each of which has its own set of management objectives, strategies and tactics. To the extent that each division has a specific need and a desire to promote its products and services via email with varying degrees of urgency, it quickly becomes apparent that there is a pressing need for centralized cat control. Without it chaos reigns.
This isn’t terribly different from brand management’s responsibility to create/identify the brand, communicate the rules surrounding its use, monitor how the rules are being followed and enforce compliance. More typical than not such an individual and, in some cases whole departments have clearly defined email responsibility within large companies. And, given the ability of any one division or group to pollute the pond for everyone else, well they should.
Like those responsible for brand compliance, it is up to the enterprise email manager to establish and communicate the email marketing rules to which everyone will adhere. These include who will be added to the opt-in list (and equally important, who will not), whether the company will use single or double opt-in, under what specific circumstances trade show attendees/verbal opt-ins will be added (or not added), how frequently the enterprise will communicate with any one individual within a given timeframe, what each and every email will include in terms of the subject line, from address not to mention graphic design and content and a myriad of other rules. All of these are designed to collectively ensure corporate compliance with both CAN SPAM and email best practices, as well as to create synergy and advance a collective voice through message uniformity and consistency.
So where does this control start? I think it starts with a corporate commitment to recognize the need and to assign both the responsibility and the authority for all email within the enterprise to one individual. Management has no difficulty doing so with respect to its brand largely because of the brand value itself, because that’s what brand managers have demanded and because doing so increases ROI. While each of these same reasons apply to email responsibility assignment within large companies, we can also add in that assigning ultimate responsibility for all email to one individual will also reduce the risk of getting blocked, blacklisted and/or sued.
At the risk of self-promotion, the channel solution we provide does more than simply help our clients monitor compliance with the rules they’ve established. By controlling images and content across all divisions and downward within divisions in fixed, locked-down formats it ensures email, brand and regulatory compliance. That in turn allows our clients more time to deal with their cats who, in true feline fashion, always want to test the bounds of email propriety, best practices and legislation.
We’ve all heard that if you’re in marketing, in particular email marketing, you should constantly be testing to maximize results. The most common test mentioned is the ubiquitous “A/B” split test, meaning a 50/50 list split to test one variable against another (graphics, copy, offer, layout, list, time of day, day of week, etc.).
