Proving You’re Wonderful!
Posted by Bill Leming on September 14th, 2009
A question came up last week regarding how to best measure the impact of a specific commercial email we recently deployed for a client and which, on the surface, appears to have produced nearly unbelievable positive results. While this endeavor is far more preferable than proving that you’re not guilty of lying, cheating or stealing, it’s nonetheless a challenge.
Anyone and everyone responsible for promoting the offer through the website, or via banner ads, social media, print and email wants to claim responsibility for the parentage of this genius idea and the attendant sales figures associated with it. So how do we “prove’ our numbers?
One suggestion that was offered was the idea of matching the email addresses of actual buyers against our email list to see how high or how low our match rate actually was. The hypothesis here was that a high match rate would indicate that there was a strong correlation between our list and those who bought and that either our email efforts must have been successful or that we had the ability to predict before deployment who was likely to buy.
Conversely, a low match rate would signify that our email efforts were less than successful. Without going into the various scenarios that might explain why this approach is less than logical and therefore less than irrefutable, suffice it to say that statisticians and most non-statistically oriented people will have problems with this methodology.
Another suggested solution put forth was to survey people and determine which one of the various media channels used actually caused the recipients to take an action. The problems with this are numerous. Like eyewitness testimony, it sounds great but is usually highly suspect. In addition if there were cumulative forces that stimulated the response, how does one measure that and how does one identify and weight each appropriately? It’s not easy and perhaps more importantly, it’s tough to sell.
Which brings me back to my old Psych 101 days and the simplest of stimulus-response models. Had we set aside a randomly selected and statistically sizeable/reliable Control Group of 5% or 10%, we could have at least quantified our results in terms of incremental revenues and incremental ROI. While doing so limits the opportunity, at least it would have told us that the Control Group’s actual purchase behavior was X% with an average sale amount of $Z and that the Mail Group’s corresponding response rate was Y% with an average sale amount of $W.
Subtracting one from the other would provide a highly reliable, easily understandable and statistically sound measure of the “incremental” units sold and the “incremental” revenue attributable to our email efforts. It would also have told us what we achieved (units sold/revenues generated) from all the other media, promotional channels combined during the period measured and at what cost. While it wouldn’t identify the contribution of each specific communication vehicle to the Control Group revenues, at least it would have quantified what our efforts achieved at what cost and in a way that is both easily understood and statistically sound. But you have to do it on the front end, not after-the-fact. Like most endeavors that are important, proving you’re wonderful has its requirements.
