Measuring the Real Results of Email Marketing

An important statistic for email marketing is that it has a higher return on investment than other direct marketing alternatives.

Which is why ROI gets mentioned a lot. However, this has both positives and negatives. Positive because it's an easy to understand concept to communicate email marketing's value. Negatives because ROI is not the only, or best measure of a successful email campaign.

How can you measure the true real value of your email marketing?

One person who is of help to us  is Kevin Hillstrom, President of MineThatData, a consultancy helping CEOs understand the complex relationship between customers, advertising, products, brands, and channels.

His longtime experience with database marketing and analytics makes him one of the internet's leading voices on evaluating and optimizing various marketing channels for profitability. He's also not afraid to speak up against accepted wisdom when that accepted wisdom is incorrect.

Here are some of his answers to some important questions:

Has the focus on email ROI led us astray?

"For instance, in non-branded paid search, you may generate 100 clicks, spending $50, to generate one order for $100. In catalog marketing, you may spend $50 sending catalogs to generate two orders for $100. In email marketing, you may spend a dollar sending 700 emails to generate one order for $100."

"So email truly does have the best return on investment, but only because there is so little cost."

So the relative value of email marketing changes when you look at other measures of financial success:

"In our example, catalog marketing would generate $1,400 sales for every 700 catalogs mailed. Non-branded paid search would generate $700 for every seven hundred clicks. Email marketing would generate $100 for every 700 emails sent."

"So, sure, email generates a great return on investment. But if your objective is to grow top-line sales, email fails miserably."

So, he says, the "best" channel depends on your goals. He adds:

"...you'll find in this example that catalog, with a lower return on investment, generates more profit dollars...email has the best ROI, but has limited potential to drive huge sales and profit increases."

It seems likely that high ROI has led us to be satisfied with low email conversions and we would do well to change our attitudes. As Simms Jenkins noted in a recent article, we need to build..

"...the business case and models that demonstrate the significant impact email can make on the bottom line."

Are we capturing email driven response properly?

But what about calculating email driven responses? Is it enough to simply track sales or downloads that follow an email click?

Kevin says retailers in particular frequently understate the value of email marketing campaigns because they focus so much on open rates, clickthrough rates, and conversion rates: all metrics that are ecommerce centric:

"The metrics fail to capture incremental sales that happen in retail stores after an email campaign occurs."

So how do we measure the true impact of email marketing?

Kevin recommends email marketers execute mail and holdout samples in every email campaign. He explains:

"Just take 10% of your email file, and split it into two equal groups. The first group receives your email, the second group does not. In the week following delivery of the email campaign, sum total sales across all channels in each group."

"Then compare the difference between the group that was mailed, and the group that was not mailed. This difference is the true impact of email marketing. This answer is more accurate than what you see via traditional metrics."

This, of course highlights the advantages of cross-channel customer data and a holistic view of the customer. Without data on multichannel responses, a proper analysis of incremental benefits from any one channel can't be done.

But what if that kind of integrated customer information isn't available? Kevin suggests focusing on one thing at a time:

"Everybody tries to solve something complex, when they can focus on simple issues. You don't need a doctorate degree in meteorology to know that when dark clouds are forming, rain becomes more likely."

"Similarly, don't set out to try and understand the combined impact of social media and mobile marketing and email marketing and paid search and natural search and direct marketing and television advertising and banner advertising and retargeting campaigns all at the same time."

He recommends doing your email campaigns and holdout tests and then measuring the impact on individual channels:

"For instance, you might find that paid search demand increases as email marketing frequency increases. This means that your email marketing campaigns drive customers to Google to do comparison shopping."

"If this is the case, then you need to manage your paid search budget in partnership with email frequency, because the two channels are interacting with each other."

What is the main advantage of mail holdout tests?

The mail and holdout approach has the benefit of measuring all the other impacts of email that don't get picked up by simply seeing if people buy/download after clicking an email link.

An example is the one cited earlier: offline sales resulting from an email campaign.

Perhaps the biggest benefit is that it eliminates those responses that would also happen without email. Kevin cites the example of online retail:

"A customer was going to order on Thursday, and then you mail them a campaign with free shipping and 25% off on Wednesday. The customer isn't dumb, she's going to use the promotion..."

"Email gets credit for this order, when in reality, the order would happen anyway."

Incrementality is the key point . . .

The crucial point is that the value of your emails isn't in the total responses measured directly for the email, but in the incremental responses generated by email across all channels. Kevin notes:

"This is a concept that most marketers fail to consider. Catalogers call this "incrementality". Each marketing activity should be measured based on the incremental sales it drives, not on the sales that modern metrics attribute to email marketing."

He notes that the accurate measurements obtained through mail/holdout tests can often come as a surprise:

"Most of the time, the results of the mail/holdout test are significantly different than what is observed via open/click/convert metrics. Because the email marketer is used to seeing traditional metrics, often 100 times a year for a decade, it is a surprise to see data that looks so different."

"The email marketer will doubt the new methodology, even if it is far more accurate, because the new methodology doesn't tie out with a thousand different campaigns compiled over the course of a decade."

"Traditional direct marketers know that mail/holdout methodologies are more accurate. Email marketers have an opportunity to adopt this style of measurement."

Mail and holdout groups also reveal segmentation, offer and frequency insights . . .

Since you're now measuring the true multichannel impact of email, this approach is perfect for testing key elements of your email efforts. For example:

"Mail and holdout groups, if segmented properly, will also help you understand that a small minority of your email list, maybe 10% to 20%, account for almost all of your incremental sales."

Kevin also strongly urges that retailers test email marketing without promotions:

"Too often, email marketing 'works' because the marketer gives away gross margin to beg for an order, offering 25% off and free shipping. Test full priced campaigns with no free shipping and no promotions."

"If you see that your traditional metrics implode, then you know that you have trained the customer to use email marketing as the way to get deals."

"If you've done this, you've done your business a disservice, and you've hurt the potential of the email channel to generate sales on its own merits."

In addition, he adds:

"If you really want to evaluate the impact of a single channel like email, then have the courage to execute a test for six months."

"Sample 20% of your email file, and split it into four groups. The first group gets every message, the second group receives 67% of email marketing messages, the third group receives 33% of email marketing messages, and the fourth group receives no email marketing messages."

"Measure the performance of each segment for six full months. Now you will see the true impact of email marketing."

"You'll also get to see the true impact of email marketing frequency on total sales, as well as on opt-out rates. In short, you'll learn a ton about what impact email marketing truly has."