November/December 2004

Split List Testing in Today’s Wired World
Convio helps nonprofits improve email response through easy-to-conduct split list testing

Many direct marketers and fundraisers have either heard, read about, or conducted, split list testing (also called A/B testing), before sending a direct mail or email communication. It can be a powerful exercise to determine the best message and format for driving a higher response.

However, nonprofit professionals might think that the “rules” of split list testing are too complex and time-consuming for their organization, so they avoid using this highly effective practice. They shouldn’t. While the rules — covered later in this article — may seem daunting, employing the basic principles of split list testing and using the latest online email tools can help continuously improve response to email communications and online campaigns.

What is Split List Testing?
Split list testing is a methodology of testing variations of a direct mail or email communication to determine which is more effective. This process includes several traditional rules:

• Test only one element at a time, such as the subject line or whether you include a graphic, to determine exactly which variable element affects response rates.
• Use only statistically valid samples for a completely accurate representation of the testing universe.
• Include margin of error calculations when determining results.

These rules work well when an organization is sending direct mailers or emails in high volume and can designate several thousand contacts for test mailing, and still have the majority of its file available to receive the communication that got the highest response in the test. While this approach is not practical for nonprofits with smaller email address files, any nonprofit can use basic principles of this proven technique to improve email communications while keeping a close eye on time and money spent.

Split List Testing: A Practical Approach for Nonprofits
Any nonprofit can improve results for an email communication, and more importantly, all future email communications, by using tools and basic principles for split list testing. By testing an element or approach with each email, an organization can continuously refine its email communications for maximum response rates. 

Here are a few tips for using split list testing to improve email communications:

• Decide what result you want to test, for example, email open rates.
• Determine which elements in an email, when varied, might affect this result. For example: a personalized subject line might get a better open rate than a generic subject line.
• Check this hypothesis using an online email tool with split list testing capabilities to test the email message variations. Convio Email Marketing, for example, allows an organization to send two or more different emails to a percentage of the target audience to test the effectiveness of each message. An organization can easily send two or more different emails to a portion of its target audience to test the effectiveness of each message, receive responses quickly, compare response rates of the different messages, and then refine the email before sending it to the entire constituent base — ensuring maximum response for each email.
• Use the test results not only for the email at hand, but also to add to the organization’s knowledge base for future email messages. For example, once the organization knows that constituents tend to open messages with personalized subject lines at a higher rate, it should consider using personalized subject lines for future email communications.
• Continue to test different results. Also, consider re-testing results after several months to see if they have changed over time.

The Value of Split List Testing: A Real-World Example
The ASPCA tested the results of targeting its donor renewal email by varying subject lines and content within its messages for different recipient groups. Though it is a large organization, the ASPCA was able to execute this simple test with a relatively small portion of its overall email address file, confirming that a large email address file and a complex test structure is not necessary to gain valuable information from a split list test.

First, the organization created a list of its constituents known to be “dog people” and another list of “cat people.” These were members who have only dogs or only cats, rather than both. The ASPCA emailed half of each list — each a randomly-chosen group of several hundred members — with a targeted renewal reminder highlighting cats only for the cat people and dogs only for the dog people. Figure 1 shows the dog-only version of the renewal email sent to the dog people. To the other half of each list, the ASPCA sent a general renewal notice with pictures of both dogs and cats, with the subject line: “These pets are counting on you” (figure 2). 

Fig. 1:  Targeted renewal notice, specific to dog people.  Note the focus on dogs in the subject line, the photos and the first line of text within the body of the email.

Fig. 2: General renewal notice, showing equal focus on both dogs and cats.

The ASPCA measured email open rates, response rates and total membership dollars raised for each of the four groups. The results showed that open and response rates were considerably higher for people responding to the targeted — or, cat-only and dog-only — messages. Dog and cat people also were more likely to donate if they saw dog- or cat-specific content within their renewal messages — response rates were double in each case. Overall, the ASPCA raised 103 percent more in dollar revenues from the constituents responding to the targeted messages than those responding to the more general cat and dog messages.

This relatively simple campaign test demonstrates the power of today’s advanced online tools to significantly improve results when email messages are tailored to constituent preferences. Consequently, the ASPCA now plans to use targeted messages on an ongoing basis.

Have a colleague who might be interested in this topic? Why not forward this article?