Going Beyond A/B: Three Long-Term Tests to Refine Email Marketing Performance

Whether your nonprofit is trying to fix a “problem” with donor engagement or you’re just committed to innovation and never satisfied with the status quo, A/B and multivariate testing is probably at the top of your list to improve communication performance. If you aren’t doing a/b testing yet, check out our previous post Getting Started with A/B Testing and this post from KissMetrics.

A/B testing can result in dramatic performance improvements. MailChimp published aggregated testing results from their customers on their blog post A/B Split Testing – Does it Help. When combined, the 11% increase in open rates and 17% increase in click-through rates would have a huge impact on revenue if used throughout the course of an entire year.

But what if you want to know how email campaigns boosted results in other channels? What if you need to know how much time and effort your team needs to dedicate to capturing an email address from existing supporters?

There are dozens of questions you could ask yourself that a simple, one-time A/B test isn’t going to give you the information you need. Fortunately, you don’t have to guess at the best approach.

Enter long-term testing.

As you’ve probably guessed, long-term testing is a series of consistent split tests over the course of weeks or months to answer big, broad questions.

Here are just a few long-term tests to email communications you should consider implementing to improve marketing performance.

Email Frequency

If I’ve heard this once from someone working at a nonprofit, I’ve heard it a dozen times: “I’d unsubscribe if you sent me X emails per month.”

Great! Now what? Am I supposed to base our entire email marketing communication calendar on your subjective opinion (that might not be the same as our target market)?

The sad truth is, many nonprofits take this anecdotal feedback and build their strategy around that one — often times, loud — voice. What we should do is let our donors tell us what the right frequency is based upon their behavior.

Setting up the test

Split your list into two equally sized groups. Group A — the control group — should continue to get email updates at whatever frequency you’re currently sending; group B should get emails twice as frequently. If you want to take it a step further, added in a third group that gets emails at half the frequency.

Testing timeline

The timeline for this test could vary greatly depending on your list size and your current email frequency. Beyond just measuring if a test is statistically valid, you need to give this test enough runway to leave an impression on its recipients. In my experience four to six months should start to give you a picture of how email frequency is affecting their behavior across all marketing channels.

Metrics to consider when determining a winner

This type of test could have effects on all manner of marketing efforts. It’s possible more frequent email communications could keep your organization’s cause top-of-mind and boost response in other channels. On the other side of the coin, more frequent emails could be burdensome on supporters resulting in an increase in unsubscribes — and you won’t know until you perform a statistically valid test.

With this test, make sure you’re getting a holistic view into donor behavior. Don’t just look at email response rates, email revenue, and open and unsubscribe rates. Look at the donor’s total response across all channels. When looking at a donor’s giving across all channels, make sure you only look at their giving for the period of time testing period.

Email Length

Ahhhh… it’s the age-old question. As a statistics and email marketing nerd, I can really geek-out on this one.

What’s better: a longer email with all the information a supporter needs to make a donation vs. a short email with a link to a landing page the drives supporters to make a donation?

Setting up the Test

Just like the email frequency test, you’ll want to split your email list in two by segment. Keep all your email recipients in consistent test groups from email to email and wait for the test to become statistically valid.

Testing Timeline

You won’t know what approach makes the most sense for your organization unless you test it. And unless you test this over a number of email drops, it will be a little difficult to come to any definitive conclusion. There are just too many variables and outlier responses that could skew your testing results if you tried to make an assumption based upon a single email or two.

Unlike the email frequency test, you can probably get valid results on this test in a matter of weeks… not months. Though it’s possible this test could have an impact on results in other channels, it’s less likely.

Measuring Results

Because this test isn’t as likely to affect results in other channels, you’re safe focusing your attention on email-specific statistics: open rates, click-through rates, social sharing, donations… and the list goes on.

Pull a list of email stats by email by test group. You should be able to total the results from each group and do a quick comparison on what the right approach is moving forward. Download a sample report I put together here.

Email Design

As a former Creative Director, I LOVE beautifully crafted communications. I love the subtlety of typography and original photography captured for a specific purpose. It doesn’t matter if it’s an email, direct mail piece, landing page, pamphlet, or conference PowerPoint: specific, unique communications just get me all jazzed-up.

But……… I wish I could say subtle design was always the driving force behind my most successful email marketing campaigns.

The truth is: the most successful email I’ve ever sent to a nonprofit supporters had zero images, used the font courier new and links to donate weren’t placed within carefully crafted language to get a recipient to click. They were bright blue and read something like “visit {http://here-is-a-really-long-url.com/some-path/finally-a-page/to-donate.aspx?cid=8362/} to make a donation.”


To be clear, my “ugly” email only worked in VERY specific scenarios and wasn’t a tactic I used on a regular basis. I did however perform a long-term test on slick highly-designed emails vs a more simple, straight-forward design — and the simple design won out.

Setting up the test

Not to be a broken record… but let this test run for a while. Don’t let seasonality or strength of offer alter your view into the success of this test.

Perform this test over a number of email campaigns to normalize any spikes in giving and outliers in the response data. You can expect to start to get a picture of which style your donors prefer after sending just a handful of emails.

Measuring results

Be brutal. It might be painful, but take off your Creative Director hat off and let the results speak for themselves. If less stylized trumps beautiful design, so be it.

Just like with the email length test, aggregate your email statistics by testing group and do a quick comparison.

Parting Thoughts

Testing is essential. A/B and multivariate testing can dramatically improve results on any given email going out next Tuesday. But you could be missing out on having an even greater impact long-term if you don’t start to answer some of the broader questions that have more than likely have been pestering you over the years.

If you cringed a little each time I used the phrase “statistically valid,” don’t worry: I’ve got your back. Use this handy test validity calculator from VWO to make sure you’re not improperly make decisions on results that don’t truly say anything.

I promise: it’s not as scary as it seems. If you’re committed to continuous improvement, take a closer look at how these long-term tests might help you more effectively raise support for your mission.

agile planning spreadsheet download

Fundraising insights you won’t delete.
Delivered to your inbox weekly.
How To Involve Your Board In Year-End Fundraising

How To Involve Your Board In Year-End Fundraising

How do you engage your board at year-end? Your board members can be important partners in your year-end fundraising campaign. But how do you inspire them to participate? What’s the…
Declutter Your Data

Declutter Your Data

Do you have messy data? When you’re a busy nonprofit, keeping up with delivering programs and communicating with donors, staying on top of your data often comes last on your…

Grow generosity with Virtuous.

Virtuous is the responsive fundraising software platform proven to help nonprofit organizations increase generosity by serving all donors personally, no matter their gift size.

“Virtuous truly understands nonprofits and the importance of our mission. And their open access to data and built-in custom reports gave us access to the data we need.”
Todd Shinabarger​
Chief Information Officer