The Nonprofit Guide to Testing and Optimization

“Creative experimentation propels our culture forward. That our stories of innovation tend to glorify the breakthroughs and edit out all the experimental mistakes doesn’t mean that mistakes play a trivial role. As any artist or scientist knows, without some protected, even sacred space for mistakes, innovation would cease.” – Evgeny Morozov

Our final blog post in our series on innovation focuses on the critical topic of experimentation and testing. Many nonprofit marketers argue that the inability to test and measure results is possibly the biggest problem facing today’s nonprofits. Many nonprofits seem incapable of setting clear, testable goals and then creating experiments that validate their hypotheses. As a result, organizations waste thousands of hours on activities that don’t directly impact the cause, or the impact they make isn’t measurable.

There are several ways to address this problem, but the basic testing principles used by startups is a great place to start. Most startups use this simple framework for testing and measuring results that can be applied at any level of a nonprofit. By using this 4-step process, nonprofits can quickly test and validate new ideas – or, just as importantly, validate and optimize existing initiatives.

Step 1: Hypothesis

Start by making a clear hypothesis (a best guess at solving a problem). The hypothesis can address both big and small problems at the organization.

For example, a hypothesis might be, “we think that unemployment in Charlotte is the number one cause of increased incarceration,” Or, “we think that sending paper receipts doesn’t create a positive financial ROI for our organization.”

Step 2: MVP

A minimum viable product (MVP) is a development technique in which a new product is developed with sufficient features to satisfy early adopters. It is the minimum amount of work that needs to be done to test a hypothesis and determine whether a concept/product will work or not. Once the hypothesis is clear, the MVP will help validate it with the smallest amount of time and money possible.

For example, “let’s set aside 10% of our donors over the next 4 months to get digital-only receipts with similar copy, then measure the ROI (cost vs. revenue) for those donors over the period.”

Step 3: Test

Set up a valid test and validate results.

Testing is simply a formal method for gathering data and validating a hypothesis. To create a valid test, consider the following:

  •      Make sure to get a statistically significant set of data points (typically over 30). (e.g. 30 donors who stop getting paper receipts)
  •      Create an A/B split test to compare the current solution against the new solution. A/B testing is a way to isolate and test ONLY the hypothesis while avoiding other variables.
  •      Be very skeptical of soft, personal stories. Try to rely on hard data over the opinion of a few.

Step 4: Pivot

Fail fast!!! On a recent Responsive Fundraising Podcast, Jon Burgess encouraged our listeners to reward failure. Yes, you heard that right, REWARD failure. Unless your team feels the freedom to fail you can never truly innovate. Fear of failure will always cause your team to drift to consensus thinking and fall back to “tried and true” methods of the past rather than taking risks and pushing toward innovation.

Most leading startups would rather prove a hypothesis wrong right away than meander in the wrong direction for years. A pivot is just a change in direction based on what has been learned. Getting clarity when it’s time to make a change can be harder than you think. In fact, I’ve seen nonprofits knowingly walk down the wrong path for months without clarity on why they should change. Testing and data-driven analysis allows a nonprofit to learn what’s wrong quickly so that it doesn’t keep making the same mistakes repeatedly. When data either validates or invalidates your hypothesis, then it’s time to be brave. Pivot!

Nonprofit Tech Ecosystem Mapping

Nonprofit Tech Ecosystem Mapping

If you’re thinking about making a technology change or even simply want to use your existing technology better, there’s one thing you need to do before you take action. Is…
5 Tips For Better Change Management 

5 Tips For Better Change Management 

A lot of nonprofit folks are hesitant to change technology, but it’s not for the reasons you think. It’s not because they love their current systems. It’s not because they…

Grow generosity with Virtuous.

Virtuous is the responsive fundraising software platform proven to help nonprofit organizations increase generosity by serving all donors personally, no matter their gift size.

“Virtuous truly understands nonprofits and the importance of our mission. And their open access to data and built-in custom reports gave us access to the data we need.”
Todd Shinabarger
Chief Information Officer