Testing Dataro across Australia for UNHCR direct mail appeals

Australia for UNHCR is the UN Refugee Agency’s national partner in Australia, raising funds and awareness to support displaced people across the globe. 

Direct mail appeals form a core part of Australia for UNHCR’s fundraising calendar. So in September 2021 the team embarked on a partnership with Dataro to seek better appeal returns using machine learning. 

After testing Dataro’s AI donor propensity software across four appeals, we found:

• Dataro’s predictions were highly accurate for Core value donors 
• Average appeal ROI from Core donors increased by 28%
• Average appeal costs for Core donors decreased by 23% 
• Average appeal net revenue from Core donors increased by 3%

Background

Dataro’s Direct Mail Appeals Module provides AI-generated predictive ranks and scores for warm supporters based on their likelihood of giving to a direct mail appeal. Australia for UNHCR tested these predictions in multiple campaigns including their 2021 ‘Winter Survival’ Christmas campaign, March 2022 ‘Ukraine’ campaign, Tax 2022 ‘Ukraine’ campaign, and September 2022 ‘Food Insecurity’ campaign. 

Testing for each campaign was run by the charity across their Core, Middle, and High Net Worth donor program areas. 

In order to validate the Direct Mail Appeal propensity model, the charity wanted to know:

1. Are the machine learning model predictions accurate for predicting appeal gifts? 
2. Is a higher ROI and Net Revenue result achieved when using Dataro’s AI targeting approach compared to segmentation alone?

Test 1 – Assessing Dataro’s donor prediction accuracy

To measure the accuracy of the Dataro predictions, the charity wanted to specifically know whether donors with a propensity score of 1% or more responded at higher rates than donors with a propensity score below 1%. 

In every campaign, the donors Dataro predicted were more likely to give to the appeal actually did give at higher rates, conclusively demonstrating the accuracy of the Dataro models. 

Amongst Core donors, which made up the vast majority of each appeal, response rates were on average 8.75 percentage points higher amongst the high propensity group (donors with a predictive score of greater than 1%) vs the low propensity group (donors with a predictive score of less than 1%). 

The same trend continued across the Middle and High Net Worth groups. Donors with a higher Dataro appeal score actually did give at a higher rate. However, as there are fewer donors in each of these cohorts, the results were less statistically significant.

Test 2 – Does Dataro’s AI targeting approach generate higher ROI and Net Revenue

In order to determine whether the accurate predictions translated into additional revenue, Australia for UNHCR considered two scenarios: 

1. ROI and Net Revenue from donors with appeal score greater than 1%
2. ROI and Net Revenue from donors selected via the traditional segmentation approach

Of course, there is a large overlap between the two cohorts as many donors with a high propensity score also fall into one of the segments the charity would normally include in their targeting. 

For Core donors in each campaign, the charity therefore compared the ‘Dataro selection’ against the ‘Australia for UNHCR’ selection to identify: 

1. Donors in the Dataro group only
2. Donors in both groups
3. Donors in the Australia for UNHCR group only

Across all four direct mail appeal campaigns, the results were clear:

Results for the Core donor cohort showed a consistently positive impact on net revenue and ROI potential of an appeal by using the Dataro Direct Mail appeal scores.

“By focusing on the higher propensity donors we will be able to target more effectively than by using our segmentation alone,” said Jo Purcell-Jones (Senior Business Intelligence Developer).

“In reducing mailing volume to high propensity donors only, we can make significant savings in costs, without jeopardizing revenue.”

Across the Middle and High Net Worth groups, the Dataro appeal propensities also resulted in a decrease in costs and an increase in ROI. However, the potential cost savings of removing low scoring Middle and High Net Worth donors from an appeal does not outweigh the value of what those donors give. 

“Whilst it is not valuable to use the scores to reduce mailing volumes for Middle (as with Core), the DM appeal model is still able to identify which donors are most likely to respond to an appeal, which we can potentially use to be more efficient in the stewardship of these donors,” said Jo. 

For improving mid-level and High Net Worth giving, Dataro recommends using additional propensities specifically developed to predict donors more likely to become mid-level and major donors. Their higher average gift amounts will typically mean it is worth including these donors in an appeal, even if their actual probability of giving right now is low.

In Summary

Based on these results, Australia for UNHCR is now only including Core donors with a Dataro DM Appeal score of greater than or equal to 1% in their direct mail appeal selections (or a lapsed donor score of greater than 1%). 

The charity will maintain mailing volumes for Middle donors and High Net Worth donors using segmentation, but will add in any additional donors with a Dataro DM Appeal score of greater than 1% to ensure they don’t miss any donations from these valuable cohorts. Australia for UNHCR is planning further testing of Dataro’s mid-level donor propensity model, and use of the DM appeal scores throughout the Middle and High Net portfolios will be reviewed periodically.