I’m Chris, and I’m a Data Science Manager here at Monzo.
In product data science we often run experiments to understand the impact of our changes. This blog post covers the approach for how we designed a regional experiment to measure the incremental impact of our referral scheme on new customer growth. In the blog I cover why we couldn’t use A/B testing and the considerations when working with geography data.
I hope you find it interesting! If you’ve got any questions, I’ll do what I can to answer them
I might’ve missed it but did you also do control v groups as a trend line, to really see if what you were doing was making a specific impact, at what stage?
@JIMMWX thanks for giving the blog a read . Our control region data was used to create a prediction for what we expected the test region trend to look like if we hadn’t launched our referral scheme. This predicted trend is what we compared our actual data against. The larger the difference between these trends the greater our impact and confidence that it was the referral scheme driving the change.
One thing that would interest me is how much cross-generational referral as opposed to peer group referral there has been.
I am in my 60s and was introduced to Monzo by my daughter (who had been attracted by one of the £5 referral offers) who is in her early 20s.
I honestly thought the £5 would be pretty much my sole interaction with Monzo; however, the functionality (and fun) is leading me towards Monzo being the main hub for my daily finances
Enough of my ramblings. Thank you again for the article.
@thehamshackman that’s a really interesting point. Analysis into our referral scheme shows a Cauchy distribution around 0 years difference between our existing (referrer) and new customer (referee). In the past year we found that ~50% of our referrals and between customers who are +/- 3 years different in age.
We see this higher similarity in other attributes beyond age. The observed behaviour refers to a sociology concept called homophily, whereby people have a tendency to associate with other people who are similar to themselves.
,