Thanks for the nomination @tristan! (This post will likely be quite technical, however I will try to keep it informative for people who are not software engineers)
As @tristan mentioned there a couple of reasons why merchant feedback stopped working, one of the reasons was the way our tooling for reviewing all the feedback worked. Previously when reviewing a merchant we needed to be able to search the other merchants in case two merchants needed to be merged Back when we implemented the merchant review tool, we only had a few hundred merchants to review each month, and it was perfectly feasible to load all the merchants into memory. This worked reasonably well for quite a long time but as you can imagine it is not a very scalable solution and now that we have ~300k merchants in our database the page took a very long time to load
However that was not the main issue, once we migrated to our new platform (which runs all of our code on Kubernetes in containers) we started to see issues. This was due to the fact that the containers have what are called “resource quotas” which restrict how much of a computers resources (such as CPU and Memory) a given application can use. Some resources (such as memory) are what are called incompressible which means that if your application exceed the limits it will be terminated Now as you can imagine when we loaded the ~300k merchants into memory we quickly hit the new limits and caused our application to crash, luckily due to the micro-services architecture we use here at Monzo there was no impact to customers as the code that was causing issues was isolated from the rest of the system.
Now for the fix, I imagine many of you have been saying to yourselves “why didn’t you just use ElasticSearch??”, well you are completely right and that is what I have been working on for the past week. As of writing this post all of our merchant data is stored in both Cassandra (the single source of truth which is still used for powering your feeds) and ElasticSearch (which is updated asynchronously whenever a merchant changes). Now that all of the merchants are easily searchable it is trivial for our merchant review tooling to search for merchants without loading everything into memory and causing any of our services to crash
Sorry for the wall of text but I hope you found it interesting If you have any further questions just ask!