We had issues with Monzo on 29th July. Here's what happened, and what we did to fix it

Monzo Chat News & Updates
44

Spanner is much older than that (see https://www.zdnet.com/article/google-reveals-spanner-the-database-tech-that-can-span-the-planet/). The relevant paper is from 2012. It’s the cloud offering that’s new but that’s because GCP is new.

Spanner offers the things that I already mentioned: scalability (size and performance), global replication (when and how it is needed) and no needs for upgrades. It is also quite proven with multiple Google products (with lots of traffic) relying on it.

If there are fewer needs (e.g. plain column storage) you can also use Bigtable (or Datastore or something equivalent from AWS), which I believe offers what Cassandra does but doesn’t offer global replication (at least not equally good) nor an SQL interface. But spanner is the future proof approach for when you end up needing global replication.

Disclaimer: These are personal opinions. I’m not advertising for my employer.

45

Just wanted to add here that I thought the blog post was great. Really informative easy to follow.

I really love that Monzo are prepared to share this information. It makes me feel much more connected. This level of transparency when things have not gone to plan, is really appreciated. It was interesting to see the timeline of events too.

As someone who works in IT, dealing with clusters also, I can appreciate the type of work involved here and it was great to get an insight in how Monzo works.

I shared the blog post around to other members in my team at work (most of whom I’ve had sign up to Monzo :slight_smile: ) It was a great example of how to communicate these things well.

Good job - I feel proud as a customer and an investor. :smile:

4 Likes
46

Like how well explained that is. though I may of gone cross-eyed at one point lol

47

Why are user PINs not stored hashed? I guess there are so few permutations that it would be easy to reverse checking every combination.

48

Yes, essentially. There are only 10,000 possible hashes, iterating through them is pure triviality.

1 Like
49

Wow, it’s great to see a company such as Monzo being so transparent! Love the amount of detail you go into. It pleases my technical hunger :smiley:

3 Likes
50

Hi Ryan & welcome :wave:
(You’ve just got your Welcome badge too :wink:)

51

First emoji badge too :wink:

2 Likes
52

Ryan earned that one himself, in his first post.
Someone else has to trigger the welcome badge :+1:

2 Likes
53

Cheers guys! Gotta collect those badges :wink:

1 Like
54

Gotta catch 'em all :wink: I got to level 2 this morning :tada:

1 Like
55

Let’s celebrate :partying_face: Beers are on you? :beers: :laughing:

1 Like
56

Only if you spill them :laughing:

57

Love these technical write ups Monzo, keep it up!

1 Like
58

This is one of the greatest root cause analyses which I have ever seen. Amazing work.
Though this one correction might be required: The figure right after 14:13 time stamp looks incorrect to me. Legend for “I’d like the data for” should be the “white box” instead of the pink one.

50%20PM
Screen Shot 2019-08-16 at 1.11.50 PM.jpg1350×762 115 KB

59

Now I think that legend is correct. Those new Cassandra nodes were indicating that they were the holders of the data, but had not been given the actual data

3 Likes
60

Welcome to the community forum Arpit. Glad you liked the blog. Agree that it was well explained

61

Thanks Michael. Actually I also thought the same at first, but then found the figure to be a bit misleading.

62

It probably could be more detailed to be clearer, but then a lot of people who read it said they were toughing it through some bits. Hard to balance. It was another excellent bit of technical writing by @anon61228674 and friends, but I hope for his sake and ours that he is spared more practice for a good long time!

2 Likes
63

Me too! :slightly_smiling_face:

3 Likes