You pretty much covered all bases for awareness. I would just suggest the community is added to the process so should an issue happen again it’s covered.
It’ll remain in the Degraded Performance state until we receive the all clear from engineers. Understandably, they would like to be confident that everything is flowing normally and the graphs look healthy before we take the in-app message down.
I didn’t spot this and tried activating my fiancée’s new CA account/card. It got to the “memorise your PIN” stage then froze. I closed the app and reopened; now it says that Monzo are verifying a few details. I guess this counts as app issues for the tracker, but thought I’d highlight it just in case
If people would like a bit more detail about what went wrong, it seems NSQ got a bit backed up and whilst trying to clear this the issues propagated affecting other services.
Was the propagation because multiple things shared common NSQ infrastructure? I have no idea how NSQ scales upwards so sorry I can’t be more specific in my terminology
NSQ is a messaging platform that our systems use to communicate and handle tasks.
The investigation is at an early stage, so I’m sure we’ll know more once the engineering team have been able to look more closely at the flow of events
This is a little bit my fault for being in the USA right now and 8 hours behind!
It looks like everything is up and running again now
For clarity:
We had two separate bouts of platform instability today resulting in the app/card being unresponsive. This was due to a fix we’d attempted to apply to another issue we’d been seeing (delayed feed items) causing additional problems.
Very conveniently (sarcasm!) - Verifone had an issue today around the same time as well. They produce point-of-sale terminals for literally thousands of UK businesses, and they had an issue today meaning their terminals didn’t work for a while too.
As my colleagues mentioned above, we did proactively reach out as soon as possible across the app, status page and social, but it shouldn’t have taken us this long to get on the forum and communicate to you all. We’re sorry for this, and we will do better!
While I’m a huge Monzo fan and love everything about it (including the book recommendations from Mikey through the in-app chat) issues like this are part of the reason I can’t yet ditch my Barclays card. As someone who no longer carries cash with them and only carries my phone and my Monzo card, I edge more towards keeping my account with Barclays and just using my Monzo card as a spending card. While this means I’ll miss out on some features, reliability is way more important to me than extra functionality.
With that said, can we expect an awesome deep delve by Oliver (or another engineer) about what went wrong here and what will be done in the future to prevent an outage like this? As someone who is technical, posts like this give me way more confidence in Monzo than some page which tells me something isn’t working!