RESOLVED: Monzo Services Degraded/Outage - Card Payments may fail (16/01/18)

You pretty much covered all bases for awareness. I would just suggest the community is added to the process so should an issue happen again it’s covered. :+1:t2:

My card worked all day :slight_smile: in Sainsbury’s, at the bakery too

Seems to be catching up now, got a notification for a transaction 30 minutes ago. :+1:t2:

Good to know but doesn’t the status page say the opposite right now?

It’ll remain in the Degraded Performance state until we receive the all clear from engineers. Understandably, they would like to be confident that everything is flowing normally and the graphs look healthy before we take the in-app message down. :slightly_smiling_face:

Doh :man_facepalming: I wasn’t clear. I’m referring to outbound payments appearing as operational

Edit: Rationale: false negative is more harmful than false positive (to a user)

Oops. :see_no_evil: Just fixed that, thanks for flagging! :pray:

I didn’t spot this and tried activating my fiancée’s new CA account/card. It got to the “memorise your PIN” stage then froze. I closed the app and reopened; now it says that Monzo are verifying a few details. I guess this counts as app issues for the tracker, but thought I’d highlight it just in case :slight_smile:

If people would like a bit more detail about what went wrong, it seems NSQ got a bit backed up and whilst trying to clear this the issues propagated affecting other services.

Hmm :thinking: NSQ I knew it :grin:

What is NSQ

Was the propagation because multiple things shared common NSQ infrastructure? I have no idea how NSQ scales upwards so sorry I can’t be more specific in my terminology :man_shrugging:

It’s a platform that provides messaging as a service. So you build services connected through it

Edit: there are loads of trendy queing / messaging things these days. I know a little about rabbitmq which implements AMQP. I bet NSQ does that too

NSQ is a messaging platform that our systems use to communicate and handle tasks.

The investigation is at an early stage, so I’m sure we’ll know more once the engineering team have been able to look more closely at the flow of events :slight_smile:

This is a little bit my fault for being in the USA right now and 8 hours behind!

It looks like everything is up and running again now :grinning:

For clarity:

We had two separate bouts of platform instability today resulting in the app/card being unresponsive. This was due to a fix we’d attempted to apply to another issue we’d been seeing (delayed feed items) causing additional problems.

Very conveniently (sarcasm!) - Verifone had an issue today around the same time as well. They produce point-of-sale terminals for literally thousands of UK businesses, and they had an issue today meaning their terminals didn’t work for a while too.

As my colleagues mentioned above, we did proactively reach out as soon as possible across the app, status page and social, but it shouldn’t have taken us this long to get on the forum and communicate to you all. We’re sorry for this, and we will do better!

Screenshot_20180116-202416__01

While I’m a huge Monzo fan and love everything about it (including the book recommendations from Mikey through the in-app chat) issues like this are part of the reason I can’t yet ditch my Barclays card. As someone who no longer carries cash with them and only carries my phone and my Monzo card, I edge more towards keeping my account with Barclays and just using my Monzo card as a spending card. While this means I’ll miss out on some features, reliability is way more important to me than extra functionality.

With that said, can we expect an awesome deep delve by Oliver (or another engineer) about what went wrong here and what will be done in the future to prevent an outage like this? As someone who is technical, posts like this give me way more confidence in Monzo than some page which tells me something isn’t working! :slight_smile:

Out of curiosity, how can Verifone have transient issues, since they make the hardware - they aren’t the processor?

Edit to add tweet.

To put today in perspective..

  • This incident caused an issue for me
  • This incident didn’t cause an issue for me

0 voters

I cannot say because I do not 100% understand this stuff, others do though - but it was definitely the case :hugs:

training wheels are still on :speak_no_evil:

Exactly, Monzo are being open which I greatly appreciate. All cards decline randomly! Monzo is being honest with us and I love it!

Thanks! Looking on Twitter, this appears to relate to processing services provided by Verifone (they are in that business), not (semi-obviously) the hardware itself. I had no idea their processing services were this widely used! Interesting… thanks!