RESOLVED: Monzo Services Degraded/Outage - Card Payments may fail (16/01/18)

Which kind of proves my point.

Any card or card system could stop working for a time, so carrying an alternative is always sensible.

I’d advise everyone to have a credit card with them as well as a debit card when out and about.

I hope it all works out for you :crossed_fingers:

Verifone are also having problems right now, so that could be the issue here.

1 Like

Not sure! Was ASDA, does that help?

In fairness it is ASDA and I’ve had issues there with all sorts of card before.

You pretty much covered all bases for awareness. I would just suggest the community is added to the process so should an issue happen again it’s covered. :+1:t2:

2 Likes

My card worked all day :slight_smile: in Sainsbury’s, at the bakery too

Seems to be catching up now, got a notification for a transaction 30 minutes ago. :+1:t2:

Good to know but doesn’t the status page say the opposite right now?

1 Like

It’ll remain in the Degraded Performance state until we receive the all clear from engineers. Understandably, they would like to be confident that everything is flowing normally and the graphs look healthy before we take the in-app message down. :slightly_smiling_face:

1 Like

Doh :man_facepalming: I wasn’t clear. I’m referring to outbound payments appearing as operational

Edit: Rationale: false negative is more harmful than false positive (to a user)

Oops. :see_no_evil: Just fixed that, thanks for flagging! :pray:

I didn’t spot this and tried activating my fiancée’s new CA account/card. It got to the “memorise your PIN” stage then froze. I closed the app and reopened; now it says that Monzo are verifying a few details. I guess this counts as app issues for the tracker, but thought I’d highlight it just in case :slight_smile:

2 Likes

If people would like a bit more detail about what went wrong, it seems NSQ got a bit backed up and whilst trying to clear this the issues propagated affecting other services.

1 Like

Hmm :thinking: NSQ I knew it :grin:

What is NSQ

3 Likes

Was the propagation because multiple things shared common NSQ infrastructure? I have no idea how NSQ scales upwards so sorry I can’t be more specific in my terminology :man_shrugging:

It’s a platform that provides messaging as a service. So you build services connected through it

Edit: there are loads of trendy queing / messaging things these days. I know a little about rabbitmq which implements AMQP. I bet NSQ does that too

NSQ is a messaging platform that our systems use to communicate and handle tasks.

The investigation is at an early stage, so I’m sure we’ll know more once the engineering team have been able to look more closely at the flow of events :slight_smile:

1 Like

This is a little bit my fault for being in the USA right now and 8 hours behind!

It looks like everything is up and running again now :grinning:

For clarity:

We had two separate bouts of platform instability today resulting in the app/card being unresponsive. This was due to a fix we’d attempted to apply to another issue we’d been seeing (delayed feed items) causing additional problems.

Very conveniently (sarcasm!) - Verifone had an issue today around the same time as well. They produce point-of-sale terminals for literally thousands of UK businesses, and they had an issue today meaning their terminals didn’t work for a while too.

As my colleagues mentioned above, we did proactively reach out as soon as possible across the app, status page and social, but it shouldn’t have taken us this long to get on the forum and communicate to you all. We’re sorry for this, and we will do better!

19 Likes

Screenshot_20180116-202416__01

While I’m a huge Monzo fan and love everything about it (including the book recommendations from Mikey through the in-app chat) issues like this are part of the reason I can’t yet ditch my Barclays card. As someone who no longer carries cash with them and only carries my phone and my Monzo card, I edge more towards keeping my account with Barclays and just using my Monzo card as a spending card. While this means I’ll miss out on some features, reliability is way more important to me than extra functionality.

With that said, can we expect an awesome deep delve by Oliver (or another engineer) about what went wrong here and what will be done in the future to prevent an outage like this? As someone who is technical, posts like this give me way more confidence in Monzo than some page which tells me something isn’t working! :slight_smile:

2 Likes

Out of curiosity, how can Verifone have transient issues, since they make the hardware - they aren’t the processor?

2 Likes