RESOLVED: Monzo Services Degraded/Outage - Card Payments may fail (16/01/18)


(Alex Sherwood) #62

Edit to add tweet.

To put today in perspective..

  • This incident caused an issue for me
  • This incident didn’t cause an issue for me

0 voters


#63

I cannot say because I do not 100% understand this stuff, others do though - but it was definitely the case :hugs:

training wheels are still on :speak_no_evil:


(Allie) #64

Exactly, Monzo are being open which I greatly appreciate. All cards decline randomly! Monzo is being honest with us and I love it!

Thanks! Looking on Twitter, this appears to relate to processing services provided by Verifone (they are in that business), not (semi-obviously) the hardware itself. I had no idea their processing services were this widely used! Interesting… thanks!


(Lawrence Ferguson) #66

Oh dear…the Starling Feedback thread best start preparing for a kicking…


(Brandon Billingham) #67

I’m sure it will all be explained later but couple things. I can see why NSQ would be used for non time sensitive parts of the system, notifications for example. Why is it impacting/used in Faster Payments or Card Payments?

Looking forward to the technical breakdown for sure :slight_smile:


(Aaron Bamford) #68

On a separate note completely unrelated to today’s issues, I’d be really interested to hear what’s the main benefit you get out of choosing NSQ over say RabbitMQ or Kafka?

Will you be providing a more detailed report of the events that lead to, and caused today’s issues? In the same way Oliver did here RESOLVED: Current account payments may fail - Major Outage (27/10/2017)

Thanks for keeping us updated with your findings! :slight_smile:


(Aaron Bamford) #69

Also interested in this. I’d hope payments aren’t being processed off the back of NSQ messages. And it they are, a good argument for doing so…


#70

I wonder if the Verifone issue is why contactless payments (including Apple Pay :angry:) weren’t working in Poundworld earlier…


(Allie) #71

Looking into Verifone a bit more, my best guess is that this is the service that went down:

If I’m right, I’m SHOCKED even chains as big as Waitrose are using it! I expected they’d have all that stuff in-house.


(Rika Raybould) #72

You’d be surprised at how many large merchants don’t control their card processing at all!


(Allie) #73

Which is probably a good thing… more stuff being in-house is part of why payments in the US are such a mess…

On the plus side, if so many are using their managed service it might mean we will start seeing newer terminals like the P200 and P400 sooner than I expected.

I did have a very strange dream last week, where I went into Tesco and they had replaced all their Ingenico iPP 350s with Verifone M400s (yes, M400, not P400). I was appalled. They looked terrible and bulky especially on self checkouts.

Anyone into dream interpretation wanna take a guess at the meaning of that one? LOL


(Richard Cook) #74

Hi everyone :wave:

I’ve marked this thread as Resolved as we’ve fixed the issues affecting payments, transfers, and top-ups. Feed items might take a while to catch up. But we’re on the case.

In the meantime, I’ll ask around to see if we can share any more information about what happened, and any changes we’re making to prevent it happening again.

Thanks.


(Brandon Billingham) #75

Thanks @cookywook. Will certainly be good to get a write up like last time as people have previously said.


#76

All seem to be working for me - Just paid for a coffee and for the notification and showed up in feed instantly.


(Peter Roberts) #77

All working here and thanks for the transparency. As a technology professional myself I accept that things happen and I hope we get a detailed technical write up like the great one previously given by @oliver

If you do publish that including what you’ve learnt from this then my confidence in Monzo will actually increase!


(Oliver Beattie) #78

Hi everyone, sorry about these issues. We will aim to share a post-mortem with you, but it might be a few days or next week until that’s possible. Hang tight :heart:


(Stephen Duerden) #79

As someone who worked Poundworld support (often on my own for a third party company) years ago, yes it would have caused issues. All of their terminals are Verifone. Those days were often very interesting to say the least as the problem was out of my hands, 150 plus stores and just myself trying to contain calls on a Saturday… fun!


(Frank) #80

I know this is marked as resolved but my partners card is still being declined. Are you aware of any other issues with card payments?


(Simon B) #81

Hi Frank, is everything working now? The issue was definitely resolved yesterday so there may be something else up with your partners card. Feel free to DM me with some info, or have your partner reach out directly via in-app chat :+1:


(Frank) #82

It was a message I got from her “card declined again” and no notification within the app. Will get her to start an in app chat.