How our security team handle secrets

Security engineer @jack-monzo shares an insight into a new system we recently designed to manage secret information safely here at Monzo.


So… Algorithms and Codes…

My simpleton understanding of what I just read:

if(informationIsSecret) { 

Just store everything secret in /dev/null - people will have a hard time accessing it then :wink: even us… :sweat_smile:




What happens if vault is sealed? Or being upgraded? Are deployments and autoscaling stopped?

We run multiple instances of Vault in High Availability Mode. This means that Vault remains available during upgrades or when there are issues with a particular instance. It also means that Vault can continue to operate even when some instances are sealed, provided that there is at least one unsealed instance online.

1 Like

Didn’t you tell me to change my Card PIN because you didn’t handle that secret? :confused:

How do you handle the secrets being leaked by Terraform in .tfstate files in plaintext? That would negate the security you introduced with Vault.

We don’t use terraform to inject any actual secret data. That would indeed be very bad. Secrets are injected with a go tool we wrote, shipper secret. We even encrypt the secrets with a public key, and then decrypt them inside of a Vault plugin for a second layer of security on top of HTTPS.

1 Like

If by key ceremony you mean unsealing, then you don’t need to do it every time you access vault. As long as you keep vault servers up and running, they will keep the decryption keys in memory. You will only need to unseal a vault server that has crashed, which they do but only rarely. Or did I misunderstand the reference?
One way to speed up the unsealing process is to keep the keys pgp encrypted (it’s a vault builtin feature), and store them somewhere where they can be easily fetched, such as an s3 bucket or even on devs laptops. The access don’t have to be guarded since all keys are encrypted.
Then you make a local script which fetches the key file, decrypts a key corresponding to the email of the person who runs the script, and send it to vault for unsealing. the person would only need to enter their password that encrypts their pgp keys. The entire process takes <2 mins.

We don’t mean unsealing, we are talking about either observing other people while they do things to Vault, to supervise, or in some cases generating root tokens, which requires multiple people collaborating.

The reason unsealing and generating root tokens takes a while for us is mainly planning, but also the fact that we require multiple people to decrypt things, no one person can do it alone, and the people also need to supervise each other.

1 Like

“The secrets were readable to anyone with access to our etcd nodes.”
There’s no real timeframe for the observations in the article but etcd, secret encryption at rest has been about a while /tasks/administer-cluster/encrypt-data/ though sadly utterly useless in the context of EKS/AKS/GoogleK8 as it requires the extra flag setting on the api.

Did you consider for handling vault or bitnami’s kubeseal for encrypting secrets in the cluster ?

Good read; always interesting to see how someone else managed the “turtles all the way down” security domain.


Hi Christopher. Welcome to the community!

1 Like

You’re probably being facetious but that is what they do just with libraries and wrappers to handle encryption/permissions/auditing. There’s nothing wrong with text.

Well it was just a simple joke suggesting they kept all the secrets in plain text in a single file called passwords.txt but it’s less funny if you explain it.

1 Like

That’s why I said “you’re probably being facetious” but the joke relies on a misconception about computer data.

@jack-monzo Really awesome blog post, very informative. Do you ever plan on open-sourcing shipper secret like you do with some of your other tools?