How our security team handle secrets

Security engineer @jack-monzo shares an insight into a new system we recently designed to manage secret information safely here at Monzo.

12 Likes

So… Algorithms and Codes…

My simpleton understanding of what I just read:

if(informationIsSecret) { 
dontShareIt(); 
}
9 Likes

Just store everything secret in /dev/null - people will have a hard time accessing it then :wink: even us… :sweat_smile:

5 Likes
passwords.txt

:point_up_2:

8 Likes

What happens if vault is sealed? Or being upgraded? Are deployments and autoscaling stopped?

We run multiple instances of Vault in High Availability Mode. This means that Vault remains available during upgrades or when there are issues with a particular instance. It also means that Vault can continue to operate even when some instances are sealed, provided that there is at least one unsealed instance online.

1 Like

Didn’t you tell me to change my Card PIN because you didn’t handle that secret? :confused:

How do you handle the secrets being leaked by Terraform in .tfstate files in plaintext? That would negate the security you introduced with Vault.

We don’t use terraform to inject any actual secret data. That would indeed be very bad. Secrets are injected with a go tool we wrote, shipper secret. We even encrypt the secrets with a public key, and then decrypt them inside of a Vault plugin for a second layer of security on top of HTTPS.

1 Like

If by key ceremony you mean unsealing, then you don’t need to do it every time you access vault. As long as you keep vault servers up and running, they will keep the decryption keys in memory. You will only need to unseal a vault server that has crashed, which they do but only rarely. Or did I misunderstand the reference?
One way to speed up the unsealing process is to keep the keys pgp encrypted (it’s a vault builtin feature), and store them somewhere where they can be easily fetched, such as an s3 bucket or even on devs laptops. The access don’t have to be guarded since all keys are encrypted.
Then you make a local script which fetches the key file, decrypts a key corresponding to the email of the person who runs the script, and send it to vault for unsealing. the person would only need to enter their password that encrypts their pgp keys. The entire process takes <2 mins.

We don’t mean unsealing, we are talking about either observing other people while they do things to Vault, to supervise, or in some cases generating root tokens, which requires multiple people collaborating.

The reason unsealing and generating root tokens takes a while for us is mainly planning, but also the fact that we require multiple people to decrypt things, no one person can do it alone, and the people also need to supervise each other.

1 Like