Security engineer @jack-monzo shares an insight into a new system we recently designed to manage secret information safely here at Monzo.
Soā¦ Algorithms and Codesā¦
My simpleton understanding of what I just read:
if(informationIsSecret) {
dontShareIt();
}
12 Likes
Just store everything secret in /dev/null - people will have a hard time accessing it then 
even usā¦ 
5 Likes
passwords.txt
10 Likes
What happens if vault is sealed? Or being upgraded? Are deployments and autoscaling stopped?
We run multiple instances of Vault in High Availability Mode. This means that Vault remains available during upgrades or when there are issues with a particular instance. It also means that Vault can continue to operate even when some instances are sealed, provided that there is at least one unsealed instance online.
1 Like
Didnāt you tell me to change my Card PIN because you didnāt handle that secret? 
How do you handle the secrets being leaked by Terraform in .tfstate files in plaintext? That would negate the security you introduced with Vault.
We donāt use terraform to inject any actual secret data. That would indeed be very bad. Secrets are injected with a go tool we wrote, shipper secret. We even encrypt the secrets with a public key, and then decrypt them inside of a Vault plugin for a second layer of security on top of HTTPS.
1 Like
If by key ceremony you mean unsealing, then you donāt need to do it every time you access vault. As long as you keep vault servers up and running, they will keep the decryption keys in memory. You will only need to unseal a vault server that has crashed, which they do but only rarely. Or did I misunderstand the reference?
One way to speed up the unsealing process is to keep the keys pgp encrypted (itās a vault builtin feature), and store them somewhere where they can be easily fetched, such as an s3 bucket or even on devs laptops. The access donāt have to be guarded since all keys are encrypted.
Then you make a local script which fetches the key file, decrypts a key corresponding to the email of the person who runs the script, and send it to vault for unsealing. the person would only need to enter their password that encrypts their pgp keys. The entire process takes <2 mins.
We donāt mean unsealing, we are talking about either observing other people while they do things to Vault, to supervise, or in some cases generating root tokens, which requires multiple people collaborating.
The reason unsealing and generating root tokens takes a while for us is mainly planning, but also the fact that we require multiple people to decrypt things, no one person can do it alone, and the people also need to supervise each other.
1 Like
āThe secrets were readable to anyone with access to our etcd nodes.ā
Thereās no real timeframe for the observations in the article but etcd, secret encryption at rest has been about a while /tasks/administer-cluster/encrypt-data/ though sadly utterly useless in the context of EKS/AKS/GoogleK8 as it requires the extra flag setting on the api.
Did you consider https://github.com/banzaicloud/bank-vaults for handling vault or bitnamiās kubeseal for encrypting secrets in the cluster https://engineering.bitnami.com/articles/sealed-secrets.html ?
Good read; always interesting to see how someone else managed the āturtles all the way downā security domain.
2 Likes
Hi Christopher. Welcome to the community!
1 Like
Youāre probably being facetious but that is what they do just with libraries and wrappers to handle encryption/permissions/auditing. Thereās nothing wrong with text.
Well it was just a simple joke suggesting they kept all the secrets in plain text in a single file called passwords.txt but itās less funny if you explain it.
1 Like
Thatās why I said āyouāre probably being facetiousā but the joke relies on a misconception about computer data.
@jack-monzo Really awesome blog post, very informative. Do you ever plan on open-sourcing shipper secret like you do with some of your other tools?