As part of an ongoing series, we’re taking a deep dive into the structure, use, and benefits of various crypto tools for devops.
The trickiest part of any secrets management system is using it consistently. Developers often will write a credential to a temporary file, forget about it, and commit the secret into the repository. In other situations, they might re-use credentials or simply use a password too weak to provide any real security.
Credential security is only meaningful security if its implemented properly and managed consistently. This involves auditing credentials to revoke leaked or aging keys and managing access when developers or systems come online or go offline. Keeping secret data secret is more complicated than just using encryption; this is where tools like Vault by HashiCorp come in.
HashiCorp Vault
Unlike Credstash or git-crypt, HashiCorp Vault is a tool for managing much more than simple secrets. Vault also supports revoking keys, regenerating credentials, auditing access, and “leasing” credentials for a limited period of time.
In addition, Vault supports the creation of credentials, directly in protected resources. In practice, this means Vault can both manage and create AWS credentials, SQL databases and passwords, SSH keys, and more.
Like the other tools we’ve covered thus far, Vault is completely open source. You can host it on your own infrastructure and completely manage both the environment and the access your team has to it.
DevOps Use Case
HashiCorp Vault is a veritable Swiss army knife for credential management for DevOps. Vault can manage credentials for AWS resources, HashiCorp Consul, RabbitMQ, and various database engines. It can also manage SSH credentials, in modes that allow for centrally-signed keys, dynamically leased (and _expiring_) credentials, or even keys paired with a one-time-use password.
Vault stores all credentials encrypted; They’re safely protected and must be unlocked on demand, either by developers or consuming services. To unlock a credential, a developer or service needs to authenticate to Vault using one of its configured Auth backends (i.e. an EC2 instance role, a TLS certificate, or a username/password pair).
Finally, Vault logs every request to the server internally. Managers can generate audit logs to track when team members create, change, or use credentials. For simplicity, Vault ships with several backends to power auditing. Managers can view files directly or stream logs to a centralized service like Datadog.
The Good and the Bad
The biggest advantage of Vault is also its biggest drawback: complexity. Vault is a massive project that supports various flexibility in terms of implementation. You can secure multiple forms of secrets, enable various differing authentication workflows, and integrate with disparate storage backends. On the one hand, this gives you the room to build a system that works perfectly for your team.
On the other hand, it means you’re only ever using a subset of the features available with the tool. Even if you never invoke all of Vault’s functionality, those other backends, interfaces, and access points still exist. A misconfiguration on your end could easily result in exposing these unused elements of Vault to people who should not have access.
Additionally, Vault has the same fundamental weakness as do other crypto management tools: you need a way to ship “secret zero” to the server to bootstrap configuration. Vault does make some attempts to solve this issue through “AppRole” authentication, but that approach is not necessarily a fit for environments necessitating strong secrecy.
Further Reading
If you’re familiar with HashiCorp, you know they distribute several tools all aimed at making devops work easier. Vagrant supports local development. Consul supports service discovery. Terraform supports automated resource provisioning in the cloud. Vault is another component in a vast ecosystem that integrates well with existing HashiCorp tools. If you’re already using their platform and utilities, use HashiCorp Vault.
Depending on where you use Vault, there might be other options available as well. Ansible also has a product called Vault that can be use for distributing secrets during server provisioning. Docker has several options for containerized credentials management – Docker Swarm has its own secrets infrastructure, as does Kubernetes.
Ultimately, the choice of the tool depends entirely on the application you’ve built and the environment into which you’ve deployed. To that end, next week we’ll talk about less-connected applications and cover how secrets are managed locally within the Java Keystore.