Every project development team faces challenges when it comes to credentials management: Who is responsible for generating secrets? Where and how are we storing them securely? How do we update keys when they change? How do we deploy an initial “secret zero” to a server for root encryption?
Different environments, likewise, call for differing approaches. Some distributed systems work well with distributed management systems. Other smaller systems might warrant keeping secrets inline with the code that uses them. Still other systems might leverage secure hardware and more sophisticated management tools.
All of these systems, however, involve one key element: cryptography. Storing secrets securely requires some form of cryptographic security. This past month, we’ve covered four different crypto tools for devops, how developers can use them, and potential drawbacks intrinsic to each approach.
Articles in this series:
Credstash
Credstash is a developer utility that leverages Amazon’s AWS infrastructure for both cryptography and storage. It encrypts secrets using Amazon’s Key Management Service (KMS) and stores them in a DynamoDB table for later retrieval. The two biggest advantages with Credstash are how it leverages Amazon’s IAM system for access control and how well it integrates with dynamic provisioning tools like Ansible.
The biggest drawback with Credstash is also, ironically, related to access management. In order to read data, a remote environment needs the same keys that allow it to potentially write data. This means a rogue actor with direct access to a provisioned server can, potentially, compromise the integrity of any secrets.
git-crypt
git-crypt is also a developer utility (read: command-line tool) that dynamically filters data stored in a Git repository to encrypt any sensitive data before it’s stored to disk. Unlike Credstash, git-crypt stores protected secrets alongside the code that uses them, possibly streamlining distribution. Another major advantage of git-crypt is how it leverages existing public-key infrastructure for controlling access to secrets.
The biggest drawback with git-crypt is, ironically, the fact that secrets are stored in a version control system. Revoking access to a secret is more involved than merely cutting out a user (or system) from the permission store. A rogue user who had permission at any one time still has access to the history of the encrypted secret! Revoking access to any one credential thus requires rolling that credential with whatever system uses it!
HashiCorp Vault
HashiCorp Vault is also a developer utility, but is meant for broader access of credentials throughout a development team. Along with passwords and API keys, Vault can also manage and create database credentials (and databases) and even SSH keys for server access. The biggest advantage of Vault is the vast flexibility of the system and the sheer amount of secret information it can manage.
Unfortunately, this broad scope leads to a great deal of complexity and is also Vault’s biggest drawback. It’s easy to configure storage or authentication backends that aren’t needed for long term use and neglectfully leave them enabled. Once available, these endpoints and interfaces could leak sensitive information to an unauthorized party. Also, while HashiCorp is making great strides in helping to solve the “secret zero” configuration problem, how the development team provisions their initial secret store is still a sticky problem.
Java KeyStore
The Java KeyStore is less of a developer utility and more of an abstraction for managing keys, secrets, and certificates within a Java or an Android application. The biggest advantage of the KeyStore is its ability to integrate with multiple storage systems. One application might handle credentials in memory, another application might store secrets to disk, and yet another might integrate with physical hardware that prevents the extraction of the sensitive data.
Like other secrets management systems, the biggest drawback of the Java KeyStore is in how to initially populate it with secrets. Many systems will require integrating with a separate tool for long-term storage and protection of secrets.
<Your Tool Here>
The frustrating reality is that a perfect crypto tool for devops does not exist. The tools above represent four discrete approaches to solve the same problem of managing secrets securely in an application or systems environment. Each has benefits. Every single one of them has critical weaknesses.
In practice, we use some mix of all of them in order to strengthen the environment.
An ideal tool would provide all of the benefit of the utilities above while also protecting the development team from the drawbacks.
Such a tool would allow for distributed, encrypted credentials with fine-grained access control locked down to individual authorized parties and systems. It would allow for the proactive management (i.e. revocation) of credentials access and automatically roll credentials in a secure, future-proof fashion. The tool would allow for both static secrets and more advanced credentials like leased SSH keys without increasing its attack surface.
Finally, such an ideal system would permit for the integration with external hardware systems and also simplify the way credentials are “bootstrapped” into a new environment.
TozStore is an end-to-end encrypted, cloud-hosted, secure data store. Developers can use this system to power everything from a personal data storage service to an encrypted filesystem; perhaps even an encrypted credential store. Perhaps you could leverage this tool to build the ideal system described above.