vault-exfiltrate
is a proof-of-concept for extracting the AES master and session keys from a core dump of an unsealed Hashicorp Vault process.
$ vault-exfiltrate extract core_file keyring_file | tee keyring.json
{"MasterKey":"+fbdF5OpHqVzGZP2Odbelk1sp3mhKcWT72pEc+abmYU=","Keys":[{"Term":1,"Version":1,"Value":"niAuiofcqFWha0FQhrqNZOSraD3zwIoCAs6nkJomfJs=","InstallTime":"2017-08-02T18:08:19.799452092-04:00"}]}
$ vault-exfiltrate decrypt keyring.json logical/90828c10-fb92-12b8-78ca-a262f150b322/test_secret ciphertext_file
{"secret_name":"secret_value"}
$ vault-exfiltrate shares keyring.json 3
nKNTMsqtc9hHwbn9x+/asTmqDoY6YqDGWxsvrLBCpqvm
174ZxDuptyYnEY+ghwBjtN/LrVdterKitpNki3uBe2fI
q+H2Gxn4OKkprWH8exr3lvtKcr27KPFP93roWf8TpzvA
Its main purpose is to demonstrate the limitations of Vault's "two-man rule" threat model (and the Shamir secret-sharing scheme more generally) and inform discussion about potential hardening techniques for Vault.
Vault is software intended to centralize and unify handling of secret data across an organization's personnel and networked services. These aspects of its security model are relevant:
- Vault supports multiple datastores (e.g., MySQL, Zookeeper, and DynamoDB) for persisting long-lived secrets; all such secrets are stored under authenticated encryption (AES-GCM in the current implementation) with a session key.
- The session keys are stored in the same backend with the same algorithm, but with a different key: the master key. The datastore entry holding the encrypted session keys is called the keyring. Both master and session keys can be rotated.
- By default, the master key is split into multiple shares with the Shamir secret sharing scheme. Shares are then distributed among operators.
- When started, a Vault process is in the non-functional "sealed" state, meaning it has no access to the master key or the session keys. Operators "unseal" the process by inputting their shares, at which point the master key can be reconstructed and the session keys retrieved and decrypted. At this point, the Vault process has the master key and the session keys stored in memory.
I believe there is a contradiction between the following two claims:
- Vault's threat model explicitly excludes attacks based on memory analysis of an unsealed Vault process: "if an attacker is able to inspect the memory state of a running Vault instance then the confidentiality of data may be compromised."
- However, the documentation suggests that the use of the Shamir scheme provides protection against malfeasance by a single operator: "Vault supports using a Two-man rule for unsealing using Shamir's Secret Sharing technique [....] The risk of distributing the master key is that a single malicious actor with access to it can decrypt the entire Vault."
vault-exfiltrate
is intended to demonstrate that instead:
- In a standard Linux environment, it's straightforward for a malicious administrator (or an attacker with root-level access) to obtain the master and session keys.
- Most real-world deployments of the Shamir scheme implicitly require a trusted third party: the environment in which the secret is reconstructed, and which is then responsible for preventing the shareholders from stealing the secret.
vault-exfiltrate
has three modes. vault-exfiltrate extract
takes an ELF core file of the vault
process as its first argument, and a file containing the exact binary ciphertext of the keyring as its second argument. The best way to obtain the core file is with the gcore utility, which is part of gdb
; under the hood, it uses the ptrace(2) system call to obtain the memory contents of the target process. The keyring is stored at the path core/keyring
within Vault's logical key-value namespace; the method of retrieving the data will depend on the physical storage backend. For example, the file
storage backend stores the keyring at the relative filesystem path core/_keyring
, wrapped in JSON and base64; the zookeeper
backend stores it as the data of the core/_keyring
node; and the mysql
backend stores it in the table row with vault_key = 'core/keyring'
. If successful, it outputs the JSON plaintext of the keyring, including the master key and all active session keys.
vault-exfiltrate decrypt
takes three arguments: a file containing the JSON keyring plaintext produced by extract
, the logical path of an entry in the storage backend, and a file containing the exact binary ciphertext of the entry. If successful, it outputs the plaintext of the entry.
vault-exfiltrate shares
takes two arguments: a file containing the JSON keyring plaintext, and a threshold number of shares n. It outputs n new shares of the master key. This can be used as a key recovery tool: if the original shares of the master key have been lost, but an unsealed vault
process is still running, this tool can produce new shares. The shares can then be used to rotate the master key, or to generate a new root token, which provides unrestricted application-level access to Vault. (Note that for both of these use cases, the threshold number must agree with the number Vault was originally configured to use; vault rekey
can be used to change the threshold.)
Originally, I tried to use delve to retrieve the master key. However, delve core
had difficulty interpreting the core dumps produced by gcore
. Fortunately, testing candidate AES-GCM keys is very cheap; my hardware can perform approximately a million guesses per second. The approach implemented here is to enumerate all read-write regions in the core file, then try every 256-bit sequence aligned to a 64-bit boundary. This should take between seconds and tens of seconds in the typical case.
vault-exfiltrate
has a runtime dependency on the readelf
executable for parsing the core file headers. It would be nice to eliminate this.
Vault should not be used to protect long-lived secrets that cannot be rotated. This is hinted at in some Vault documentation, in particular for PKI: "Vault storage is secure, but not as secure as a piece of paper in a bank vault [....] If your root CA is hosted outside of Vault, don't put it in Vault as well; instead, issue a shorter-lived intermediate CA certificate and put this into Vault."
It is possible to harden Vault against this attack:
- By default, Go programs should not produce core dumps on crashes. However, it's conceivable that a bug or exploit in the Go runtime or in dynamically linked native libraries could result in a core dump being written to disk, at which point it could be exposed deliberately or accidentally. Vault can be prevented from dumping core on crashes via the standard resource limit technique (ensuring
RLIMIT_CORE
is set to0
). This recommendation now appears in Vault's production hardening guide. - An attacker with root privileges can still produce core dumps, either by raising the core limit with prlimit(1) or by using
ptrace(2)
as discussed above. It's possible in principle to modify the Linux kernel to permanently disable both core dumps andptrace(2)
. In order not to have to build multiple versions of the kernel, it would probably be best to implement this as a kernel command-line option.
I believe these measures, combined with secure boot and restrictions on loadable kernel modules, would block all key recovery attacks against unsealed vault
processes. However, a persistent attacker with root access doesn't need to read the keys from an unsealed instance, but can instead backdoor userspace binaries (vault
itself, or sshd
or a shell) and wait for the next time operators perform an unseal. It's an open question whether Vault can be feasibly deployed under something like the ChromeOS verified boot model, with a chain of trust that starts in the hardware and then verifies the kernel and finally all relevant parts of the userland.