Vault is a HashiCorp product that allows secure secrets management. Vault will store and encrypt secrets for your applications such as passwords and SSL certificates.
When Vault is first started, it is in a sealed state. In the sealed state, Vault does not know how to decrypt your data. The first operation when starting a Vault server is to unseal the vault. This means that we will provide Vault with the necessary keys to decrypt the underlying secrets.
Shamir’s secret sharing
Vault uses Shamir’s secret sharing algorithm to decrypt your data. All of the data in Vault is encrypted with the “Encryption Key”. This encryption key is then encrypted with the “Master Key”. The master key is divided into parts, where each part is distributed to a trusted participant. The trusted participants are typically the administrators for Vault. By default, Vault will split the “Master Key” into 5 parts, with 3 out of 5 shares required to unseal the Vault. These number of shares, as well as the threshold to unseal, can be changed when initializing Vault.
Note: The Encryption Key is never displayed, it is created when unsealing the Vault using shares of the Master Key. If Vault is sealed, the Encryption key is destroyed and a new Encryption key will be generated upon the next ‘unseal’ command.
Initializing the Vault
Before unsealing the Vault, we need to initialize it. This is the process that generates the master key and applies Shamir’s secret sharing algorithm to split the master key into the desired number of shares. These shares are the ‘unseal keys’
> vault operator init
Unseal Key 1: KdSPt8UF3Y9hz7tFVN7/WJyszuiTt7ecRVUg1oeR04MC
Unseal Key 2: pAYjScJHcT4xDt4yZ6SVj+9pq4LWjpoj/bt2m0DG8FRN
Unseal Key 3: YW/AHB0mPATZtUeGRNd6ocKIYrH5a+l4GVdAbAmIFeJj
Unseal Key 4: +lmVrcRrf9kFtDRG6UK7LCbtqjk8qaD8luZsYmezY/XG
Unseal Key 5: XKXOAih+yJ7tew3KB22HiLHDxLwbwm6jtnlVPjnybex4
Initial Root Token: UWN0vSwDNb3P5krPPEnl3uGw
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See “vault operator rekey” for more information.
This will initialize the Vault sever with the default configuration. (5 key shares, 3 required to unseal). We can see in the output that the unseal keys are printed to the screen. At this point, we should distribute one key to each trusted Vault administrator.
Note: We can specify more options on the command line like so:
> vault operator init -key-shares=7 -key-threshold=5
This will initialize a Vault server with 7 key shares and 5 required to unseal.
More information about init is available here.
Now that Vault has been initialized (i.e. the master key has been generated and split into parts), we are ready to being the unsealing process.
Each user with an unseal key would connect to the Vault server and perform the following command:
> vault operator unseal
Key (will be hidden):
Upon entering this command, we can see the unseal progress. This is a counter telling us how many unseal keys have been entered so far, as well as how many more unseal keys we need to enter to unseal the Vault server.
Problems with Unsealing:
Unsealing the Vault server can create some issues in production. For example, if the server reboots or stops, the vault service will also stop. This means that Vault will automatically become sealed again. Remember that in a sealed state Vault cannot decrypt your data. This means that any applications that rely on Vault to provide passwords or database credentials will start to fail until Vault is unsealed again.
Imagine if this happens at 3 AM on a Tuesday night, when the business critical application may be processing payment information. Each of the unseal key holders (or at least 3 out of 5, in our example) will need to be woken up to unseal the Vault server.
There is also a problem with scalability. Assuming there may be multiple Vault clusters, the number of key shares will start to grow and become harder to manage. This is where Auto Unsealing comes into play.
Vault Enterprise Auto Unseal
A feature of Vault Enterprise Pro is automatic unsealing in the event of failure or when creating new Vault clusters. This is done by unsealing with a trusted cloud provider like AWS Key Management Service, Azure Key Vault, or Google Cloud Key Management Service. When using auto unseal, we no longer split the master key into shares. Instead, we use the Cloud key from the trusted provider as an encryption key for the master key.
The auto unseal feature is not enabled by default, so I will show a quick demo on how to enable this feature for Google Cloud KMS.
Google Cloud Setup
1. Create a new project on Google Cloud named ‘vault-project’
2. Go the the “APIs & Services” screen and select “Enable APIs and Services”
3. Search for “KMS” and Enable the “Cloud Key Management Service (KMS) API”
4. After enabling KMS, you must create credentials so Vault can access the KMS API.
5. Select “Cloud Key Management Service (KMS) API” from the drop-down
6. Create a service account for vault titled ‘vault-service’ with the Role ‘Cloud KMS CryptoKey Encrypter/Decrypter’. This will allow the vault-service account to access the Cloud KMS keys. Download the key as JSON. Save the JSON file, as we will need to copy it to the Vault server.
7. Now search for ‘Cryptographic Keys’ in the search bar and create a key ring for Vault.
8. Create a keyring for vault titled ‘vault-keyring’. You can set the key ring location to whatever you like. I will choose us-east1 for this example.
9. Now create the key with your specified parameters. This is the key that will be used to encrypt and decrypt the master key.
10. We should now see the following screen. We have created a ‘vault-key’ that is part of our ‘vault-keyring’
11. Now select the key, and add member permissions on the right. We want to add the ‘vault-service’ account created in step 6, so we will search for ‘vault-service’
12. Select the ‘Viewer’ role for the vault-service account. This gives the service account permissions to read the ‘vault-key’. Remember we already gave the vault-service account the role of ‘Cloud KMS CryptoKey Encrypter/Decrypter’ which allows it to encrypt and decrypt the vault-key.
At this point, we have setup the GCP cloud key for encrypting the master key in Vault. Next, we need to configure the Vault server to be able to retrieve this trusted key from GCP.
1. Copy the credentials JSON we downloaded from GCP to the Vault Server. These are the credentials Vault will need to interact with our GCP account.
2. Add the ‘seal’ stanza to the Vault config file. (/etc/vault/vault_server.hcl in my case)
Set the above parameters according to the project name, region, key ring name, and
crypto key name we setup in GCP. If you followed the names I used above, this stanza
should be identical except for the path to the JSON credentials file.
3. When you first login to your Vault server, run:
> vault status
Error checking seal status: Error making API request.
URL: GET https://vault-s1.shadow-soft.com:8200/v1/sys/seal-status
Code: 400. Errors:
* server is not yet initialized
We should see that the server is not yet initialized.
4. Now we need to initialize the server with the cloud key. We will use the following
command. This will use the cloud key to create the Master key for vault. We are only
using one key share because we are no longer splitting the master key into shares and
distributing the shares to trusted participants. In this case, GCP is our trusted participant.
> vault operator init -stored-shares=1 -recovery-shares=1
-recovery-threshold=1 -key-shares=1 -key-threshold=1
Recovery Key 1: eZn+KCypEDm4vP/ak83Ge9UlavH8JL034jqFNMd1W+k=
Initial Root Token: 3JhjbZMZf8PvQa28LW1KbYHV
Success! Vault is initialized
Recovery key initialized with 1 key shares and a key threshold of 1. Please
securely distribute the key shares printed above.
The output will contain a recovery key that can be used if the GCP key is ever lost. Be sure to keep the recovery key in a safe location. The
‘vault operator unseal’ command will accept the recovery key if Vault needs to be unsealed manually.
5. Now that auto unseal has been setup, we can test it is working correctly by shutting down
the vault server and then restarting it. Upon restart, the vault should remain in an
Note: Although we have enabled auto unseal, we can still use the
‘vault operator seal’ command to seal the vault in the case of the Vault server being compromised. When sealed manually, GCP KMS will not auto-unseal the Vault.
Interested in Vault? Learn how Shadow-Soft can help you evaluate, adopt and integrate Vault
Vault Enterprise Auto Unseal is a valuable feature that prevents downtime when vault machines go offline or restart. It eliminates the need for admins to manually enter unseal keys. This makes managing multiple Vault clusters easier.