Using Chef & Hashicorp Vault for secrets management

Chef is a configuration management tool that promotes the idea of infrastructure as code. This means that we can define the desired state of a system and automatically roll out changes to multiple servers at once.

Chef can be used to set up applications that may require passwords, tokens, or keys upon startup. However, what happens when we need to use secrets in Chef while maintaining confidentiality of those values?

Hashicorp Vault is a tool for managing secrets. It secures, stores, and controls access to tokens, passwords, certificates, and any other secrets you may need to store for an application. This is a simple Vault CLI command to read data from a key value store. Each key-value entry can be customized to only allow certain users to access the data.

$ vault kv get secret/my-app
Key Value
--- -----
username bob
password P@##w0rd

**Vault can also store dynamic secrets that are generated when needed. I will be focusing on the simple key value store for this post, but remember that dynamic secrets can be loaded using Chef with the same methods described here.

So what’s the problem?

Our goal is to populate a config file with the username and password shown above. The resulting config file (/etc/my-app/config) will look like this:

[my-app-config]
Username = bob
Password = P@##w0rd

This seems trivial in Chef, as we can directly write the values for the config file. However, it is bad practice to hard code secrets in Chef code as this creates a security risk.

This is an example of what NOT to do in Chef:

file '/etc/my-app/config' do
content <<-EOF
[my-app-config]
Username = bob
Password = P@##w0rd
EOF
end

Imagine what happens if this Chef recipe falls into the wrong hands, we would have exposed the username and password for our application!

How do we fix this?

Chef and Hashicorp Vault can be used together to solve the problem of secrets management. Instead of storing secrets in our Chef recipes, we can use Vault inside our cookbooks to read secrets without ever exposing the data to any users.

Here is an example of populating the config file in a Chef recipe without hard coding secrets.

read_vault 'Read secret at secret/data/my-app' do
path "secret/data/my-app"
address 'http://127.0.0.1:8200'
token '96d0a802-fd00-5b57-87b4-0b15ed2dbe3c'
role_name 'chef-role'
notifies :create, "template[/etc/my-app/config]", :immediately
end

template '/etc/my-app/config' do
source 'my-app.conf'
owner 'my-app'
group 'my-app'
mode '0600'
sensitive true
variables lazy {
{
  :username => node.run_state["secret/data/my-app"].data[:data][:username],
  :password => node.run_state["secret/data/my-app"].data[:data][:password],
}
}
action :nothing
end

The ‘read_vault’ resource will take in the following parameters and read the desired secret:

path: the path of the secret you want to read
address: the address of the vault server
token: a token that allows us to talk to vault (I used the root token for a proof of concept)
role_name (optional): If using approle for authentication, you can specify the role name

We are using the template resource to populate our config file this time. Instead of hard coding the secrets, we will use lazy-loaded variables in the template resource. The ‘run_state’ attributes store the secret only in memory during the chef run. This means that the data for the secret is never sent back to the Chef Server nor stored in the node object.

This piece of code will only update the template when the ‘read_vault’ resource updates. The ‘read_vault’ resource is a custom resource that utilizes the vault ruby gem and allows us to read a secret from Vault. To read a secret from Vault, we will need some way to authenticate. For example, a token.

Note: there are more ways to authenticate. Please see the Vault documentation (https://www.vaultproject.io/intro/getting-started/authentication.html)

Let’s take a look at the custom resource now:

# Custom resource for reading a Vault secret
require 'vault'

resource_name :read_vault

# path : a String corresponding to the secret path in Vault (ex. 'kv/my-app')
# address: the address where the vault server is running (ex. 'http://127.0.0.1:8200')
# token : one of the ways to authenticate with Vault
# role_id: another way to authenticate with Vault. This assumes you have an approle created (https://www.vaultproject.io/docs/auth/approle.html)
property :path, String, required: true
property :address, String, required: true
property :token, String, required: true
property :role_name, String, required: false

action :read do
# Need to set the vault address
Vault.address = new_resource.address

# Authenticate with the token
Vault.token = new_resource.token

if property_is_set?(:role_name) # Authenticate to Vault using the role_id
approle_id = Vault.approle.role_id(new_resource.role_name)
secret_id = (Vault.approle.create_secret_id(new_resource.role_name)).data[:secret_id]
Vault.auth.approle(approle_id, secret_id)
end

# Attempt to read the secret
secret = Vault.logical.read(new_resource.path)
if secret.nil?
raise "Could not read secret '#{new_resource.path}'!"
end

# Store the secret in memory only
node.run_state[new_resource.path] = secret

# Whether or not this resource was updated
# True = allows notifications to start
updated_by_last_action(true)
end

This custom resource makes use of the Vault methods from the vault ruby gem. This allows us to do things like “Vault.logical.read(“secret/path”)”. This is the most important piece of our code, as it actually does the Vault secret read.

Before we can do this read we need to authenticate with Vault. I will be showing 2 methods for authentication:

1. Pass in the root token

1.1. Allows reading of any secret (UNSECURE)

2. Pass in a token that allows us to authenticate with an AppRole.

2.1. Allows us to access the role-id and secret-id. When together, they form a second token that allows us to read the secrets assigned to the given role.

Let’s setup a Vault Service on Chef

Let’s setup a Vault server on a Chef node to test our code:

1. Install Vault on the Chef node

2. Start the Dev Server

https://www.vaultproject.io/intro/getting-started/dev-server.html

2.1. vault server -dev

2.2. Save the ‘Root Token’ for future use

3. Create secrets

3.1. Open a new terminal on the Vault Server

3.2. vault kv put secret/my-app username=bob password=P@$$w0rd

3.3. vault kv get secret/my-app

4. Run the Chef recipe and use the read_vault resource

read_vault 'Read secret at secret/data/my-app' do
path "secret/data/my-app"
address 'http://127.0.0.1:8200'
token ''
notifies :create, "template[/etc/my-app/config]", :immediately
end

In this example, we are using the Chef recipe to read the secret at ‘secret/data/my-app’. This is done by passing in the root token from initial setup. The root token allows us to read any path on the Vault server.

We have to pass in the address for the Vault server (it’s the localhost on the node in my case). When a read on the secret occurs, we will automatically update our template with the correct secret values.

Important note about paths:

I am referencing the path ‘secret/data/my-app’ even though I wrote to ‘secret/my-app’ in the Vault CLI.

Adding the ‘data’ to the path is required if your key value store is KV version 2. This is because the KV version 2 store uses a slightly different API call that requires the string ‘data’ to come directly after the top level secret location.

If you are reading from a KV version 1 store, you can change the path to ‘secret/my-app’

To determine which KV version you are using:

vault secrets list -detailed

You should see ‘version:2’ under options for the ‘secret/’ path

How to authenticate with an Approle:

Now that we have shown how to read secrets using the root token, we need a more secure method to authenticate with Vault. It’s not good practice to pass around the root token, as that allows access to all secrets in Vault.

In this example, we will create an app-role for Chef that will allow it to read secrets specific to Chef. The app role will be associated with a policy that will allow us to fine tune which secrets the App Role can access.

1. Create a policy on the Vault server

1.1. Write a simple policy file (chef-policy.hcl) that allows read access to ‘secret/data/my-app’ only.

Note: we have to specify ‘data’ in the path because this is KV version 2
path "secret/data/my-app" {
capabilities = ["read"]
}

1.2. vault policy write chef-policy chef-policy.hcl

2. Create an AppRole to authenticate

2.1. vault auth enable approle

2.2. vault write auth/approle/role/chef-role policies=chef-policy

2.2.1. We need to associate the chef policy with this new chef-role

3. Create a second policy that allows us to access the role information

3.1. We will create a new policy titled ‘chef-role-token’. This will allow us to use a token for authenticating with AppRole.

3.2. We need to allow an ‘update’ to secret-id. This means that the policy allows generating a new, temporary secret-id. We also need to allow ‘read’ permissions on the role-id. These permissions ensure that we will be able to generate a token for the AppRole.

path "auth/approle/role/chef-role/secret-id" {
capabilities = ["update"]
}
path "auth/approle/role/chef-role/role-id" {
capabilities = ["read"]

3.3. vault policy write chef-role-token chef-role-token.hcl

4. Generate the token (needed for Chef) for the chef-role-token policy.

This allows Chef to get the role-id and secret-id for the chef-role. The role-id and secret-id will be combined to create a NEW token that will let us read the secrets assigned to chef-role

4.1. vault token create -policy=chef-role-token

4.2. Use this token in the Chef code

read_vault 'Read secret at secret/data/my-app' do
path "secret/data/my-app"
address 'http://127.0.0.1:8200'
token “”
role_name 'chef-role'
notifies :create, "template[/etc/my-app/config]", :immediately
end

In this example, we are also passing in a role_name. this is essential for our authentication process. The full authentication is as follows:

1. Pass in token for ‘chef-role-token’ policy (in recipe)

2. Use this token to read the role-id and secret-id (in custom resource)

3. Use the role-id and secret-id to generate a new token that is associated with the chef-role (in custom resource)

4. The chef-role has access defined by ‘chef-policy’, therefore it can access ‘secret/data/my-app’

Further Considerations:

Although we are no longer using the root token, it is best practice to avoid tokens hard coded into Chef.

If these tokens are compromised, that means that someone may have access to the secrets permitted to the given policy. (In our case, the token would allow a user to login to the chef-role and access ‘secrets/data/my-app’)

To remedy this, we could use databags to store the encrypted values of the tokens for added security.

Conclusion

Chef and Hashicorp Vault are compatible technologies that allow secrets to be moved out of Chef. The addition of Hashicorp Vault to an existing Chef infrastructure can help alleviate security risks and make secrets management easier. There are certain considerations to keep in mind when using Chef and Vault, such as ensuring that the root token is not exposed. However, when used properly, these technologies can ensure your data and application secrets are secure.

Need more help? Visit these resources:

Related Posts