Vault Kubernetes: Introduction

This course will give you the introductory knowledge and skills you need to use HashiCorp Vault in Kubernetes. 

Table of Contents

Overview

This course will introduce you to the basics of using HashiCorp Vault in Kubernetes. You will learn how to deploy Vault in Kubernetes, configure Vault for use with Kubernetes, integrate Vault with Kubernetes, and use Vault in your Kubernetes applications.

Vault Kubernetes Deployment:
Vault can be deployed in Kubernetes using the official HashiCorp Vault Helm chart. The Helm chart allows you to deploy Vault in a variety of configurations, including a single in-memory Vault server for testing, a standalone Vault server persisting to a volume, or a highly available cluster of Vault servers.

Vault Kubernetes Configuration:
Once Vault is deployed in Kubernetes, you need to configure it for use with your applications. You will need to create a Kubernetes secret that contains the Vault token for your applications to use. You can also configure Vault to use different authentication methods, such as LDAP or Active Directory.

Vault Kubernetes Integration:
Once Vault is configured, you can integrate it with your Kubernetes applications. You can do this by using the Vault Kubernetes Auth method, which allows your applications to authenticate with Vault using a Kubernetes Service Account Token. You can also use the Vault Kubernetes Agent, which injects a Vault sidecar container into your pods that can access Vault secrets.

Vault Kubernetes Application:
Once Vault is integrated with your Kubernetes applications, you can use it to store secrets for your applications. You can store passwords, certificates, SSH keys, and other sensitive data in Vault. Vault provides a variety of secret engines that you can use to store different types of secrets.

Section 1: Vault Kubernetes Deployment

Lesson 1 - Vault Kubernetes Deployment: Introduction to Vault

Purpose of Vaults:

1. Secure, store, and tightly control access to tokens, passwords, certificates, and encryption keys for protecting secrets and other sensitive data using a UI, CLI, or HTTP API.
2. Vault brokers deeply integrate with trusted identities to automate access to secrets, data, and systems.
3. Secure applications and systems with machine identity and automate credential issuance, rotation, and more. Enable attestation of application and workload identity, using Vault as the trusted authority.
4. Leverage trusted identity platforms you use every day to secure, store, and access credentials and resources.

Lesson 2 - Vault Kubernetes Deployment: Introduction to Vault on Kubernetes

1. Running a Vault Service - The Vault server cluster can run directly on Kubernetes. This can be used by applications running within Kubernetes as well as external to Kubernetes as long as they can communicate to the server via the network.
2. Accessing and Storing Secrets - Applications using the Vault service running in Kubernetes can access and store secrets from Vault using a number of different secret engines and authentication methods.
3. Running a Highly Available Vault Service - By using pod affinities, highly available backend storage (such as Raft), and auto-unseal, Vault can become a highly available service in Kubernetes.
4. Encryption as a Service - Applications using the Vault service running in Kubernetes can leverage the Transit secret engine as "encryption as a service." This allows applications to offload encryption needs to Vault before storing data at rest.
5. Audit Logs for Vault -  Operators can choose to attach a persistent volume to the Vault cluster, which can be used to store audit logs.
6. Vault can run directly on Kubernetes - In addition to the native integrations provided by Vault itself, any other tool built for Kubernetes can choose to leverage Vault.

Lesson 3 - Vault Kubernetes Deployment: Prerequisites

1. Install statically linked binary executable clients.
   • Kubectl: Install Tools | Kubernetes
   • Helm: Installing Helm
   • Vault: Install | Vault | HashiCorp Developer
2. A valid and connectable Kubernetes cluster at a version >= 1.24.0
3. Versions used for the current revision of this material:
   • Kubectl: 1.27.1
   • Helm: 3.11.3
   • Vault: 1.12.6 client and 1.13.1 server
   • Kubernetes: 1.26.4
4. A valid Kube config for kubectl and helm to connect to the Kubernetes cluster: https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/

Lesson 4 - Vault Kubernetes Deployment: Install Vault on Kubernetes with Helm

The Vault Helm chart is the recommended way to install and configure Vault on Kubernetes. In addition to running Vault itself, the Helm chart is the primary method for installing and configuring Vault to integrate with other services, such as Consul for High Availability (HA) deployments.

• Subscribe to the Hashicorp Helm chart repository and refresh the metadata:

helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update

• Install a Vault standalone server cluster with Helm:

helm install -n vault-system --create-namespace --set ui.enabled=true vault hashicorp/vault

• Verify successful deployment with Helm:

helm -n vault-system list

• Verify successful deployment with Kubectl:

kubectl -n vault-system get all

Lesson 5 - Vault Kubernetes Deployment: Vault Cluster Ingress

You will need network access to the Vault cluster running on Kubernetes.

If your cluster’s ingress is already set up to allow connectivity, then feel free to modify the networking to connect to the Vault API and UI port at 8200. The path prefix will be “/”.

kubectl proxy --port=8200

helm -n vault-system upgrade --set 'ingress.enabled=true' vault

Lesson 6 - Vault Kubernetes Deployment: Vault Cluster Initialization

When a Vault server is started, it starts in a sealed state. In this state, Vault is configured to know where and how to access the physical storage but doesn't know how to decrypt any of it.

Unsealing is the process of obtaining the plaintext root key necessary to read the decryption key to decrypt the data, allowing access to the Vault.

Prior to unsealing, almost no operations were possible with Vault. For example, authentication, managing the mount tables, etc., are all not possible. The only possible operations are to unseal the Vault and check the status of the seal.

The data stored by Vault is encrypted. Vault needs the encryption key in order to decrypt the data. The encryption key is also stored with the data (in the keyring) but encrypted with another encryption key known as the root key.

Therefore, to decrypt the data, Vault must decrypt the encryption key, which requires the root key. Unsealing is the process of getting access to this root key. The root key is stored alongside all other Vault data but is encrypted by yet another mechanism: the unseal key.

Upon initially visiting the UI there will be an initialization setup prompt.

Enter the following defaults:

5 key shares

• 3 key threshold
• Explore the PGP encryption options if you like
• Select “Initialize”.

Retain the unseal keys and root token for later, and then select “Continue to unseal”

Things to Note:

• Pods will fail Readiness Probes until the Vault node is unsealed
• Retain the unseal keys in a readily available and persistent location for these labs
• If the cluster nodes restart during this lab, then you will need to unseal again

Enter at least three unseal keys to unseal Vault, and to access the initial authentication page

Verify Vault cluster readiness: 

• kubectl -n vault-system exec -ti vault-0 -- vault list secrets
• It should return “permission denied”, and definitely not otherwise e.g. “connection refused.”

You have successfully deployed a basic Vault cluster to Kubernetes.

[Back to the top]

Section 2: Vault Kubernetes Configuration

Lesson 1 - Vault Kubernetes Configuration: Authentication

Authentication in Vault is the process by which user or machine supplied information is verified against an internal or external system.

• Vault supports: GitHub, LDAP, AppRole, and more.
• Each auth method has a specific use case.

Before a client can interact with Vault, it must authenticate against an auth method. Upon authentication, a token is generated. This token is conceptually similar to a session ID on a website. The token may have an attached policy, which is mapped at authentication time. This process is described in detail in the policies concepts documentation.

• Authenticate against an auth method
• Upon authentication, a token is generated for Vault

• Most authentication backends must be enabled before use.

Lesson 2 -  Vault Kubernetes Configuration: Root Token Authentication

Tokens are the core method for authentication within Vault.

• Tokens can be used directly, or auth methods can be used to dynamically generate tokens based on external identities.

Within Vault, tokens map to information:

• The most important information mapped to a token is a set of one or more attached policies.
  
• These policies control what the token holder is allowed to do within Vault. Other mapped information includes metadata that can be viewed and is added to the audit log, such as creation time, last renewal time, and more.
  

Root tokens are tokens that have the root policy attached to them

• Root tokens can do anything in Vault.
• In addition, they are the only type of token within Vault that can be set to never expire without any renewal needed. As a result, it is purposefully hard to create root tokens.

Root tokens are useful in development but should be extremely carefully guarded in production.

• In fact, the Vault team recommends that root tokens are only used for just enough initial setup (usually, setting up auth methods and policies necessary to allow administrators to acquire more limited tokens) or in emergencies, and are revoked immediately after they are no longer needed.
• If a new root token is needed, the operator generate-root command and associated API endpoint can be used to generate one on-the-fly.

It is also good security practice for there to be multiple eyes on a terminal whenever a root token is live.

• This way multiple people can verify as to the tasks performed with the root token, and that the token was revoked immediately after these tasks were completed.

Lesson 3 -  Vault Kubernetes Configuration: UI

The “vault-ui” Kubernetes service provides the UI at port 8200 by default

With proper ingress, forwarding, or proxying the UI can be accessed in a browser at:

• http://<hostname>.<domain>:<mapped port>/ui
• If you setup networking for ingress, then you may need to remove the “/ui” depending upon the path prefix

Login to the UI with token based authentication and the root token

Explore the interface, and note the only current secrets mount is the root user’s cubbyhole

Lesson 4 -  Vault Kubernetes Configuration: Kubernetes Authentication Engine

Auth methods are the components in Vault that perform authentication and are responsible for assigning identity and a set of policies to a user

Auth methods are the components in Vault that perform authentication and are responsible for assigning identity and a set of policies to a user. Vault will enforce authentication in all cases as part of the request processing. In most cases, Vault will delegate the authentication administration and decision to the relevant configured external auth method (e.g., Amazon Web Services, GitHub, Google Cloud Platform, Kubernetes, Microsoft Azure, Okta ...).

Having multiple auth methods enables you to use an auth method that makes the most sense for your use case of Vault and your organization.

For example: 

• On developer machines, the GitHub auth method is easiest to use.
• On servers, the AppRole method is the recommended choice.

The Kubernetes auth method can be used to authenticate with Vault using a Kubernetes Service Account Token. This method of authentication makes it easy to introduce a Vault token into a Kubernetes Pod.

You can also use a Kubernetes Service Account Token to log in via JWT auth.

See the section on How to work with short-lived Kubernetes tokens for a summary of why you might want to use JWT auth instead and how it compares to Kubernetes auth.

Lesson 5 -  Vault Kubernetes Configuration: KV2 Secrets Engine

The kv secrets engine is used to store arbitrary secrets within the configured physical storage for Vault.

Key names must always be strings.

• If you write non-string values directly via the CLI, they will be converted into strings.
• You can preserve non-string values by writing the key/value pairs to Vault from a JSON file or using the REST API

It honors the distinction between the create and update capabilities inside ACL policies

• Patch capability is also supported which is used to represent partial updates, whereas the update capability represents full overwrites

After the secrets engine is configured and a user/machine has a Vault token with the proper permission, then it can generate credentials

The kv secrets engine allows for writing keys with arbitrary values.

Lesson 6 -  Vault Kubernetes Configuration: Vault Policies

Everything in Vault is path-based, and policies are no exception.

Policies provide a declarative manner to grant or forbid access to certain paths and operations in Vault.

Policy workflows and syntaxes:

• Policies are deny by default so an empty policy grants no permission in the system
• Policies use path-based matching to test the set of capabilities against a request
• A policy path may specify an exact path to match, or it could specify a glob pattern which instructs Vault to use a prefix match
• Each path must define one or more capabilities which provide fine-grained control over permitted (or denied) operations

Select the menu items Policies→Create ACL Policy.

Name the policy “webapp-one-read”, and use the following content for the policy:

path "kv/data/webapps/app_one" {
  capabilities = ["read"]
}

Create similar read policies for webapp-two-read, backendapp-one-read, and backendapp-two-read

These will enable read access to the previously created kv engine secrets when attached as authorization to an authentication entity

You have successfully configured a basic Vault cluster for Kubernetes authentication, configured KV2 secrets management, and authorized an authenticated Kubernetes agent to retrieve the secrets.

[Back to the top]

Section 3: Vault Kubernetes Integration

Lesson 1 - Vault Kubernetes Integration: Kubernetes Authentication Role

1. Select the menu items Access → Kubernetes → Create role
2. Enter:

1. • “webapp-one” for role
   • “webapp-one” for Bound service account names → select Add
   • “default” for Bound service account namespaces → select Add
   • At the bottom there is a dropdown menu item for “Tokens”.
     • Toggle that item to reveal an entry for “Generated Token's Policies”.
2. • Enter the corresponding policy “webapp-one-read” and select “Add” when finished. Select “Save” when finished with the role.
   • Select “Save” when finished with the role.
3. Create three more differently named Vault roles with similarly mapped Vault policies for the Kubernetes Bound service account names webapp-two, backendapp-one, and backendapp-two.

Bonus: Create a role for both the Bound service account names webapp-one and webapp-two, and a role for both the Bound service account names backendapp-one and backendapp-two.

• The first role should be mapped to both Vault policies webapp-one-read and webapp-two-read.
• The second role should similarly have read access to both backend app secrets.
• Suggested names would be “webapps” and “backendapps” for the roles, and “webapp” and “backendapp” for the service accounts.

Lesson 2 -  Vault Kubernetes Integration: Kubernetes Authentication Service Account

1. Create an application service account in the “default” namespace corresponding to the Bound service account name from the previous steps --> kubectl create sa webapp-one
2. Verify the service account --> kubectl describe sa webapp-one
3. Repeat this for the other service accounts webapp-two, backendapp-one, and backendapp-two
   • Bonus: If you created two Vault roles in the Kubernetes authentication engine, each mapped to a webapp and backendapp Kubernetes service account, then create those two service accounts now as well.
   • Note: Most sandbox Kubernetes clusters do not have RBAC enabled by default. If yours does, then consult the Intermediate lesson beginning for the necessary ClusterRole and ClusterRoleBinding for the Vault service account to review token requests for Kubernetes service accounts.
4. Create a Kubernetes secret from the following manifest to configure the service account:
   apiVersion: v1
kind: Secret
metadata:
  name: webapp-one-configure
  annotations:
    kubernetes.io/service-account.name: webapp-one
type: kubernetes.io/service-account-token
5. Create similar Kubernetes secrets for the other service accounts with modified annotations.

Lesson 3 -  Vault Kubernetes Integration: Kubernetes Workload Annotations

1. Create a basic deployment in the default workspace:
   • kubectl create deployment webapp-one --image httpd
2. Verify the pod workload: 
   • kubectl describe pod/webapp-one-<sha hash>
3. Quickly verify that the default Vault secret mount path in the container is inaccessible: 
   • kubectl exec $(kubectl get pod -l app=webapp-one -o jsonpath="{.items[0].metadata.name}") --container httpd -- ls /vault/secrets
4. Modify the service account for the deployment to be the service account we created associated with the Kubernetes authentication Vault role: 
   • kubectl patch deployment/webapp-one --patch '{"spec": {"template": {"spec": {"serviceAccountName": "webapp-one"}}}}'
5. Quickly verify that the default Vault secret mount point is still inaccessible:
   • kubectl exec $(kubectl get pod -l app=webapp-one -o jsonpath="{.items[0].metadata.name}") --container httpd -- ls /vault/secrets

Lesson 4 -  Vault Kubernetes Integration: Kubernetes Secrets Engine

The Kubernetes Secrets Engine for Vault generates Kubernetes service account tokens.

• Also, optionally, service accounts, role bindings, and roles
• Created service account tokens have a configurable TTL, and any objects created are automatically deleted when the Vault lease expires.

For each lease, Vault will create a service account token attached to the defined service account, with the service account token returned to the caller.

1. Select the menu items Secrets → Enable New Engine.
   • Notice that the Kubernetes secrets engine cannot be managed in the UI as it is not an option
2. Enable the Kubernetes secrets engine:
   • kubectl -n vault-system exec -ti vault-0 -- /bin/sh -c 'VAULT_TOKEN=<root token> vault secrets enable kubernetes'
3. Configure the Kubernetes secrets engine: 
   • kubectl -n vault-system exec -ti vault-0 -- /bin/sh -c 'VAULT_TOKEN=<root token> vault write -f kubernetes/config'
4. Configure the Vault role for the Kubernetes secrets engine to map to the “webapp-one” service account and “default” namespace: 
   • kubectl -n vault-system exec -ti vault-0 -- /bin/sh -c 'VAULT_TOKEN=<root token> vault write kubernetes/roles/webapp-one service_account_name=webapp-one allowed_kubernetes_namespaces=default' 
5. Create credentials for the “webapp-one” service account in the “default” namespace, and thereby return a service account token: 
   • kubectl -n vault-system exec -ti vault-0 -- /bin/sh -c 'VAULT_TOKEN=<root token> vault write kubernetes/creds/webapp-one kubernetes_namespace=default'

Note that you can now view the Kubernetes secrets engine and its basic configuration in the UI; otherwise, you have no control over it.

You have successfully integrated the Vault cluster into a Kubernetes cluster for service account token authentication and workload secrets injection.

[Back to the top]

Section 4: Vault Kubernetes Application

Lesson 1 - Vault Kubernetes Application: Secrets Retrieval Methods

The Vault Agent Injector alters pod specifications to include Vault Agent init-containers that render Vault secrets to a shared memory volume using Vault Agent Templates.

• By rendering secrets to a shared volume, containers within the pod can consume Vault secrets without being Vault aware.

The injector is a Kubernetes Mutation Webhook Controller.

• The controller intercepts pod events and applies mutations to the pod if annotations exist within the request.
• This functionality is provided by the vault-k8s project and can be automatically installed and configured using the Vault Helm chart.

The Secrets Store CSI driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume.

• Once the Volume is attached, the data in it is mounted into the container's file system.

The Vault Sidecar Agent Injector leverages the sidecar pattern to alter pod specifications to include a Vault Agent container that renders Vault secrets to a shared memory volume.

• By rendering secrets to a shared volume, containers within the pod can consume Vault secrets without being Vault-aware. The injector is a Kubernetes mutating webhook controller. The controller intercepts pod events and applies mutations to the pod if annotations exist within the request. This functionality is provided by the vault-k8s project and can be automatically installed and configured using the Vault Helm chart.
• In summary, The Vault Sidecar Agent Injector leverages the sidecar pattern to alter pod specifications to include a Vault Agent container that renders Vault secrets to a shared memory volume.

The CSI Secrets Store driver enables users to create SecretProviderClass objects.

• Objects define which secret provider to use and what secrets to retrieve.

When pods requesting CSI volumes are made, the CSI Secrets Store driver sends the request to the Vault CSI provider if the provider is Vault.

• The Vault CSI provider then uses the specified SecretProviderClass and the pod’s service account to retrieve the secrets from Vault, and mount them into the pod’s CSI volume

Lesson 2 - Vault Kubernetes Application: Vault Agent

Modify the “webapp-one” deployment annotations for the associated pods to inject the Vault init-container. This will mount the secrets from “webapps/app_one” at “/vault/secrets/mysecret” in the pod’s container, and authenticate via the mapping between the Kubernetes service account and the Vault role through the Kubernetes authentication engine:

kubectl patch deployment/webapp-one --patch '{"spec": {"template": {"metadata": {"annotations": {"vault.hashicorp.com/agent-inject": "true", "vault.hashicorp.com/role": "webapp-one", "vault.hashicorp.com/agent-inject-secret-mysecret": "kv/data/webapps/app_one"}}}}}'

Verify the annotations and service account:

kubectl describe deployment/webapp-one

Verify the pod has a corresponding Vault init container (“vault-agent-init”) mounting the secrets correctly: 

kubectl describe pod/webapp-one-<sha hash>

Verify that the default Vault secret mount point is now accessible:

kubectl exec $(kubectl get pod -l app=webapp-one -o jsonpath="{.items[0].metadata.name}") --container httpd -- ls /vault/secrets

Lesson 3 - Vault Kubernetes Application: CSI Driver

Add the secrets store CSI driver Helm repo:

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

Install the driver alongside Vault: 

helm -n vault-system install csi secrets-store-csi-driver/secrets-store-csi-driver --set syncSecret.enabled=true

Verify a successful driver installation by confirming the new running pod:

 kubectl -n vault-system get pod

This enables the use of the CSI driver for secret storage

• The complexity of utilizing this plugin falls outside the introductory level and will be further explored at the intermediate level.

Additionally, the Vault installation must be configured to enable integration with this driver, and that also will be explored further at the intermediate level.

Lesson 4 - Vault Kubernetes Application: Kubernetes Secrets Sync

The Vault CSI plugin is also capable of mapping Vault secrets to Kubernetes secrets.

The Secrets Store CSI Driver also supports syncing to Kubernetes secret objects.

Kubernetes secrets are populated with the contents of files from your CSI volume, and their lifetime is closely tied to the lifetime of the pod for which they are created.

You have successfully provided Vault secrets for retrieval within the container as a file via Vault Agent init-container injection and installed the secrets store CSI driver as a future alternative to the Vault Agent.

Contact us with any questions. We are here to help!