Custom Vault Integrations: Python

Introduction

Hashicorp’s Vault has become the industry standard tool for secrets management. It comes packaged with multiple solutions for engines supporting various authentication methods, secrets integrations, and storage. However, not every application has a first-party, third-party, or community secrets engine integration with Vault. This is especially true for internally developed custom applications and software that your company may utilize internally or as customer-facing products. In these situations, you often need to develop a custom integration with Vault.

These articles will guide you in developing a basic custom secrets engine integration with Vault in each of four common languages. It is assumed that you are already familiar with the fundamentals of administrating a Vault cluster, the common Vault engines, and coding in the language(s) of your choice that are demonstrated in each article.

This article will focus on a custom Vault integration developed in Python. Note that the assumed version of Python in this article is in the standard 3.7-3.9 range for the time of this writing.

Bindings Installation

Vault has a well documented REST API, and you can certainly interact with it through the REST API bindings in the application language. However, bindings to the Vault API already exist in several languages, and these should be used for their obvious advantages. In this section, we will see how to install those bindings in Python.

The Python bindings for Vault are community developed and maintained but still used commonly across the field. They implement all of the expected functionality you would expect in a clean Pythonic interface. Simply add the equivalent installation instructions for the bindings to your setup.py:

import setuptools

setup(
  ...
  install_requires=['hvac']
  ...
)

And install it as a Python dependency as per normal.

The Python [Read the Docs documentation] for these bindings is quite good. They are well organized into secrets, authentication, and system categories. They also contain excellent instructive MCVE snippets for those who are unfamiliar with bindings documentation and who would feel more comfortable with examples.

Client Construction and Authentication

The first step in interacting with the Vault API through the bindings is to construct a client for the interface. After constructing the client, the member method for authentication with a token can then be invoked. This code is actually remarkably similar across the different language bindings.

An easy and efficient procedure for initializing a Vault client in Python is through a class constructor. We can assign the client to a member variable and therefore do not need to continually reconnect and re-authenticate to the Vault cluster. Note that in the previous Go and Ruby articles, the client struct/object was private by default. However, in Python, this object will be public by default. Therefore, we use the stylistic __ to denote the member should be considered private.

Although the typical constructor design pattern in object oriented programming returns a class object (usually for external initialization), in this situation, we want the client to be private. Therefore, the constructor will not return a class object but does successfully construct the class object.

We initialize a Vault client to the member variable at the specified address. We also provide a member method for returning a client object authenticated with an input Vault token.

import os

import hvac


class Vault():
    """class for interfacing python with vault"""

    # constructor
    def __init__(self,
                 address: str = os.environ['VAULT_ADDR']) -> None:
        """instantiates and returns a vault client"""
        self.__client = hvac.Client(url=address)

    def token_auth(self,
                   token: str = os.environ['VAULT_TOKEN']) -> None:
        """authenticates a client with a vault token"""

        if len(token) != 26:
            raise RuntimeError('The Vault token is invalid')
        self.__client.token = token

Alternate Authentication Engines

You may not want to use the basic Token Authentication engine (and understandably so) for authentication with the Vault client. In that situation, we can utilize a few alternatives.

In the below method, we demonstrate how to use LDAP authentication. The username and password are input via in-process environment variables to ensure minimal detection overhead. The client is then authenticated with a login method bound to the ldap authentication engine. This method actually returns a response dictionary that contains the authenticating token, which is then validated in the final line of code.

def ldap_auth(self,
              username: str = os.environ['LDAP_USERNAME'],
              password: str = os.environ['LDAP_PASSWORD']) -> None:
    """authenticates a client with ldap"""
    response = self.__client.auth.ldap.login(
        username=username,
        password=password
    )

    if len(response['auth']['client_token']) != 26:
        raise RuntimeError('The returned Vault token from the LDAP authentication engine is invalid')

In the below method, we demonstrate how to use AppRole authentication with the “push” method. Passing in the role id and secret id as input arguments configures an authenticated client according to the associated role and policy. We can use the approle helper method in the auth proxied class to easily authenticate with a login method.

Anyone assiduously reading the articles up to this point will notice a major difference between the Python bindings for this authentication versus the Go and Ruby bindings. The Python bindings for AppRole will end-to-end authenticate the client through the proxied class in a single method invocation. The Go and Ruby bindings’ method instead return a token that can be input for client authentication. The advantages of a single method include more efficient code, and improved security with an obscured token without a transmission to a third persona in a “pull” model. The disadvantage includes the decrease in security by obscuring via the additional persona in the “pull” model procedure. You will need to weigh the trade-offs for your environment accordingly.

def approle_auth(self,
                 role_id: str,
                 secret_id: str) -> None:
    """authenticates a client with approle push auth"""
    self.__client.auth.approle.login(
        role_id=role_id,
        secret_id=secret_id
    )

Secrets Retrieval (KV2 Engine)

Now that we have a connected, authenticated, and authorized client, we are able to begin interfacing with the KV2 secrets engine and retrieve secrets. Similar to the methods above, the Python bindings for Vault have a very streamlined method for reading KV2 secrets. It is one line of code, which was not even true for the very convenient Ruby bindings’ methods.

We begin by passing the secret path to the method. Note that the bindings assume by default that the mount point is at secret/. The code retrieves the secret at the specified path, and returns it as a dictionary. The code verifies the retrieved secret with a “falsiness” check. The code then returns the key value pairs for the secret as a dictionary. Similar to the Go and Ruby bindings, note that the first data key in the dictionary is the bindings accessor (implicitly type converted by the method) for the actual secrets values in the returned value for the .read_secret_version, and the second data key is the top-level key corresponding to the values within the accessed secret for the KV2 engine. This coincidence can be confusing for those new to developing custom Vault integrations.

def kv2_read(self,
             secrets_path: str) -> dict:
    """returns a dictionary of key value pairs from a Vault secret"""
    secret = self.__client.secrets.kv.read_secret_version(path=secrets_path)

    if not secret:
        raise RuntimeError('No valid secret found in Vault at' + secrets_path)

    return secret['data']['data']

If your KV2 secret engine is not mounted at the default path, then you need to configure thusly prior to secrets retrieval:

self.__client.secrets.kv.v2.read_configuration(mount_point='not_secret')

Vault Auto-Unsealing

Vault cluster servers require unsealing during initialization, but they also often require unsealing during immutable upgrades, self-healing, and other server replacement procedures. In addition to rejoining the cluster via the Raft gossip protocol, the Python bindings are a popular choice for the auto-unseal process.

The unseal keys are passed as an input to the method. The server (presumably not the leader in the cluster) is verified to be sealed. The unseal keys are then used to auto-unseal the server in the cluster with a very streamlined bindings method invocation.

def unseal(self,
             unseal_keys: list) -> None:
      """auto-unseal the vault server"""
      if not self.__client.is_sealed():
          return

      self.__client.sys.submit_unseal_keys(keys=unseal_keys)

Conclusion

Now we have a custom Vault integration for your application developed in Python. This code can be used as a package within your application or other software so that it can independently connect, authenticate, authorize, and retrieve secrets with an existing Vault cluster in a self-contained procedure. The application is now capable of direct interfacing to Vault, and does not require workarounds using additional tooling. Once you feel comfortable implementing the functionality in this article, you can then expand to support additional client configs, and other authentication and secrets engines.

If your organization is interested in custom Vault integrations for your applications or other software you develop or otherwise utilize, contact Shadow-Soft.