Custom Vault Integrations: Ruby

Introduction

Hashicorp’s Vault has become the industry standard tool for secrets management. It comes packaged with multiple solutions for engines supporting various authentication methods, secret integrations, and storage. However, not every application has a first-party, third-party, or community secrets engine integration with Vault. This is especially true for internally developed custom applications and software that your company may utilize internally or as customer-facing products. In these situations, you often need to develop a custom integration with Vault.

These articles will guide you in developing a basic custom secrets engine integration with Vault in each of the four common languages. It is assumed that you are already familiar with the fundamentals of administrating a Vault cluster, the common Vault engines, and coding in the language(s) of your choice that are demonstrated in each article.

This article will focus on a custom Vault integration developed in Ruby.

Bindings Installation

Vault has a well documented REST API, and you can certainly interact with it through the REST API bindings in the application language. However, bindings to the Vault API already exist in several languages, and these should be used for their obvious advantages. In this section, we will see how to install those bindings in Ruby.

Seth Vargo’s Vault bindings for Ruby are very robust and easily interfaced. Although the bindings have been very lightly maintained since around the time the KV2 engine was introduced, they are still high quality. Simply add the bindings to your Gemfile:

gem 'vault', '~> 0.16'

and install it as a Ruby dependency as per normal.

The rdoc documentation for these bindings is excellent. Of all the Vault bindings, the documentation for Ruby is absolutely the most extensive, and removes a great deal of ambiguity in usage and initial ramp-up time.

An important note for these Ruby bindings is that they frequently recognize the intrinsic Vault environment variables (e.g. VAULT_ADDR, VAULT_TOKEN, etc.) for Vault client configuration values.

Another great feature of the Ruby bindings is the multitude of helper methods for Vault functionality. With other bindings, one must often use direct logical actions against the API in the bindings. However, the Ruby bindings often wraps these logicals with convenient methods.

Client Construction and Authentication

The first step in interacting with the Vault API through the bindings is to construct a client for the interface. After constructing the client, the member method for authentication with a token can then be invoked. This code is actually remarkably similar across the different language bindings.

An easy and efficient procedure for initializing a Vault client in Ruby is through a class constructor. We can assign the client to an instance variable and therefore do not need to continually reconnect and re-authenticate to the Vault cluster. Note that the documentation also suggests using a singleton for this efficiency, but all methods then need to be encapsulated within that singleton.

We initialize a Vault client to the instance variable at the specified address. We then assign a CA certificate for SSL verification to the client using its appropriate member method. We also provide a member method for returning a client object authenticated with an input Vault token.

class Vault
  require 'vault'

  def initialize(address, ca_cert)
    @client = Vault::Client.new(address: address)

    raise "#{ca_cert} does not exist or is not readable" unless File.readable?(ca_cert)
    @client.ssl_ca_cert = ca_cert
    @client.ssl_verify = true
  end

  def token_auth(token)
    raise 'The Vault token is invalid' if token.length != 26
    @client.token = token
  end
end

Note there are some additional configuration options for SSL verification and connection. We can also assign a PEM file or contents to the client. We can configure a connection or SSL connection timeout for the client.

@client.ssl_pem_file = ssl_pem_file
@client.ssl_pem_contents = ssl_pem_contents
@client.timeout = timeout
@client.ssl_timeout = ssl_timeout

Note that these and many other methods are proxied through the Client class. One can also invoke methods and access values through the class directly e.g. Vault.token. These will execute external to the client. This can be useful when methods should be extricated from the instantiated client. However, the above connection and authentication configuration should, of course, be associated with a client object.

Alternate Authentication Engines

You may not want to use the basic Token Authentication engine (and understandably so) for authentication with the Vault client. In that situation, we can utilize a few alternatives.

In the below method, we demonstrate how to use AppRole authentication with the “push” method. Passing in the role id and secret id as input arguments configures an authenticated client according to the associated role and policy. We can use the approle helper method in the auth member to easily return a secret object. This secret object has an auth member with a client_token accessor to return the already unwrapped token (remember: “push” method). This token is then assigned to the client instance object for authentication.

def approle_auth(role_id, secret_id)
  secret = @client.auth.approle(role_id, secret_id)
  @client.token = secret.auth.client_token
end

In the below method, we demonstrate how to use AppRole authentication with the “pull” method. This is the newer procedure that is more secure than the “push” procedure and also requires multiple personas. The first persona executes the first method to create a secret id from the provided role id. The first persona must be authenticated with a policy that authorizes it to perform this action. The second persona then executes the second method to create a wrapped token from the generated secret id. The second persona must also be authenticated with a policy that authorizes it to perform this action. The token has a TTL of two minutes, and ideally, the secret id is also short lived for security reasons. The third persona then receives the wrapped token and unwraps it. This unwrapped token provides the authentication for the third persona’s authorization to perform the necessary interactions with Vault.

For the purposes of demonstration, the approle class’s member method create_secret_id is invoked from the base Vault class instead of the client object. This demonstrates usage where a configured client’s proxy to other classes is not necessary because we are simply establishing a role id and not configuring toward or from an existing client in any manner.

def approle_secret_id(role_id)
  @client.approle.create_secret_id(Vault.approle.role_id(role_id)).data[:secret_id])
end

def approle_wrapped_token(secret_id)
  secret = @client.auth.approle(@client.approle.role_id(role_id), secret_id)
  @client.token = secret.auth.client_token
  @client.auth_token.create(wrap_ttl: '120s')
end

def approle_unwrapped_token(wrapped_token)
  @client.token = @client.logical.unwrap_token(wrapped_token)
end

Secrets Retrieval (KV2 Engine)

Now that we have a connected, authenticated, and authorized client, we are able to begin interfacing with the KV2 secrets engine and retrieve secrets. The following method requires the secrets engine mount path (path preceding data) and the secrets path (path succeeding data) as inputs. First, the Vault cluster is checked for sealed status, and then an actual connection is initiated with the cluster. We make three attempts to retrieve the given secret object at the specified path. After retrieving the secret object, we check for the existence of data in the secret. Once this is confirmed, we return the secret key-value pairs. Similar to the Go bindings, note that .data is the bindings accessor for the actual secrets values in the returned value for the .read, and[:data] is the top-level key corresponding to the values within the accessed secret for the KV2 engine. This coincidence can be confusing for those new to developing custom Vault integrations.

If you need multiple secrets retrieved, then a more efficient implementation would be to iteratively retrieve them within the singleton. This should be preferred to multiple method invocations.

def kv2_read(secrets_mount_path, secrets_path)
  raise 'Vault is currently sealed.' if @client.sys.seal_status.sealed?

  @client.with_retries(Vault::HTTPConnectionError, Vault::HTTPError, Vault::HTTPClientError, Vault::HTTPServerError, attempts: 3, base: 0.05, max_wait: 5.0) do |attempt, except|
    warn "Received exception #{except} from Vault on attempt number #{attempt}." unless except.nil?

    secrets_path = File.join(secrets_mount_path, 'data', secrets_path)
    secret = @client.logical.read(secrets_path)

    raise "No secret found at #{secrets_path}." if secret.nil?

    secret.data[:data]
  end
end

There is actually an easy conditional to dynamically return the secrets hash depending upon whether the engine is KV1 or KV2:

secret.data.key?(:data) ? secret.data[:data] : secret.data

We can also use some Ruby recursion to read all of the secrets from a parent path. The below method will gather all of the secrets’ key-value pairs from a parent path. We input the same values for the paths to this method. We assign the list of secrets at this path to the variable secrets with a helper method. list_secrets (example code for this helper method not shown below). We initialize a hash to store all of the secrets’ key-value pairs and iterate over the retrieved list of secrets at the given path. We determine the nested path for the secret in this lambda iteration of the enumerable. If this is a bottom level secret, we then add the key-value pairs to the hash by passing the secret to the previously defined kv2_read method and merging the return. If this is not a bottom-level secret, then the path is input recursively to this method to continue delving into the tree and gather all of the secrets. The method will finally return the hash of the secrets’ key-value pairs.

def kv2_recursive_read(secrets_mount_path, secrets_path)
  secrets = list_secrets(path)
  aggregate_secrets = {}

  secrets.each do |secret|
    nested_secrets_path = File.join(secrets_path, secret)

    if list_secrets(nested_secrets_path).empty?
      aggregate_secrets.merge!({ nested_secrets_path.to_sym => kv2_read(secrets_mount_path, nested_secrets_path) })
    else
      aggregate_secrets.merge!(kv2_recursive_read(secrets_mount_path, nested_secrets_path))
    end
  end

  aggregate_secrets
end

Conclusion

Now we have a custom Vault integration for your application developed in Ruby. This code can be used as a package within your application or other software so that it can independently connect, authenticate, authorize, and retrieve secrets with an existing Vault cluster in a self-contained procedure. The application is now capable of direct interfacing to Vault and does not require workarounds using additional tooling. Once you feel comfortable implementing the functionality in this article, you can then expand to support additional client configs, and other authentication and secrets engines.

If your organization is interested in custom Vault integrations for your applications or other software you develop or otherwise utilize, you can contact Shadow-Soft to learn more about the added value we can deliver to your organization with these tools.