Custom Vault Integrations: Go

Introduction

Hashicorp’s Vault has become the industry standard tool for secrets management. It comes packaged with multiple solutions for engines supporting various authentication methods, secrets integrations, and storage. However, not every application has a first-party, third-party, or community secrets engine integration with Vault. This is especially true for internally developed custom applications and software that your company may utilize internally, or as customer-facing products. In these situations you often need to develop a custom integration with Vault.

These articles will guide you in developing a basic custom secrets engine integration with Vault in each of four common languages. It is assumed that you are already familiar with the fundamentals of administrating a Vault cluster, the common Vault engines, and coding in the language(s) of your choice that are demonstrated in each article.

This article will focus on a custom Vault integration developed in Go.

Bindings Installation

Vault has a well documented REST API, and you can certainly interact with it through the REST API bindings in the application language. However, bindings to the Vault API already exist in several languages, and these should be used for their obvious advantages. In this section, we will see how to install those bindings in Go.

Since Vault is developed in Go, this is the easiest to accomplish. Simply add the Vault Go API bindings to your Go module file:

module github.com/org/repo

go 1.16

require (
  github.com/hashicorp/vault/api v1.3.1
)

and install it as a Go dependency as per normal. Note that this version of the bindings requires a minimum Go version of 1.16. You can downgrade to 1.2.0 if your version of Go is lower, but you will be restricted with respect to authentication engine choices.

The generated go documentation for the Vault API is also very helpful.

Client Construction and Authentication

The first step in interacting with the Vault API through the bindings is to construct a client for the interface. After constructing the client, the member method for authentication with a token can then be invoked. This code is actually remarkably similar across the different language bindings.

We begin with the necessary imports, and then declare a type struct as a convenience for the Vault options. We use the normal Vault environment variables for inputs, assign them to a Vault Config type struct for convenience, and initialize a client object from the Vault client constructor with the specified cluster address. We then authenticate the client with a Vault token.

package mypackage

import (
  "os"
  "fmt"

  vault "github.com/hashicorp/vault/api"
)

type VaultConfig struct {
  vaultAddr string
  token     string
}

func vaultClient() (*vault.Client, error) {
  vaultConfig := VaultConfig{
    vaultAddr: os.Getenv("VAULT_ADDR"),
    token:     os.Getenv("VAULT_TOKEN"),
  }

  client, err := vault.NewClient(&vault.Config{Address: vaultConfig.vaultAddr})
  if err != nil {
    fmt.Println("Vault client failed to initialize")
    return nil, err
  }

  if len(vaultConfig.token) != 26 {
    fmt.Println("The Vault token is invalid")
    return nil, err
  }
  client.SetToken(vaultConfig.token)

  return client, nil
}

If you are in a development/sandbox environment and do not want to deal with GNU TLS, then you can actually use a function associated with the *VaultConfig type struct to disable TLS while modifying the struct:

insecureVaultConfig := &vault.Config{Address: vaultConfig.vaultAddr}

err := insecureVaultConfig.ConfigureTLS(&vault.TLSConfig{Insecure: true})
if err != nil {
  fmt.Println("Vault TLS configuration failed to initialize")
  return nil, err
}

Note that using the standard environment variable VAULT_SKIP_VERIFY as an input for this will require a string to boolean conversion prior to input argument in the TLSConfig struct.

Alternate Authentication Engines

You may not want to use the basic Token Authentication engine (and understandably so) for authentication with the Vault client. In that situation, we can utilize a few alternatives.

In the below function, we demonstrate how to use AppRole authentication with the “push” method (“pull” would require an additional persona). Passing in the role id, secret id, and client reference returns a token according to the associated role and policy. There is no helper function in the bindings to the Vault API, so we must use the Logical to directly interface with the API. The Vault API then returns an already unwrapped token (remember: “push” method) as part of the response. This can be accessed within the ClientToken in the Auth of the client response.

func appRoleToken(roleID string, secretID string, client *api.Client) (string, error) {
  response, err := client.Logical().Write("auth/approle/login", map[string]interface{}{
    "role_id":   roleID,
    "secret_id": secretID,
  })

  if err != nil {
    fmt.Println(err)
    return "", err
  }

  return response.Auth.ClientToken, nil
}

If your Vault cluster runs on AWS and Vault has a role mapped to an AWS IAM role, then you can also use the associated IAM instance profile with the Vault AWS authentication engine. You must also add github.com/hashicorp/vault/api/auth/aws v0.1.0 to your go.mod (this requires Go >= 1.16) and import it appropriately. Assuming an already initialized client object, the function below will initialize AWS IAM authentication, and perform a Vault login authentication with the bindings to AWS IAM authentication. The authentication will bind to the AWS IAM role (the attached AWS IAM policies should provide appropriate AWS authorization outside of the Vault policies’ authorization).

import (
  "context"
  ...
  auth "github.com/hashicorp/vault/api/auth/aws"
)

func vaultClientAWSAuth(client *vault.Client) (*vault.Client, error) {
  awsAuth, err := auth.NewAWSAuth(auth.WithIAMAuth())
  if err != nil {
    return nil, errors.New("Unable to initialize AWS IAM authentication")
  }

  authInfo, err := client.Auth().Login(context.TODO(), awsAuth)
  if err != nil {
    return nil, errors.New("Unable to login to AWS IAM auth method")
  }
  if authInfo == nil {
    return nil, errors.New("No auth info was returned after login")
  }

  return client, nil
}

Secrets Retrieval (KV2 Engine)

Now that we have an authenticated and authorized client, we can begin the fun of retrieving secrets from the KV2 secrets engine. The code below demonstrates a logical read of the secret at a specified mount and secret path. Then, the secret is verified to actually contain secrets data at the specified path. Note that .Data is the bindings accessor for the actual secrets values in the returned value for the Read() (note also how both are public), and ["data"] is the top-level key corresponding to the values within the accessed secret for the KV2 engine. This coincidence can be confusing for those new to developing custom Vault integrations.

func retrieveKVSecret(mountPath string, secretPath string, client *vault.Client) (map[string]interface{}, error) {
  secret, err := client.Logical().Read(mountPath + "/data/" + secretPath)
  if err != nil {
    fmt.Println(err)
    return nil, err
  }

  secretMap, ok := secret.Data["data"].(map[string]interface{})
  if !ok {
    fmt.Printf("Retrieved type: %T, retrieved value: %#v\n", secret.Data["data"], secret.Data["data"])
    return nil, fmt.Errorf("the KV engine is version 1, and only version 2 is supported, or the retrieved value cannot be converted to map[string]interface{} (see above type and value)")
  }

  return secretMap, nil
}

Conclusion

Now we have a custom Vault integration for your application developed in Go. This code can be used as a package within your application or other software so that it can independently connect, authenticate, authorize, and retrieve secrets with an existing Vault cluster in a self-contained procedure. The application is now capable of direct interfacing to Vault, and does not require workarounds using additional tooling. Once you feel comfortable implementing the functionality in this article, you can then expand to support additional client configs, and other authentication and secrets engines.

If your organization is interested in custom Vault integrations for your applications or other software you develop or otherwise utilize, contact Shadow-Soft below.

Contact us - Vault

  • This field is for validation purposes and should be left unchanged.