DIY Vault Backup

Introduction

Hashicorp’s Vault is the industry standard for secrets management. In version 1.4 of Vault, the integrated storage backend supplied by Raft was promoted from beta to general availability. This Raft integrated storage backend has replaced Consul as the default and most popular choice for Vault storage.

Unless you are only using Vault in a development/sandbox lifecycle environment, then you have a disaster recovery and failover plan for Vault. Thankfully, the intrinsic clustering provided by the Raft gossip protocol ensures data replication within a cluster. However, you also will want to backup the Vault data external to the cluster to be safe.

Vault Enterprise ships with a fantastic tool for automated Raft backups. However, with some light knowledge of the Vault API, you can also create an automated Vault Raft backup tool yourself. In this article we will introduce an implementation satisfying the minimal functionality for creating your own automated backup software tool for Vault with the Raft storage backend.

Prerequsities

Environment

A local filesystem should exist with permissions and storage capable of writing a Raft snapshot. For the example shipping and storage, authentication and authorization should exist in AWS for listing, reading, and writing objects to a S3 bucket. It also requires a S3 bucket capable of storing the backup snapshot.

Vault

This implementation is primarily tested against Vault 1.8 and 1.9, but should also work with other minor release versions of 1.4 and later. A Vault server cluster with Raft integrated storage should exist and enable connections. Authentication and authorization is required for the Raft backup capability. A simple Vault policy for the authorization appears like:

path "sys/storage/raft/snapshot" {
  capabilities = ["read"]
}

Golang

This implementation requires Golang version 1.16 or later. The Go module file also requires github.com/hashicorp/vault/api v1.3.1 or later, and an optional github.com/aws/aws-sdk-go v1.42.11 or later if you are integrating with AWS for the example shipping and storage.

Vault Raft Backup

We will assume that we have an authenticated and authorized client. You may need to consult the Vault documentation for basic authentication engine configuration, and basic policy attaching for authorization. A detailed reference for client initialization in Golang can be found in the previous article titled Custom Vault Integrations: Go.

We can use the following function to create a backup Raft snapshot and store it on the local filesystem.

// vault raft snapshot creation
func vaultRaftSnapshot(client *vault.Client, snapshotPath string) (*os.File, error) {
    // prepare snapshot file
    snapshotFile, err := os.OpenFile(snapshotPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644)
    if err != nil {
        fmt.Println("snapshot file at " + snapshotPath + " could not be created")
        fmt.Println(err)
        return nil, err
    }

    // defer snapshot close
    defer snapshotFileClose(snapshotFile)

    // execute raft snapshot
    err = client.Sys().RaftSnapshot(snapshotFile)
    if err != nil {
        snapshotFile.Close()
        fmt.Println("Vault Raft snapshot invocation failed")
        fmt.Println(err)
        return nil, err
    }

    return snapshotFile, nil
}

// close snapshot file
func snapshotFileClose(snapshotFile *os.File) {
    // close file
    err := snapshotFile.Close()
    if err != nil {
        fmt.Println("Vault raft snapshot file failed to close")
        log.Fatalln(err)
    }
}

This function returns a pointer to the file where the Raft backup snapshot is stored on the local filesystem. Please note that the file must be re-opened if you want to ship it to be stored somewhere else. Alternatively, the file close could be deferred until after the backup is shipped.

Backup Shipping and Storage

Now we have a Vault Raft snapshot ready to be shipped and stored somewhere else if so desired. As an example, we will explore shipping to and storing in a S3 bucket. We need to re-open the snapshot file before shipping, but otherwise this code follows the common pattern for uploading a file to S3.

// snapshot upload to s3
func snapshotS3Upload(config *AWSConfig, snapshotPath string) (*s3manager.UploadOutput, error) {
    // open snapshot and defer closing
    snapshotFile, err := os.Open(snapshotPath)
    if err != nil {
        fmt.Printf("Failed to open snapshot file %q: %v", snapshotPath, err)
        return nil, err
    }
    defer snapshotFileClose(snapshotFile)

    // aws session
    awsSession := session.Must(session.NewSession(&aws.Config{
        Region: aws.String(config.s3Region),
    }))

    // initialize an uploader with the session and default options
    uploader := s3manager.NewUploader(awsSession)

    // determine vault backup base for s3 key
    snapshotPathBase := filepath.Base(snapshotPath)

    // upload the snapshot to the s3bucket at specified key
    uploadResult, err := uploader.Upload(&s3manager.UploadInput{
        Bucket: aws.String(config.s3Bucket),
        Key:    aws.String(config.s3Prefix + "-" + snapshotPathBase),
        Body:   snapshotFile,
    })
    if err != nil {
        fmt.Println("Vault backup failed to upload to S3 bucket " + config.s3Bucket)
        fmt.Println(err)
        return nil, err
    }

    return uploadResult, nil
}

Now we have code to ship and store the Vault Raft backup snapshots in a dedicated S3 bucket. Since the S3 bucket could begin to grow quite large with these backups, it would be recommended to cleanup the S3 bucket with a lifecycle policy. It would not be a good idea to incorporate that functionality into this tool, because then an entire scheduler would need to be coded and integrated also.

Conclusion

Now you have the core code functionality for a software tool to create backup Vault Raft snapshots, and ship them to a dedicated external storage solution. It is now possible to expand and extend this implementation for more functionality. It is also easy to wrap this software tool inside your choice of automation tool to ensure scheduled customized backups. While this solution is not on par with the tooling in Vault Enterprise, it does provide a solid homegrown solution if you need one.

If your organization is interested in extensions, custom tooling, and custom integrations with Vault and other secrets management tools, then contact Shadow-Soft.