Packer: Secret Features Revealed

Introduction

After the release of version 1.8.0, Packer entered a maintenance and upkeep mode where the primary objectives were major bugs, and new features and medium or lower bugs are triaged. Part of this goal re-alignment means that multiple new features implemented in Packer 1.7.0 up to 1.8.0 are not present in the documentation. In this article, we will explore several of these secret recently-implemented features.

Note that since none of these features are documented, there is an associated inherent risk with leveraging any of them. However, it is worth noting that multiple Packer codebases are actively utilizing these algorithms in company environments and that multiple use cases with Packer become impossible without these features, so you may want to weigh that implication in your decision-making factors.

Plugin Filesystem Location

The default plugin filesystem location was changed in Packer 1.8.0. We will use Linux as an example for the remainder of this section.

The default location prior to 1.8.0 (and what is present in the documentation) was $HOME/packer.d/plugins. However, the new location is actually $HOME/.config/packer/plugins. Also packer init will leverage both locations, and while it prefers the new location, it will also manage plugin existence in the old location. Using the common method for verbose logging in Packer by setting the environment variable PACKER_LOG, we can observe further:

2022/12/07 12:20:43 Old default config directory found: $HOME/.packer.d
2022/12/07 12:20:43 [TRACE] discovering plugins in $HOME/.packer.d/plugins

What if we remove the old config directory?

2022/12/07 12:25:14 [TRACE] discovering plugins in $HOME/.config/packer/plugins

As an interesting side note: the plugins directory was a former valid location for installing a locally managed plugin. However, contrary to documentation this is no longer a valid search path for Packer, and you will need to install them to the cwd or the same executable path where the Packer statically linked binary is installed.

Optional Provisioner Blocks

Many people are already familiar with the algorithm for optional blocks in HCL2 in general, including especially optional blocks in Terraform. However, this becomes trickier for provisioner blocks in Packer because of their unique syntax relative to the other dynamic blocks. We can observe a general example for an algorithm below:

dynamic "provisioner" {
  for_each = <returns empty/non-empty>
  labels   = ["<provisioner name>"]

  content {
    ...
  }
}

A concrete example for a shell provisioner would be:

locals {
  my_bool = true || false
}

dynamic "provisioner" {
  for_each = local.my_bool ? ["this"] : []
  labels   = ["shell"]

  content {
    inline = ["echo foo"]
  }
}

In this manner we can construct optional provisioner blocks within build blocks in Packer.

Enumerable Source Blocks

This one actually comes to us courtesy of either Wilken, Adrien, or Sylvia (unclear who communicated it first). It also requires the prerequisite knowledge that it is possible to define a source block nested in a build block to append values to a “parent” source within a specific build.

With this in mind it is possible to do the following:

source "amazon-ebs" "this" {
  ...
}

build {
  dynamic "source" {
    for_each = ["one", "two"]
    labels   = ["amazon-ebs.this"]

    content {
      ...
    }
  }
}

While this may seem exciting and interesting initially, it is worth noting that limiting the iterable source block to appending values to “parent” source means that the actual functionality in real use cases is very limited. It also often either causes the code to become differently messy or more disorganized (especially given it is nested within build), or it becomes completely unfeasible. However, this is still another tool to keep in your kit if you discover a use case where it is actually beneficial. Until then we will maintain hope for support for enumerable non-nested blocks.

Role-Specific Ansible Variables

This next section is actually a limitation of HCL2 in general. Therefore, this also applies to other tools such as Terraform. In Terraform 0.12 a heterogeneous type map would cause a segmentation fault. This was corrected in 0.13 to throw an error. However, this does still pose the problem that values within a map in HCL2 must all be of the same type:

variable "my_map" {
  default = {
    good = {
      "foo" = "bar"
      "baz" = "bot"
    }
    error = {
      "foo" = true
      "baz" = ["one", "two"]
    }
  }
}

Note there is an uncommon but present misconception that the any type, or the map(any) complex type will overcome this limitation. The any type is interpreted at runtime, but it still must resolve to a singular type. This is also especially perplexing given how both struct and recasts to/from interface{} in Go do not possess this limitation, and could potentially be leveraged here (however ugly with the latter). One would also believe the introduction of generics in Go 1.18 could possibly overcome this limitation as well.

Therefore, if we wanted to do role-specific Ansible variables for the Ansible provisioner, we would expect to do something like:

variable "role_vars" {
  default = {
    "my-role" = {
      "foo"    = "bar"
      "baz"    = true
      "foobaz" = ["me", "too"]
    }
  }
}

yamlencode(var.role_vars) # ERROR!

However, this throws an error for the reasons explained above. Therefore, we would need to use literal YAML format strings:

variable "role_vars" {
  type    = map(set(string))
  default = {
    "my-role" = [
      "foo: bar",
      "baz: true",
      "foobaz: ['me', 'too']"
    ]
  }
}

and proceed as normal to use the variables in an included Ansible role.

Null Data

This final section is a feature implemented by Megan for completely different reasons initially that enabled an unexpected workaround to a bug. First, let us show a couple examples to get a feel for the functionality:

data "null" "my_string" {
  input = "a string"
}
# data.null.my_string.output --> "a string"

data "null" "my_bool" {
  intput = true
}
# data.null.my_bool.output --> true

Note that complex types are not allowed as outputs for null data. At first glance this probably appears to be a pointless feature. However, there is a bug in Packer whereby locals block scope variables are not allowed in a data block. This was to be fixed by Adrien developing a dependency hierarchy determination in Packer analogous to Terraform, but that was unfortunately not completed. For example, note the following difference below:

# terraform
data "provider" "one" {
  ...
}

resource "provider" "two" {
  argument = data.provider.one.id
}

# resource two dependency constructed onto data one
# packer
data "plugin" "this" {
  ...
}

source "plugin" "this" {
  argument = data.plugin.this.id
}

# no dependency constructed

Therefore, if you want to use a local variable in a data block, then it must be converted into a null data:

locals {
  my_var = "foo"
}

data "null" "my_var" {
  input = "foo"
}

data "plugin" "this" {
  argument = data.null.my_var.id # good
  argument = local.my_var # error
}

and thus Megan inadvertently saved the day.

Conclusion

In this article we explored multiple Packer features that have not yet received a corresponding update to the documentation. With these additional tricks in your arsenal, you should be able to expand the functionality of your Packer codebase, and reduce the technical debt. It is also quite possible that one or more of these will enable capabilities that previously seemed impossible.

If your organization is interested in codified management of your instance images for any cloud, virtual, or containerized platform, and to vastly reduce the difficulty of managing many servers into managing a single image, then contact Shadow-Soft below.

  • This field is for validation purposes and should be left unchanged.