Formatting support for HCL

[joshsleeper]

historically I'd used vscode's "files.associations": {} setting to tie hcl files to the terraform extension to get simple formatting support, like so:

{
    "files.associations": {
        "*.hcl": "terraform"
    }
}

if there isn't already a plan to add formatting support to this extension I'd like to formally request one!

[jpogran]

This is the place to ask 😁

What kind of HCL are you writing? For example are you writing Packer or Boundary, etc.

In this first release we wanted to see where the need was before moving in one direction or another.

[joshsleeper]

atm I'm working with a combination of vault policy files and packer definition files

[radeksimko]

We could implement a product-agnostic formatting via hclfmt/hclwrite.Format(). The only downside is that this is likely to differ from e.g. packer fmt, vault policy fmt, waypoint fmt or other product-specific formatters.

[joshsleeper]

ahhhh, that's ringing some bells actually.

I remember asking for generic HCL formatting a year or two back and referencing the existing terraform fmt implementation, but was told that each tool-specific fmt was shockingly not-that-standardized.

help me understand a bit if you can: if they're all HCL2 (terraform, packer, vault policies, etc.) what makes the formatting so hard to standardize?
like, why aren't they all just calling a generic HCL2 formatter and passing it a list of their tool-specific keywords, since presumably the syntax should be identical across all of them, because... it's all HCL2?

to be clear, as much as I want a general-purpose formatter, I do not like the idea of having a low-level formatter that can easily conflict with the higher-level tool-specific formatters.

[apparentlymart]

The application-specific formatting tools should typically be a superset of the generic HCL formatting, by which I mean that running terraform fmt followed by generic HCL format should not undo anything that terraform fmt did. (assuming that both are using the same version of HCL, so the formatting rules are the same)

Taking Terraform's formatting command as an example, what it's doing is:

1. Parse using the hclwrite package's parser, producing a hybrid AST / physical syntax tree that we can make surgical updates to.
2. Use Terraform-specific logic to identify parts of the language where there is more than one way to write the same thing, and normalize those to be the idiomatic shape we recommend or illustrate in the documentation.
3. Use hclwrite's formatter to serialize the result back to HCL native syntax again.

A generic HCL format is essentially the same but without the step 2, and so running generic HCL format after running terraform fmt amounts to just repeating step 3 again, which should therefore produce the same result as long as the formatter is idempotent. (and if it isn't idempotent then that's a bug we should fix in hclwrite)

There may be other reasons that this wouldn't be desirable beyond my knowledge -- for example, I don't know how formatters provided for the same file by several different VSCode extensions interact with one another -- but at least from a pure HCL perspective it seems plausible that we could offer a generic subset of formatting in the generic extension without causing troubles for application-specific formatting done elsewhere. Generic formatting probably also applies generic syntax checking, which could perhaps allow for some basic "red squiggles" support in the generic HCL extension, for the subset of errors that HCL itself detects.

(Over in hashicorp/hcl#362 I previously discussed some similar tradeoffs about generic vs. application-specific validation. What I said there is still broadly true, but it might be that the generic HCL extension is sufficient motivation to revisit the priority tradeoff I mentioned there and offer a generic HCL format / validate as an installable package. It's not for me to say whether that would be the best approach to integrating this into this extension, though; I expect there are some other interesting strategies to consider such as compiling the relevant subset of HCL to portable WebAssembly and embedding it directly inside the extension. 🤔 )

---

To more directly answer the question "isn't it all just HCL?": unlike some other formats like JSON and YAML, a HCL file is more like a program to be executed than a data structure to be parsed, and so there's considerably more application-level interpretation to be done than you might be accustomed to with other grammars.

HCL is designed as a toolkit for building languages rather than as a language in its own right, but it's true that a bunch of the existing HCL-based languages aren't doing that much above what HCL itself offers, aside from defining their expected block types and attributes.

The languages that allow for e.g. creating relationships between declared objects via expressions, or writing "libraries" like Terraform's modules, will tend to bend HCL in more complicated ways than where HCL is being used mainly just as a serialization of a flat data structure. To be specific, I would expect the Terraform Language, the Packer Language and the Waypoint Language to all eventually benefit from application-specific extensions with their own formatters, but something like Vault's policy language or Consul's agent configuration files would probably suffice with a generic HCL extension and generic formatter.

[joshsleeper]

man, I love that you take the time to write out detailed responses like this @apparentlymart
big thanks for the amount of effort you put into explaining things to the community~

---

To be specific, I would expect the Terraform Language, the Packer Language and the Waypoint Language to all eventually benefit from application-specific extensions with their own formatters, but something like Vault's policy language or Consul's agent configuration files would probably suffice with a generic HCL extension and generic formatter.

this resonates with me perfectly, and a lack of a stable, supported packer extension for vscode combined with the recent launch of this generic HCL extension is actually what convinced me to file this in the first place.

To more directly answer the question "isn't it all just HCL?": unlike some other formats like JSON and YAML, a HCL file is more like a program to be executed than a data structure to be parsed, and so there's considerably more application-level interpretation to be done than you might be accustomed to with other grammars.

this also tracks for the most part, although I'm not entirely sure I can think of a good example. that is, HCL is still a syntax/language spec from what I understand, so I'm curious if you have a concrete example of something terraform or similar does that's truly extending the language syntactically as opposed to simply adding keywords that have specific meaning to the high-level tool.

Taking Terraform's formatting command as an example, what it's doing is:

1. Parse using the hclwrite package's parser, producing a hybrid AST / physical syntax tree that we can make surgical updates to.
2. Use Terraform-specific logic to identify parts of the language where there is more than one way to write the same thing, and normalize those to be the idiomatic shape we recommend or illustrate in the documentation.
3. Use hclwrite's formatter to serialize the result back to HCL native syntax again.

so, given this, I guess where that leaves me is the thought that hclwrite should maybe be enhanced to take some set of patterns/rules when called that define how to handle the tool-specific formatting in addition to the generic HCL formatting.
the end goal of that would be that tools built on top of hclwrite hopefully wouldn't need to deal with low level parsing of the AST at all?
that's all assuming that higher-level tooling can't literally add completely new syntax though, which I'm still unclear on if that's the case or not based on your description.

A generic HCL format is essentially the same but without the step 2, and so running generic HCL format after running terraform fmt amounts to just repeating step 3 again, which should therefore produce the same result as long as the formatter is idempotent. (and if it isn't idempotent then that's a bug we should fix in hclwrite)

also, I strongly agree with your interpretation here that hclwrite should likely be providing a non-conflicting proper subset of formatting that tool-specific formatters provide.
I'm unsure if it'd belong in each higher-level tool repo or the hclwrite repo, but if hclwrite isn't already being automatically tested to verify that it can successfully run against well-formed .tf / .pkr.hcl / etc. files it probably should be.

---

phew, that got sorta meta~
I guess maybe the most practical solution would be adding basic HCL formatting to the extension but also convincing it to defer to tool-specific extensions if they're present?
I can't immediately find any great docs explaining how to nicely handle formatter priority/deference with vscode extensions unfortunately...

[joshsleeper]

or maybe, very specific to this issue, the better takeaway could be to focus on creating/enabling well-supported extensions for the high-level tooling and not wasting time on adding format support to this one? ¯\_(ツ)_/¯

[acalvino4]

Looking for a formatting tool for hcl files, I came across this: https://marketplace.visualstudio.com/items?itemName=fredwangwang.vscode-hcl-format

it seems to rely on a hclfmt command, but it's not quite clear where I'm supposed to download it from. I only mention in case said tool can provide any insight into the implementation of this feature.

[mehdiMj-ir]

Hi, will hcl reformatting be supported on 2024?

[yiskaneto]

Looking for a formatting tool for hcl files, I came across this: https://marketplace.visualstudio.com/items?itemName=fredwangwang.vscode-hcl-format

it seems to rely on a hclfmt command, but it's not quite clear where I'm supposed to download it from. I only mention in case said tool can provide any insight into the implementation of this feature.

I did try to use https://marketplace.visualstudio.com/items?itemName=fredwangwang.vscode-hcl-format to format terragrunt.hcl files but even by downloading the binary it didn't work, at the end I end up using https://marketplace.visualstudio.com/items?itemName=jkillian.custom-local-formatters with the conf below:

{
    "[hcl]": {
      "editor.defaultFormatter": "jkillian.custom-local-formatters",
    },
    "customLocalFormatters.formatters": [
        {
          "command": "terragrunt hclfmt",
          "languages": ["hcl"]
        }
      ],
}

So to auto format you would need to press shift+alt+f on windows

[petr-tichy]

https://marketplace.visualstudio.com/items?itemName=jkillian.custom-local-formatters is great, but the command is expected to act like a filter, with input on stdin. So the relevant settings.json is

{
  "[hcl]": {
    "editor.defaultFormatter": "jkillian.custom-local-formatters",
  },
  "customLocalFormatters.formatters": [
    {
      "command": "terraform fmt -",
      "languages": [
        "hcl"
      ]
    }
  ]
}

[radeksimko]

It is true that hclfmt currently does not support formatting from STDIN. It looks like that may be one of the pre-requisites for implementing formatting support in the extension.

That said as was already mentioned above at length, terraform fmt may support STDIN-based formatting but it is not designed for formatting arbitrary HCL files. Just like packer fmt or vault policy fmt, they each implement product-specific logic which may not be relevant for other HCL files.

[brettcurtis]

I'm here because of the new native terraform test, hoping I can format with this since I'm a fan of the right click format option :) Or maybe the Terraform extension will be updated to support test files?

[dbanck]

I'm here because of the new native terraform test, hoping I can format with this since I'm a fan of the right click format option :) Or maybe the Terraform extension will be updated to support test files?

When terraform test support lands in the extension, it will include support for formatting test files. 🙂 We track the related work here: hashicorp/vscode-terraform#1534

[mdepot]

Borrowing the idea of vim modelines, what if we were to allow for a comment near the top of HCL files that provided hints about the specific type/flavor of the file? Something like # hcl: style:packer for example. This could be read by code editors and formatters, while being ignored by interpreters since it's just a comment. That would allow a monolithic formatter to act appropriately for the given file type, or it would allow handoff to a product specific formatter.

[radeksimko]

@mdepot The problem is not in detecting flavour/product - that can already be done relatively easily by reading the file extensions (such as *.tf, *.hcl.pkr etc.).

The problem is in scoping - there is potentially infinite number of HCL-based languages and it seems unrealistic that a single tool/extension would be able to format them all reliably/accurately. Product-specific problems should have product specific solutions.

I'm not implying that there can't be formatting support in vscode-hcl extension but it would likely be unable to format configuration according to product-specific rules.

[0xDones]

I use the following solution with the runOnSave extension:

    "emeraldwalk.runonsave": {
        "commands": [
            {
                "match": "\\.tf*",
                "cmd": "terraform fmt ${fileDirname}"
            },
            {
                "match": "\\.hcl",
                "cmd": "terragrunt hclfmt ${fileDirname}"
            }
        ]
    },