Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

meta: Support Terraform Test Framework #1534

Open
dbanck opened this issue Jul 12, 2023 · 18 comments
Open

meta: Support Terraform Test Framework #1534

dbanck opened this issue Jul 12, 2023 · 18 comments
Assignees
Labels
enhancement New feature or request meta

Comments

@dbanck
Copy link
Member

dbanck commented Jul 12, 2023

Problem Statement

Terraform 1.6 will ship with an updated and finalised terraform test command for running tests against your Terraform files. Tests will use a new file extension (.tftest.hcl & .tftest.json) and introduce new run, assert, ... blocks. Right now, a user will not get any syntax highlighting or advanced IDE features.

User Impact

The test command and test file syntax are aimed at module authors who wish to validate and test their shared modules. However, it is also possible to use the test command to validate root modules. So any user writing tests for Terraform code will benefit from this.

Expected User Experience

Syntax Highlighting

When you open a .tftest.hcl file, it should be highlighted in a similar way to how Terraform is highlighted today. This should happen automatically, without the user having to set the file type manually. The vscode-hcl extension will provide some basic syntax highlighting for the files, but we can enhance this with support in vscode-terraform.

Completion

A user can use auto-completion within a test file. Triggering completion at the top level will bring up all available blocks. Triggering completion within a block will bring up all available attributes and child blocks.

Hover

Hovering over any block, attribute or value will display additional information, similar to working within Terraform files.

References

Within a test file, a user can refer to resources and modules. The go-to definition should work for all resource and module sources and open the appropriate definition when triggered.

Completion should also bring up the existing resources which are defined in the Terraform files.

Semantic Tokens

We send accurate semantic tokens for all identifiers in a test file to further improve highlighting.

Formatting

Similar to how we format .tf files, we should implement formatting for .tftest.hcl files.

VS Code Integration

The Testing API allows VS Code extensions to discover tests in the workspace and publish results. Users can execute tests in the Test Explorer view, from decorations, and inside commands.

CleanShot 2023-07-12 at 14 47 45@2x

Proposal

  • Syntax Highlighting
    • Create a new grammar for .tftest.hcl files
    • Include the grammar and language support in VS Code
  • Schema
    • Introduce a new schema for .tftest.hcl files
  • Language Server Support
    • Handle & parse .tftest.hcl files
    • Handle & parse .tftest.json files
    • Enable autocompletion for .tftest.hcl files
    • Enable hover for .tftest.hcl files
    • Collect references for .tftest.hcl files
    • Send semantic tokens for .tftest.hcl files
    • Formatting for .tftest.hcl files
  • Terraform VS Code extension

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment
@dbanck dbanck added the meta label Jul 12, 2023
@radeksimko
Copy link
Member

Related: #636

Maybe that issue can be broken down into the individual file formats but there may (or may not) be some higher-level work to do to support multiple formats easily/efficiently throughout all the libraries (hcl-lang + terraform-schema). Right now we support basically just *.tfvars and *.tf (JSON versions of those are mostly implied) and per the linked issue we may end up supporting many more, so we should ensure that this isn't just a matter of piling one edge case onto another. 😅

@prototype7630
Copy link

prototype7630 commented Oct 9, 2023

note: as per https://developer.hashicorp.com/terraform/language/tests#syntax the extension is either .tftest.hcl or .tftest.json

@bew
Copy link

bew commented Oct 20, 2023

Now that the test framework was released with terraform 1.6, I think this issue is more pressing as people want to use it effectively in vscode!

@martinOrigin8Cares
Copy link

Also don't forget the formatter. At the moment of writting this comment, the command terraform fmt work on tftest.hcl file. The format command of vscode should be able to support it also.

@martinOrigin8Cares
Copy link

Question: I'm a newbie on the internal of terraform in general but if the terraform-cli already parce the "test framework" file, this code cannot be re-user for the language server? or even the "tokenization" that need to be perform for the command terraform fmt ?

@dbanck
Copy link
Member Author

dbanck commented Oct 30, 2023

@martinOrigin8Cares thanks for reminding me about formatting, I've added it to the list.

Under the hood, both Terraform and terraform-ls, use https://github.com/hashicorp/hcl to parse/tokenize the Terraform HCL files. After that step both tools work a bit differently. terraform-ls supports any Terraform version >= 0.12 and we load the specific schema/features for the user's version.

In terms of formatting: We currently use terraform fmt internally to format files, by passing the file contents via stdin (terraform fmt -), but we don't currently support .tftest.hcl files as a whole. So, when support for test files lands, formatting will be one of the first features we implement.

@radeksimko radeksimko added the enhancement New feature or request label Oct 31, 2023
@dbanck dbanck self-assigned this Nov 6, 2023
@ghost
Copy link

ghost commented Nov 29, 2023

@dbanck Any progress on support for terraform test in the vscode extension?

I'm keen to help out if required - have dug into the code a bit, but not quite sure where to start.

@ghost
Copy link

ghost commented Nov 29, 2023

From what I can tell the following is required:

terraform-ls:

internal/terraform/ast/tests.go

Add AST for test files
// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0

package ast

import (
	"strings"

	"github.com/hashicorp/hcl/v2"
)

type TestsFilename string

func NewTestsFilename(name string) (TestsFilename, bool) {
	if IsTestsFilename(name) {
		return TestsFilename(name), true
	}
	return "", false
}

func IsTestsFilename(name string) bool {
	return strings.HasSuffix(name, ".tftest.hcl") ||
		strings.HasSuffix(name, ".tftest.json")
}

func (tf TestsFilename) String() string {
	return string(tf)
}

func (tf TestsFilename) IsJSON() bool {
	return strings.HasSuffix(string(tf), ".json")
}

type TestsFiles map[TestsFilename]*hcl.File

func TestsFilesFromMap(m map[string]*hcl.File) TestsFiles {
	mf := make(TestsFiles, len(m))
	for name, file := range m {
		mf[TestsFilename(name)] = file
	}
	return mf
}

func (tf TestsFiles) Copy() TestsFiles {
	m := make(TestsFiles, len(tf))
	for name, file := range tf {
		m[name] = file
	}
	return m
}

type TestsDiags map[TestsFilename]hcl.Diagnostics

func TestsDiagsFromMap(m map[string]hcl.Diagnostics) TestsDiags {
	mf := make(TestsDiags, len(m))
	for name, file := range m {
		mf[TestsFilename(name)] = file
	}
	return mf
}

func (td TestsDiags) Copy() TestsDiags {
	m := make(TestsDiags, len(td))
	for name, file := range td {
		m[name] = file
	}
	return m
}

func (td TestsDiags) AsMap() map[string]hcl.Diagnostics {
	m := make(map[string]hcl.Diagnostics, len(td))
	for name, diags := range td {
		m[string(name)] = diags
	}
	return m
}

func (td TestsDiags) Count() int {
	count := 0
	for _, diags := range td {
		count += len(diags)
	}
	return count
}

type SourceTestsDiags map[DiagnosticSource]TestsDiags

func (std SourceTestsDiags) Count() int {
	count := 0
	for _, diags := range std {
		count += diags.Count()
	}
	return count
}

internal/langserver/handlers/did_change_watched_files.go

// Line ~260
	if !ast.IsModuleFilename(fi.Name()) && !ast.IsVarsFilename(fi.Name()) && !ast.IsTestsFilename(fi.Name()) {
            ...
	}

vscode-terraform

src/extension.ts

activate > clientOptions > synchronize > fileEvents

       // Line ~82
        vscode.workspace.createFileSystemWatcher('**/*.tfvars'),
        vscode.workspace.createFileSystemWatcher('**/*.tftest.hcl'),
        vscode.workspace.createFileSystemWatcher('**/*.tftest.json'),

package.json

contributes > grammars

   // line ~113  
      {
        "language": "terraform-test",
        "scopeName": "source.hcl.tftest",
        "path": "./syntaxes/tftest.tmGrammar.json"
      }

syntaxes/tftest.tmGrammar.json

{
  "scopeName": "source.hcl.tftest",
  "name": "HashiCorp Terraform",
  "uuid": "d9db10d3-db70-48aa-8d44-f96ccbaa29f3",
  "fileTypes": [
    "tftest.hcl",
    "tftest.json"
  ],
  
  /// ???
  
}

Kind of stuck at the tftest.tmGrammar.json with the following:

  • Is the terraform.tmGrammar.json autogenerated, and can we generate a similar structure for the tftest.tmGrammar.json?
  • Due to the nested scope of some grammar in the tftest.hcl files, can we refer to the terraform.tmGrammar.json file or does the AST require that the embedded nodes are duplicated?
  • How do we effectively enable auto-complete for the references within assert scopes?
  • How do we effectively enable auto-complete for variables defined in both the test scope and the parent module scope?

@dbanck
Copy link
Member Author

dbanck commented Dec 11, 2023

Hi @nagytech!

Thanks for the offer to help! I'm actively researching and working on the Terraform test integration and we plan to ship it early next year.

It will also require changes in terraform-schema and hcl-lang and some refactoring in terraform-ls. I will ticket out the work over the next month and we can see if anything comes up that you want to contribute.

For the static grammar, we will probably reuse the existing Terraform grammar and extend the list of blocks. This way we avoid maintaining and shipping another grammar file.

PS: If you want to keep digging, there are some poc branches in the repositories.

@ghs

This comment was marked as off-topic.

@dbanck

This comment was marked as off-topic.

@novekm
Copy link

novekm commented Jan 23, 2024

@dbanck Hi Daniel, any updates on the status of this? Also happy to help if needed. Currently syntax validation is throwing errors for the new variables and run blocks, as well as other things included in the Terraform Test Framework added in v1.6.

At the minimum, is there a way to just disable the errors temporarily on my end? I'm trying to create an AWS workshop on using this new feature of Terraform (as part of an advanced pt.2 follow up to the Intro to Terraform on AWS Workshop, and currently I have to quit and re-open vscode every time I make any modification to the files. This makes it quite difficult to take any screenshots or make video explanations.

@dbanck
Copy link
Member Author

dbanck commented Jan 24, 2024

@novekm You shouldn't get validation errors for Terraform test files today, because the extension doesn't (yet) claim the new extension .tftest.hcl. Only if you have manually changed the file association for test files to Terraform, you will get these errors. We don't recommend this because the LS doesn't know how to handle test files (yet).

You can get syntax highlighting for test files by installing the HashiCorp HCL extension.

Hope this helps! :)

@HenriBlacksmith
Copy link

HenriBlacksmith commented Feb 1, 2024

You can add to the checklist the support for .tfmock.hcl files which have been added in 1.7 and the four new HCL blocks which go with that:

  • override_data
  • override_resource
  • mock_data
  • mock_resource

Also the content of those files is referenced in new mock_provider blocks in .tftest.hcl files.

@sjparkinson
Copy link

An interesting catch I hit in a project using terraform test.

We have a CI step to run terraform fmt -check -diff -recursive, which fails on unformatted .tftest.hcl files:

services/otel-graphite-collector/modules/otel_graphite_collector/tests/ecs_service.tftest.hcl
--- old/services/otel-graphite-collector/modules/otel_graphite_collector/tests/ecs_service.tftest.hcl
+++ new/services/otel-graphite-collector/modules/otel_graphite_collector/tests/ecs_service.tftest.hcl
@@ -6,7 +6,7 @@
   command = plan
 
   assert {
-    condition = aws_ecs_service.this.desired_count > 0
+    condition     = aws_ecs_service.this.desired_count > 0
     error_message = "The desired_count must be greater than 0."
   }
 

Exited with code exit status 3

But unfortunately the Terraform extension isn't formatting these files yet so I'm manually having to run terraform fmt -recursive in the project, when I do happen to remember!

@Taha-cmd
Copy link

It has been almost a year and Terraform is currently at version 1.9. Any ETA on this issue?

@sammcj
Copy link

sammcj commented Aug 27, 2024

Still seems to be broken / missing as of 1.9.5

@jpogran
Copy link
Contributor

jpogran commented Sep 6, 2024

I am happy to say that this work has been prioritized and parts of this has already been released in https://github.com/hashicorp/vscode-terraform/releases/tag/v2.32.3.

We now support syntax highlighting for Terraform Test and Terraform Mock files in v2.32.3. We will start to add new features in the coming weeks, and will update this ticket with links as they are delivered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request meta
Projects
None yet
Development

No branches or pull requests