feat: Add an output option to filter output for custom run steps

[mightyguava]

what

Add an output argument to the run step used for custom commands that can post-process command output. Allows you to do something like

    plan:
      steps:
        - run:
            command: terragrunt plan -out $PLANFILE --terragrunt-ignore-external-dependencies
            output: strip_refreshing
        - run: terragrunt show -json $PLANFILE > $SHOWFILE

which will filter out Refreshing state... lines similar to how it's done when using the default terraform plan steps.

The valid options for output are:

• show - the default, current behavior of echoing the output
• hide - hide the output for the entire command. Useful for setup steps whose output isn't useful to show in the posted plan comment. Theoretically for the example above, it can be used for the terragrunt plan step, and then adding another terrarunt show step that outputs the plan in human-readable form.
• strip_refreshing - runs the StripRefreshingFromPlanOutput that currently runs for the default plan action

Alternatively, I also considered adding a different (or composable) option to run an arbitrary command to filter. E.g. maybe an outputFilter option that accepts a shell command similar to command. It would receive in stdin the full output of the run command to do arbitrary filtering on. Decided not to though since it'd be a bit more complex and may not really be needed.

Please let me know if this is a reasonable approach, and if so I'll polish it up and add docs!

why

This significantly reduces the verbosity of the plan output posted in PR comments for workflows that use custom commands, letting these workflows match the behavior of the default plan action.

There is a potential workaround per #3386 (comment), but doing so hides the filtered output from real-time streaming logs as well. The user would have no way of seeing these filtered out log lines even if they wanted to.

tests

Tested by running the unit tests and running in a real AWS account set up using terragrunt. The strip_refreshing option configured above works as expected, omitting everything before the last "Refreshing state..." marker :

Before this PR

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding hashicorp/aws versions matching ">= 2.49.0"...
- Installing hashicorp/aws v5.3.0...
- Installed hashicorp/aws v5.3.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!
aws_route53_zone.this["<REDACTED>"]: Refreshing state... [id=<REDACTED>]
aws_route53_zone.this["<REDACTED>"]: Refreshing state... [id=<REDACTED>]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

... truncated ...

After this PR

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

... truncated ...

The hide option also works as expected, completely hiding the output for the step it is set for.

references

• Closes Atlantis output in GitHub PRs unexpectedly showing "Initializing backend..." and "Refreshing state..." #3386

[mightyguava]

hi @runatlantis/maintainers, is anyone available to review this PR?

[GenPage]

I think this is perfect for now. A very clean and scoped approach. There is no need to run arbitrary commands and I'm hesitant to overload custom workflows that much.

[GenPage]

Please polish and add docs. I'll merge once complete.

[mightyguava]

Here's the updated docs. I changed the formatting a bit on the env one to be consistent too.

[jmreicha]

I saw this missed v0.25.0. Is there any estimate on when this will be released?

[mightyguava]

@jmreicha This was included in both v0.24.4 and v0.25.0. Not sure how it escaped the changelog though.

[jmreicha]

Ah, good deal. Thanks for the clarification!

[jamengual]

we have an issue to fix the autogenerated changelog , the filters are a bit wrong.

…

On Sat, Aug 12, 2023, 8:53 a.m. Josh Reichardt ***@***.***> wrote: Ah, good deal. Thanks for the clarification! — Reply to this email directly, view it on GitHub <#3518 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ3ERF44XI5EFU4WNG3VPTXU6RHZANCNFSM6AAAAAAZFSEILU> . You are receiving this because your review was requested.Message ID: ***@***.***>