Terragrunt issues should not be reported as module bugs

[GrantSheehan]

Hello Terragrunt,

It looks like the shiny new feature to allow pulling modules from the Terraform registry is broken on any modules pulled that have outputs deemed "sensitive".

See this issue from the GCP module I was trying to apply for context.

This is the module I'm attempting to use.

[yorinasub17]

I agree with @morgante on the comment here: terraform-google-modules/terraform-google-sql-db#229 (comment)

Terragrunt really shouldn't be used for modules that are not designed to be called as a root module. The primary use case of Terragrunt is to help with calling the root modules, but it does not have features to magically support modules that are not designed to be called directly, like the one you referenced. This is the same as Terraform Cloud and Terraform Enterprise.

The only reason terragrunt sort of allows it is because it is cost prohibitive to implement features in terragrunt that prevents one from pursuing that use case. In other words, if we had features that tell terragrunt what is designed to be called as a root module and what isn't, then we can easily implement a feature that converts a module from one that can't be called directly to one that can. I don't think we have any intention to implement a feature to support such a use case - the surface area would be large, and it would be hard to keep up with terraform changes like this one about sensitive outputs.

As @morgante mentioned in that issue, you should create a new wrapper module for that module and call that module in terragrunt instead of directly calling it. Just as it is an anti-pattern in terraform to call those kinds of modules, it is an anti-pattern in terragrunt as well to directly call submodules that are not designed for direct calling.

Closing this as a won't fix.

[ellisio]

@yorinasub17 So, from what I'm gathering by that response, is that you don't want someone sourcing the following from a terragrunt.hcl:

source = "terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster"

I'm honestly a little confused, as I see that all over the place in the registry itself in "examples", blog posts, learning material, etc. Can you please provide context where this is considered an anti-pattern for clarification? Might help others who stumble upon this PR once it starts being indexed by Google.

The reason I bring this up is that some modules, like the official GKE one above, expose these submodules as pre-defined configurations for their module to make consumption easy.

[morgante]

I don't use Terragrunt, so take this with a grain of salt. But maybe I can clarify the module development philosophy.

We absolutely do want you to use our modules (with pre-defined configurations) to make your life easier. We just don't think it's best to use them directly as your root module (ie. at the root of your Terraform state). When you leave them as the root module, it becomes hard to inject any of your own business logic.

This is why our example usage doesn't involve running git clone and applying directly. Instead we give you a usage snippet that should be copied into your own root or "wrapper" module.

[yorinasub17]

Apologies for the confusion here. Terragrunt does follow the same philosophy as what @morgante said right above:

We absolutely do want you to use our modules (with pre-defined configurations) to make your life easier. We just don't think it's best to use them directly as your root module (ie. at the root of your Terraform state). When you leave them as the root module, it becomes hard to inject any of your own business logic.

To clarify further, Terragrunt should be used to call modules that support being the root module. Note that some modules in the registry are designed this way (e.g., AFAIK, the root module of https://github.com/terraform-aws-modules/terraform-aws-vpc is one such module - @antonbabenko can correct me if I am wrong!), and some are not.

If you look at the Terragrunt example, you will see that the Terragrunt approach is actually based on two repos:

• A live infrastructure configuration that has your terragrunt code (e.g. terragrunt-infrastructure-live-example)
• A terraform module repo that has your wrapper modules that assemble modules into root modules (e.g., terragrunt-infrastructure-modules-example)

The infrastructure-modules repo is key here for taking modules from the registry and turning them into root modules to inject your business logic. This is the canonical way terragrunt is expected to be used.

However, sometimes, there are modules in the registry that you can actually use directly and supports being a root module (like the VPC one I mentioned above). This might be because you don't actually have a whole lot of business logic for that module, or the module exposes sufficient variables so that you can configure it without a wrapper. For those, it is extra code to maintain a wrapper module that just passes through the vars and duplicates the outputs. That's why we implemented the registry calling feature recently.

So what you can do or should do is really dependent on how the upstream module is designed. If the module supports being called as a root module, then you can use it directly with terragrunt. If not, then you need to use a wrapper module.

Ideally terragrunt can detect this and provide helpful error messages to guide the user, but unfortunately, terraform doesn't really give us a whole lot of classification tools to determine what modules can be used as a root module, so it's hard to implement.

I think a general rule of thumb to adhere to here is to assume that modules on the registry are not designed as root modules (since that is not directly supported by the terraform CLI), and that it is the exception for them to support being called directly as the root. If you know it can support being called as the root, then only then abandon the wrapper module.

---

FWIW, we (at Gruntwork) generally use the canonical approach mentioned above, where we have a specific repository dedicated to only holding root modules. We call these "service modules" and use that as a guide for our customers and internal users for what modules can be called directly by terragrunt. You can read more about this distinction in our blog post. This doesn't apply to the community at large, but hopefully gives some sense as to how we are thinking about what modules should be or should not be called directly (by classifying them explicitly as such).

[morgante]

Honestly, I think Terragrunt should avoid promoting the usage of modules from the registry as root modules. Looking at https://github.com/terraform-aws-modules/terraform-aws-vpc, the usage example clearly shows copying the module into your Terraform config - not attempting to apply it directly.

I don't want to get too much into the details here, but it's a continuous source of pain for module authors that Terragrunt/Gruntwork has a unique approach which diverges from the community at large. We continuously get issues which are only applicable to Terragrunt and would break with the design philosophy of Terraform at large.

If you are encouraging customers to compose multiple modules together in terragrunt.hcl then that effectively is your root module and IMO you should treat it as such.

[ellisio]

Thanks folks, appreciate the clarification.

[antonbabenko]

Hi guys!

Terminology is hard with all kinds of modules :)

My 5 cents. All terraform-aws-modules (including https://github.com/terraform-aws-modules/terraform-aws-vpc and most of the submodules placed in modules directory such as in terraform-aws-iam (the submodule name is not starting with an underscore like _templates)) are following the same principles:

1. terraform-aws-modules can be referenced in terragrunt.hcl without wrapping of any kind.
2. Sometimes it makes sense to bring custom logic or to call multiple modules at once, so then users make their own infrastructure modules and reference those in terragrunt.hcl.
3. Some modules like s3-bucket can be used to manage multiple resources at once, so the wrapping modules are also available there. This is a working solution for Terragrunt users because there is no way to use for_each with modules.

In general, I would be very surprised and question it if someone tells me that they have to make some wrappers for the Terraform AWS modules in order to be able to use those with Terragrunt.

[brikis98]

In the Terraform world, everything is called a "module" and that sometimes leads to confusion. Let's try to clarify that first.

As @yorinasub17 tried to clarify:

• Module: a small, reusable building block that is not meant to be applied directly. Sometimes called "sub-modules."
• Root module: combines modules, resources, providers, input vars, and output vars into the final end unit that is meant to be applied directly.

OK, so with the terminology out of the way, Terragrunt is designed to deploy root modules. This is exactly the same as Terraform Cloud, Terraform Enterprise, etc.

So, what does the Terraform Registry contain? The design of the Registry is that:

• In the root of a GitHub repo, you have a root module that can be applied directly.
• In the modules sub-folder, you typically have (non-root) modules that are not meant to be applied directly.
• In the examples sub-folder, you typically have other root modules that show examples of how to combine the (non-root) modules from the modules folder.

What's described above is the standard module structure recommended by HashiCorp. It's not fully enforced, so not every module author follows it, but it tends to be pretty common.

Honestly, I think Terragrunt should avoid promoting the usage of modules from the registry as root modules. Looking at https://github.com/terraform-aws-modules/terraform-aws-vpc, the usage example clearly shows copying the module into your Terraform config - not attempting to apply it directly.

I don't want to get too much into the details here, but it's a continuous source of pain for module authors that Terragrunt/Gruntwork has a unique approach which diverges from the community at large. We continuously get issues which are only applicable to Terragrunt and would break with the design philosophy of Terraform at large.

Terragrunt did not support deploying directly from the Registry until yesterday (see #1767), so I'm not sure what issues you "continuously" were getting? Also, we have only ever promoted Terragrunt as a tool for deploying root modules. As per what I wrote above, HashiCorp's official documentation, and @antonbabenko comment, the Terraform Registry should and does contain root modules, so using those with Terragrunt, or plain Terraform, or Terraform Cloud/Enterprise is totally fine. But, as we wrote above in the response to this comment, using Terragrunt (or any of the other tools) to try to directly apply non-root modules is not supported or encouraged.

[yorinasub17]

I don't want to get too much into the details here, but it's a continuous source of pain for module authors that Terragrunt/Gruntwork has a unique approach which diverges from the community at large. We continuously get issues which are only applicable to Terragrunt and would break with the design philosophy of Terraform at large.

I'm genuinely sorry to hear that Terragrunt is a source of pain for module authors. As @brikis98 mentioned, this is neither our intention nor encouraged. If you run into any issues filed by users about terragrunt usage that is counter to the module philosophy, please feel free to send them over to us and we will take the burden of informing how to work with terragrunt to use the module as is. The last thing we want is for Terragrunt to cause undue burden to the module maintainer community.

[lorengordon]

My 2 cents. I largely echo Anton's sentiments.

In general, I would be very surprised and question it if someone tells me that they have to make some wrappers for the Terraform AWS modules in order to be able to use those with Terragrunt.

I have seen one place where Terragrunt cannot deploy a module without a wrapper. And it does not matter whether it is a registry module or not, nor whether it is a submodule. The problem is when the module defines provider configs. I don't know about GCP, but with AWS this used to be common for cross-account and cross-region workflows, where a second, aliased provider is necessary. But as of Terraform 0.15 and the configuration_aliases attribute in the required_providers block, I think that issue can be addressed. Terragrunt can use a generate block to configure the provider, instead.

I'd also like to highlight what the Terraform docs say about nested modules, as it contradicts much of what is being said in this thread about submodules:

Nested modules. Nested modules should exist under the modules/ subdirectory. Any nested module with a README.md is considered usable by an external user. If a README doesn't exist, it is considered for internal use only.

From: https://www.terraform.io/docs/language/providers/configuration.html#alias-multiple-provider-configurations

[yorinasub17]

I have seen one place where Terragrunt cannot deploy a module without a wrapper.

There is one more place, which is the source of this ticket, where terraform now requires sensitive outputs from the resource to be marked as such in top level, root module outputs if they reference them. This one I don't think there is a terragrunt native solution for.

I'd also like to highlight what the Terraform docs say about nested modules, as it contradicts much of what is being said in this thread about submodules:

I think the primary issue at hand here is that a submodule may be marked for public consumption, but that doesn't mean it is public for use as a root module (that is, a module that can be applied directly without any wrappers). There are submodules whose primary intention is to be composed with other modules and should only be used within other terraform modules, using the module block, but not directly deployed (e.g., with terraform using -init-from-module, with terragrunt using tfr://, with TFC/TFE).

For those modules, it is accurate to say that terragrunt should not be used against them as they are intended to be composed in terraform. But it's also correct that those modules are public, because they can be publicly consumed within a terraform module block.

[morgante]

As everyone knows, module is a very overloaded term and these are the definitions I use:

1. Root module: the module where terraform apply is applied from—ie. the entrypoint into the Terraform configuration.
2. Public modules: modules in the registry which have a README (including submodules) that are designed to be publicly referencable directly from your Terraform configuration.
3. Private modules: submodules without a README which users should not directly use

As per what I wrote above, HashiCorp's official documentation, and @antonbabenko comment, the Terraform Registry should and does contain root modules, so using those with Terragrunt, or plain Terraform, or Terraform Cloud/Enterprise is totally fine.

I am not sure this is true. The registry definitely does include public modules (90% of our modules are public), but that doesn't make them root modules.

If they were designed as root modules, I would expect the usage documentation to say you should git clone the module then run terraform apply directly (as this is the only way to consume a root module in Terraform core). @antonbabenko Is that how you expect people to use your modules?

From my perspective, the usage documentation (including on the registry) clearly guides users towards referencing the module from their root configuration and not applying it directly:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.6.0"
  # insert the 19 required variables here
}

This doesn't necessarily require a "wrapper" module, but it does require users to have their own main.tf file at some point. For the most part, users of Terraform core get and understand this—it's only Terragrunt that encourages applying modules without ever having any main.tf of your own.

From my (likely naive) perspective, the terragrunt.hcl file effectively is operating as your root module in that it fulfills the same purpose as the root module in Terraform core—wiring together different modules, declaring providers, etc... As such, I wonder if it would make sense to make it behave more like a root module (ex. generating module blocks when you go to apply, instead of directly applying modules).

Terragrunt did not support deploying directly from the Registry until yesterday (see #1767), so I'm not sure what issues you "continuously" were getting?

I don't mean to bash Terragrunt, but it does introduce non-standard notions / requirements on top of Terraform that have led to issues over time. (Another example: Terragrunt requires all variables to have types, which is certainly a reasonable request but not something Terraform core required.) As a module author, my focus is on making sure it works with Terraform Core—every time Terragrunt introduces things which diverge from the core behavior, that leads to bugs which are harder for me to fix.

[antonbabenko]

@morgante terraform-aws-modules should be used as normal modules (without git clone). They can be used using all available options as @yorinasub17 described above.

You said that "Terragrunt requires all variables to have types". This is not true. Could you explain what makes you think so?

[morgante]

@morgante terraform-aws-modules should be used as normal modules (without git clone). They can be used using all available options as @yorinasub17 described above.

Right, so that sounds like you don't intend them to be used as root modules (same as terraform-google-modules).

You said that "Terragrunt requires all variables to have types". This is not true. Could you explain what makes you think so?

We got a series of issues about Terragrunt requiring type constraints about modules. It was a perfectly reasonable thing to add, but an example of an issue which only showed up for Terragrunt users.

[lorengordon]

There are submodules whose primary intention is to be composed with other modules and should only be used within other terraform modules, using the module block, but not directly deployed (e.g., with terraform using -init-from-module, with terragrunt using tfr://, with TFC/TFE).

Terraform has no explicit way to express this. Submodules may certainly be used as root modules.

To me, any module where terraform init && terraform apply does not work is pretty badly broken. And that's not a terragrunt problem.

[antonbabenko]

My English is bad :)

Modules can be used as all available options described above. All options are supported equally well. If a user wants they can do git clone but it is not required as long as they use terraform or terragrunt. Hope it makes sense now.

[lorengordon]

I don't mean to bash Terragrunt, but it does introduce non-standard notions / requirements on top of Terraform that have led to issues over time.

Strongly disagree here. In 5 years, I've seen nothing non-standard about how Terragrunt operates. Just users that fail to differentiate fully between Terragrunt and Terraform. Anything you can do with Terragrunt, you can do directly with Terraform. Terragrunt just makes it easier.

[morgante]

Terraform has no explicit way to express this. Submodules may certainly be used as root modules.

This has nothing to do with submodules, as I explained above. You're correct that Terraform has no way to enforce that a module should not be invoked directly, but all the documentation and usage suggests that you should have your own main.tf and not directly apply the module.

To me, any module where terraform init && terraform apply does not work is pretty badly broken. And that's not a terragrunt problem.

Strong disagree. I recommend consulting with someone from HashiCorp about this before making such assertions. Our modules are used in production for tens of thousands of customers. I'm pretty sure they're not "broken."

Terraform core explicitly treats root modules differently from referenced modules. See hashicorp/terraform#28472.

In 5 years, I've seen nothing non-standard about how Terragrunt operates. Just users that fail to differentiate fully between Terragrunt and Terraform. Anything you can do with Terragrunt, you can do directly with Terraform. Terragrunt just makes it easier.

It might not be a strictly technical problem, but I do think it's bad that Terragrunt promotes patterns which diverge from the Terraform community at large—including the source of this issue.

In the Terraform community at large, I almost never see users attempt to deploy without having a main.tf of their own. Yet that's exactly the pattern that Terragrunt encourages.

[lorengordon]

Terraform core explicitly treats root modules differently from referenced modules. See hashicorp/terraform#28472.

Indeed, they do treat root modules differently. But what is a root module? In the practical definition of a root module as it pertains to that PR, it is one where there is a terraform state. I.e. where you run terraform apply. It is not one that happens to be at the root of a git repo. It is unfortunate that there are two definitions of a root module.

[morgante]

But what is a root module? In the practical definition of a root module as it pertains to that PR, it is one where there is a terraform state. I.e. where you run terraform apply. It is not one that happens to be at the root of a git repo. It is unfortunate that there are two definitions of a root module.

Correct, and that is also the definition I am using.

My concern is that Terragrunt seems to encourage using modules (from the registry) directly as your root module instead of having a customer-owned main.tf. I've actually built several tools like Terragrunt internally which wrap Terraform, but the approach we always took was to auto-generate a root module instead of attempting to directly invoke modules.

[lorengordon]

And again, yes, any module should support running terraform init && terraform apply. Otherwise, the problem is not with Terragrunt, it is with how that module is self-constraining its own usage. Maybe you don't want to consider that "broken" but it is a self-imposed constraint and not a limitation of Terraform or Terragrunt.

[morgante]

And again, yes, any module should support running terraform init && terraform apply.

Are you asserting that any module should be used as a root module? You're free to make that assertion, but you are conflicting with HashiCorp and the community norms.

if that's the case, why does hashicorp/terraform#28472 draw any distinction between the two? Maybe @apparentlymart or @alisdair care to explain, but I think it's pretty clear that root modules are different from referenced modules.

[lorengordon]

Are you asserting that any module should be used as a root module?

"Should" is of course a strong word. I value flexibility and choice, so no I am not saying that at all. You are welcome to write your modules however you like! Maybe just also document which of your modules do and do not support being used as a root module, to help your users understand how they need to invoke them.

And I'll continue to write my modules in a way where they all work as a root module, so I don't have to do that, and so I don't get the same issues from my users that you get from yours.

[apparentlymart]

Hi all! I see I was tagged here and I'm not really equipped to get into a discussion about what Terragrunt does or does not do, or should or should not do, but I can at least confirm that there is a somewhat soft sense in Terraform of a "root module" being different than a "shared module". Some differences I can think of off the top of my head:

• The backend block belongs to the root module, because that's a setting for the configuration as a whole ("configuration" here meaning the root module and all of the descendant shared modules taken together) and not for each module separately.

  This distinction usually ends up being the main discriminator, because practical use of Terraform in production typically requires specifying a remote backend. (The default "local" backend is only useful for local development and experimentation.)

• provider blocks to configure providers belong in the root module, because provider configurations are a cross-module concept. (They either inherit automatically to child modules in the simple case, or we can pass them explicitly for more complicated situations.)

• Input variables for the root module come from various sources that the CLI layer manages. Input variables for shared modules come from a module call, represented as a module block either in the root module or in one of the other shared modules.

• Output values from the root module get saved as part of the new state snapshot after apply so that they're available for later reference. Output values from shared modules exist only transiently in memory during a Terraform run and get discarded once the run is complete.

  • This is also, indirectly, the reason for the exception in that PR core: Loosen output value sensitivity requirement hashicorp/terraform#28472, but for a pragmatic reason: the state file format was defined in terms of Terraform's old static sensitivity model rather than the modern dynamic sensitivity model, and so it can only remember whether a particular output value is sensitive as a whole, not whether individual parts of it are sensitive.

    But that doesn't need to apply to shared modules because their output values exist only transiently in memory. In fact, it mustn't apply to shared modules because a shared module isn't in control of the sensitivity of all values in its scope: callers can pass in sensitive values to any of the input variables, and that's what this PR was in response to, as we can see in v0.15 - Passing a sensitive value to a non-sensitive module variable causes an error if the module derives a non-sensitive output from that variable hashicorp/terraform#28431.

    This particular detail might be able to change in a future version of Terraform with an updated state snapshot format, but that can't come in the near future because it would break compatibility with existing terraform_remote_state usage in older Terraform versions, and we don't yet have a design to avoid that incompatibility.

• There's always only one "instance" of the root module, whereas shared modules you call can potentially be instantiated multiple times using for_each or count inside the module block. This means that a shared module often needs to provide additional mechanisms to allow the caller to ensure that multiple instances of the module don't collide when declaring objects that have uniqueness constraints.

As others noted above, Terraform doesn't strongly enforce this distinction due to a mixture of historical compatibility and pragmatic reasons. From a practical perspective though, I think it is worth thinking of these as two distinct categories of modules with different expectations, though I'll leave it to folks who work on Terragrunt to determine what (if anything) Terragrunt ought to do about that situation. Since Terragrunt includes features for modifying the root module source code before running Terraform CLI, I imagine that it could in principle try to make a shared module behave as a root module, though I don't know if that's desirable or practical.

("Root module" and "shared module" are terms we tend to use in new Terraform documentation, but the Terraform docs are large and many years old at this point so I'm sure there's some older documentation that we've not had a chance to revise recently. For the purpose of what I'm describing here, a "root module" is one designed for running terraform init and the other workflow commands, while a "shared module" is one designed to be called with a module block in another module. "Shared" can be a bit of a misnomer if a module is only called by one other module, but naming is always a bit of a fuzzy business. 🤷‍♂️ )

I hope that's helpful in some way!

[morgante]

@apparentlymart Thanks for weighing in, you summarized it better than I could. 😄 In general, the modules I develop on the registry are designed to be used as shared modules (not root modules).

[yorinasub17]

Yes that was a very good explanation. I think this is the key:

Since Terragrunt includes features for modifying the root module source code before running Terraform CLI, I imagine that it could in principle try to make a shared module behave as a root module, though I don't know if that's desirable or practical.

Terragrunt is really a preprocessor trying to turn shared modules into root modules by injecting the key components of backend, provider, and tfvars configurations. This wasn't always the case (and thus I haven't really looked at it in this light), but it has evolved into that.

Given that, I don't think the burden should be on module maintainers to support Terragrunt to do that. It is (greatly) appreciated that folks like @lorengordon and @antonbabenko make modules in that way, but the default should always be that Terraform module developers focus on supporting Terraform, and Terragrunt should work in those limits.

As I have been saying in this thread, it is not our intention to be a burden to the module community. Every case where Terragrunt is leading to issues should be kicked back here as a bug, and the answer is almost always going to be (a) Terragrunt will be updated to support that use case, or (b) create a wrapper module to use the upstream module correctly. It won't be "the upstream module needs to be updated" (NOTE: to be fair, I have said something similar to that in the past, but going forward I will refrain from saying similar things given this new light of what Terragrunt is really doing).

In this light, I'm going to reopen this ticket, as now I think there is something that Terragrunt can do here to improve the situation for module authors in regards to sensitive outputs to support another path of shared module -> root module transition.

[morgante]

Terragrunt is really a preprocessor trying to turn shared modules into root modules by injecting the key components of backend, provider, and tfvars configurations. This wasn't always the case (and thus I haven't really looked at it in this light), but it has evolved into that.

I think that's a very accurate summary. It's similar to what some of the internal tools I've built do; we synthesize a root module on the fly from multiple input modules + parameters. I'd be happy to give more info on our approach if it would be helpful.

For what it's worth, I genuinely do want Terragrunt to be able to use our modules. While we don't have the resources to test everything with Terragrunt, I'm very happy to help give advice or answer questions about the approach if there's any way I can be of assistance.

Thank you for the constructive engagement and support for the module community.

[lorengordon]

In this light, I'm going to reopen this ticket, as now I think there is something that Terragrunt can do here to improve the situation for module authors in regards to sensitive outputs to support another path of shared module -> root module transition.

I'm curious to see what you have in mind @yorinasub17 ... I feel like Terragrunt would need to inspect the source module and make changes to the source files directly, or inspect the source to detect all variables and outputs and generate a wrapper module. I don't think Terragrunt has those kinds of features currently (though maybe something could be hacked together using hooks)?

[yorinasub17]

I'm curious to see what you have in mind @yorinasub17 ... I feel like Terragrunt would need to inspect the source module and make changes to the source files directly, or inspect the source to detect all variables and outputs and generate a wrapper module. I don't think Terragrunt has those kinds of features currently (though maybe something could be hacked together using hooks)?

Yup this is why I marked as needs design. I have a vague idea for a short term solution (using the same tricks as the aws-provider-patch command), but I want to mull over it a bit before getting to work. I will most likely draft an RFC with my initial thoughts sometime next week. Stay tuned!

[brikis98]

Yes that was a very good explanation. I think this is the key:

Since Terragrunt includes features for modifying the root module source code before running Terraform CLI, I imagine that it could in principle try to make a shared module behave as a root module, though I don't know if that's desirable or practical.

Terragrunt is really a preprocessor trying to turn shared modules into root modules by injecting the key components of backend, provider, and tfvars configurations. This wasn't always the case (and thus I haven't really looked at it in this light), but it has evolved into that.

Hm, I feel like this is missing some important nuance/context, and as a result, may be a bit misleading. It makes it sound like the goal of Terragrunt is to take any shared module, and make it deployable as a root module. But I don't think that's the goal.

To use one of our own repos as an example, the consul-security-groups module is a shared module that is not meant to be deployed directly—not by Terragrunt or any other tool; you're supposed to combine it with other shared modules like consul-cluster and consul-iam-policies. That combination of shared modules creates a root module that can be deployed by Terragrunt, Terraform, Terraform Cloud, etc.

The real goal of Terragrunt comes from the fact that root modules in Terraform are hard to keep DRY. For example, if you have a shared module foo, and you want to create a root module to deploy foo into multiple different environments (e.g., dev, stage, prod), each with a slightly different config, then with Terraform, the two most common ways to do that are:

1. Create a root module that wraps foo, set the input parameters that foo needs in the dev environment, add a provider block, add a backend block, proxy through the output variables you care about, add a terraform_remote_state data source to read data from other modules, and so on. You then have to copy and paste this root module for every other environment (stage, prod, etc), which means you're often copy/pasting hundreds of lines of Terraform code and slightly tweaking dozens of input params, backend configs, provider configs, etc.
2. Alternatively, you create just a single root module as described in the previous option, but it's a partially configured root module. That is, you expose all the params that vary between environments as input variables—including various inputs to the foo module, provider block settings, terraform_remote_state data source settings, etc—so they can be provided externally via .tfvars files (e.g., dev.tfvars, stage.tfvars, prod.tfars), CLI args, or env vars. You also don't set many of the backend configs (this is also officially known as a partial configuration) so that too can be set externally, although not by variables, as Terraform doesn't support variables with backend blocks, but with .hcl files (e.g., dev-backend.hcl, stage-backend.hcl, etc) or CLI args. Finally, you create wrapper scripts for Terraform that add the appropriate -var-file, -backend-config, env var, etc params for each environment. This requires less copy/pasting of Terraform code, but requires that you create wrapper scripts, and always use those rather than using terraform directly.

The Terragrunt approach is to take the idea in item (2), but instead of creating custom wrapper scripts, you use Terragrunt as a canonical wrapper script.

So, it's not that Terragrunt encourages the use of shared modules as root modules, but that Terragrunt is a canonical Terraform wrapper script for deploying partially configured root modules. This is a bit of a mouthful...

The difference between a shared module and a partially configured root module may seem subtle—if you go far enough down the root of "partially configured," a root module will look an awful lot like a shared module—but it's still important to differentiate between these two, as a root module, including a partially configured root module, is designed to be applied directly (once you fill in a few parameters to "fully" configure it); on the other hand, many shared modules, such as consul-security-group-rules, are not designed to be applied directly, ever, and are only supposed to be used by wrapping them with a dedicated root module.

The "module" terminology in the Terraform world is quite confusing, so to reduce ambiguity, this is one of the reasons at Gruntwork we started using the terms modules and services (see here for the full details), where a module is a low-level, reusable building block NOT designed to ever be applied directly, whereas a service combines multiple modules and IS designed to be applied directly. So the distinction that matters is not whether you have provider and backend blocks, but whether the intent of the author was for the code to be applied directly or not.

So, using Gruntwork terminology, Terragrunt is designed to deploy services; not modules. That seems like less of a mouthful 😁

As I have been saying in this thread, it is not our intention to be a burden to the module community. Every case where Terragrunt is leading to issues should be kicked back here as a bug, and the answer is almost always going to be (a) Terragrunt will be updated to support that use case, or (b) create a wrapper module to use the upstream module correctly. It won't be "the upstream module needs to be updated" (NOTE: to be fair, I have said something similar to that in the past, but going forward I will refrain from saying similar things given this new light of what Terragrunt is really doing).

💯

If the author intended for their code to be applied directly, Terragrunt should make that as easy as possible; if the author did not intend their code to be applied directly, we should not encourage Terragrunt users to do so. We should do our best to make that distinction clearer.

[yorinasub17]

Given the evolution of the discussion here (and the involvement of many people who may not be interested in the original feature request but the general discussion about terragrunt and module community), I am going to break off the original feature request into a new ticket: #1808

Please follow that ticket to be notified when we work on the feature to support sensitive outputs. I'm going to go ahead and rename this ticket to be about the relation between Terragrunt and the module community as a whole.