Proposal: module location manifest for easy reuse

[thefotios]

While developing our new TF based infrastructure, we're trying to keep separate repositories of low level modules (1:1 with an AWS resource), higher order components (opinionated compositions of modules), and then our main infrastructure (environment/region specific permutations of components).

In doing this, we're noticing a significant number of modules are referenced by the same source location. While not impossible, it does become arduous to maintain all of the proper references as well as audit what references are being used.

I would like a propose a simple manifest based lookup for sources. I think there are a few benefits to this and I think it could be accomplished with minimal changes.

Here is what I envision:

Some sort of manifest file linking keys to a valid module source location. Relative sources would be resolved relative to the yaml file.

---
network_v1:   github.com/my-org/modules//network#v1
network_v2:   github.com/my-org/modules//network#v2
app_stack:    github.com/other-org/modules//app-stack#master
local_module: ./modules/foo

Then in the module declarations, they can simply be used by some sort of leading marker (in this case, I chose !)

module "network_foo" {
  source = "!network_v1"
  ...
}

module "network_bar" {
  source = "!network_v2"
  ...
}

module "my_app" {
  source = "!app_stack" 
  ...
}

In this scenario, when TF commands are run that parse the files, if they encounter that marker, they simply look it up in the manifest, choose the proper source ref, and continue on like normal. Once the code is checked out, this should be transparent to the rest of the workflow.

Anyway, this was just an idea that I had kicking around. There might be some details to work out, for instance is this some sort of explicitly agreed upon file name, or would there be some sort of data/provider block declaring it explicitly. I think this might also help pave the way for some of the package management discussion and at least help provide some simplification.

Related:

• Proposal: plugin manager #8347

[mitchellh]

Not a bad idea! Thanks for the thoughtful proposal. We actually will probably do something like this, but not exactly this in the future. We've actually been thinking through module versioning more carefully and while we don't have a solution for it yet, I'll add this to the set of things to consider while thinking through versioning.

Its still probably at least a couple major releases out. Thanks though! I'm going to close this because it'll otherwise probably bit-rot since we're too early stage for a meaningful discussion on it and I don't want to make other changes to the module system until we think through the major one.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.