cleanup docs and add diagrams

README.md

@@ -1,21 +1,39 @@
-This modules makes it easy to extract terraform state at at the time of
+This module makes it easy to extract terraform state at the time of
 `terraform apply` and automatically update config files used by other
-systems outside of terraform but which depend on some infrastructure 
-state or settings. 
+systems or tools separate from terraform such as helm or kubectl, but 
+which depend on some infrastructure state or settings such as database
+addresses, security group ids, etc. 
 
-Currently, the module focusses on kubernetes and hence the docs are 
-written based on "stacks", "namespaces" and "config" files. This module 
-has been used in several closed source projects to generate helm values 
-files and kustomize files. This module is especially useful to those 
+This module has been used in several closed source projects to generate 
+helm values files and could surely be used to generate kustomize files, 
+ansible files, etc. This module is especially useful to those 
 not using the kubernetes or helm providers. 
 
-Using it is very simple, and best understood by looking in the
+The main concepts in this module "stacks", "namespaces" and "config" 
+files: 
+- a stack is an instance of a set of resources that describe a complete
+  environment. Eg a VPC, subnets etc, plus an EKS cluster, plus 
+  IAM, security groups, databases, lambdas, all for one environment, 
+  is a stack. There could be a dev stack, a staging stack, a prod 
+  stack. In the dev stack there could be multiple instances of an 
+  application, each for a different developer. The precise meaning 
+  will depend on your situation, but the term is used to refer to the
+  totality of a set of resources that logically belong together. 
+- a namespace is a subdivision of this stack into blocks. Think 
+  kubernetes, but this is just the term used for subdivision.
+- a config is a text file or set of text files that contain settings
+  that are required to do something outside of terraform with the 
+  resources of the stack/namespace. This could be the values files 
+  for a particular deployment of an "app" in a namespace of the 
+  stack (vpc + cluster).
+
+Using this is very simple, and best understood by looking in the
 examples folder. Simplest example: you have `YOUR_TF_MODULE/main.tf`, 
 and you need to get the AWS RDS DB address into the helm chart values 
 file of YOUR_APP_DIR which represents a container deployed in your 
 kubernetes cluster: 
 
-- In `YOUR_APP_DIR/config/_templates_/auto-root-values.yaml` 
+- In `YOUR_APP_DIR/config/_templates_/base-auto-root-values.yaml` 
   put the following: 
   ```
   user_url: http://${cluster_name}/user/id
@@ -36,25 +54,25 @@ kubernetes cluster:
     # the keys and values in template_vars are completely arbitrary:
     template_vars = {
       cluster_name  = "YOUR_CLUSTER_NAME"
-      sg_ingress    = aws_security_group.cluster_alb.id
-      aws_region    = var.region
+      sg_ingress    = "SOME_SG_ID"
+      aws_region    = "us-east-1"
     }
   }
   ```
 
-- In `YOUR_TF_MODULE`, run `terraform apply`: this will load the 
-  `YOUR_APP_DIR/config/_templates_/auto-root-values.yaml` 
+- In `YOUR_TF_MODULE`, run `terraform init` then `terraform apply`: 
+  this will load the `YOUR_APP_DIR/config/_templates_/base-auto-root-values.yaml` 
   file as a terraform template, replace all occurrences of the keys of 
-  `template_vars` by the corresponding values, and write the result 
-  to `YOUR_APP_DIR/config/stacks/YOUR_CLUSTER_NAME/auto-root-values.yaml`. 
+  `tmpl_vars` by the corresponding values, and write the result to 
+  `YOUR_APP_DIR/config/stacks/YOUR_CLUSTER_NAME/base-auto-root-values.yaml`. 
 
 This scales nicely:
 
 - you can have multiple stacks/clusters: each app will have the 
   correct values for all the different stacks they need values from, 
   in clearly recognizable folders. 
 - you can "specialize" the templates for individual namespaces (check 
-  out `k8s_ns` variable), and for individual clusters
+  out `namespace` variable), and for individual clusters
 - you can have a global config file at the root of the stacks folder
   (it is therefore static)
 - you can have a large number of services, you don't need to remember 
@@ -78,6 +96,7 @@ generated configs take precedence over any static ones. Since the
 files are known for each service, a helmfile can be useful to 
 capture the list of config files and their order. 
 
+
 Contributions
 =============
 

docs/configs-with-infra-data.drawio

@@ -0,0 +1 @@
+<mxfile host="Electron" modified="2021-03-15T03:36:52.390Z" agent="5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.4.3 Chrome/87.0.4280.141 Electron/11.3.0 Safari/537.36" etag="Q1G6qGhGY6MhEu5N92iK" version="14.4.3" type="device" pages="2"><diagram name="from tfstate" id="c7558073-3199-34d8-9f00-42111426c3f3">3Vldb5swFP01SNtDK9uEEB7btOuktVK1dmrz6AYH3AJGxjTJfv3sYAIOSUiVD6L1pfj6XmMfn2ufSyx7GM/uOE7DB+aTyELAn1n2jYUQtB1H/lOWeWFx+6AwBJz62qkyPNG/RBtLt5z6JDMcBWORoKlpHLMkIWNh2DDnbGq6TVhkvjXFAWkYnsY4alpfqC/CwuoAUNl/EhqE+s29suMNjz8CzvJEv85C9mTxV3THuBxK+2ch9tm0ZrJvLXvIGRPFUzwbkkhBW6JWxP3Y0LucNieJ2CXgN39Hfu/510N/5P35eH5Jnu+SC8/TkxPzEg/iS3h0k3ERsoAlOLqtrNeLRRM1LJCtyueesVQaoTS+EyHmeq9xLpg0hSKOdC+ZUfFaex6poS4d3bqZ6ZEXjXnZSASfv9YbtSjVrMIWrTKuWJ9a1EbctCljOR+TbWC5mp6YB0RscYSg2mCZN4TFRE5JRnISYUE/zalgTeBg6bcMfWRUThIBnWy2ZxchOtU8YI5QTEwHVVSQD7VZVKYFQb5AFgj0jD9xlOs1jDnBQqbv2bDIPXsaQbg7j9B6Hmk+XIBLu490Cu/JLQcODG4hsBu5qoFKRzaZZORIBHQaBJT88zulX3VwjeqEa2FfRbhROd6ZHWJe/yhnGAQmzyA87SFWLMugkCCc4wnjsVqduo/l7c38XNJklVYmaaYhFeQpxQu8p1IdmQTR7yFckNn2bduY4HBgHvg9jdW0EirlHRDWNEofbN4VA88vgzdogIfTNJp3kX+HPJFBb9ekcPdMiv3gd7dxt3OuriS22yTr4JRkheUNtiHVMyGVS+ew2XY7bK5zUthQAzZZe01ooKorGimxB77JBYvvZwCeeT7aoHPO9RrgXY0F4w2oZDmYqsc8jgoH+1pBQWVheo/fSPTIMiooS6TLGxOCxTWHq4gGqkOwFUhZLiKakOGyVAaHwdlp56h9Wpj7XVw5S/kGrSMVD8e/gNarMtRvUf/FHXq80hI2lYVS9p3u8kqJ6LZt8/qCFG0vSA8pY75QWO77gWK/9IVNIcNESLhVfPvr/F6BnkpqIyEGXV/LqKlmeJ6cU4Kc9BuKtyvVYaeSfTnR/1oQrMuO0wqCcgk1mGEDYjkQTTPSfoTgLC3gmtCZypSDnCmoXUShNZih42HWFProvDFbx7MDYSab1W8xhaypfu+yb/8B</diagram><diagram name="directly-from-terraform" id="b5wH2oGcDkkSYLAKmtvX">7VhdU+IwFP01zOw+4LQNBXlUUPdDd91hdpR9i21oo2lvJ02F+us3oelHKKKsgjqzvNCc3twkJ+fe3LSDRtHijOMkvACfsI5j+YsOGnccx0auK/8UkhfIoG8VQMCpr41qYEIfiAZLs4z6JDUMBQATNDFBD+KYeMLAMOcwN81mwMxRExyQFjDxMGujV9QXYYG6llXjXwgNQj0ycvWLG+zdBRyyWA/XcdBs+SteR7h0pe3TEPswb0DopINGHEAUT9FiRJiitmSN9r2rlE5n6feLrxc/g2/j2+SqWzg73aZLtUBOYvHKrvt6bSIv6SS+ZFc3gYsQAogxO6nR4yVnRLm1ZKu2OQdIJGhL8JYIkWup4EyAhEIRMf2WLKi4bjxPlasDV7fGC+152cjLRix4ft1sNHqpZt1t2Sr7FetTi1pRyxNkarsUMu6RDXaV5DEPyCaHysop9+aMQETkNGVHThgW9N6cHtYxEVR2VddLoHLijqXjt2fp8M2reDZdFBPTvWp9yIfGNGpoqZptFDQohrvHLNNr8DjBQmaE/8rak7LQemVphXStA9R3hoWzF6rNRZahNqf3PLXVjkpDmM1SshtFHrYUKQXpv6keaw1OmxJ8Qo+1BKelv4+gR3u4k0xnO6b2bHfPmW7Y0pUgnOMZ8EitThUCsmwAP5PaWdWaqaR5SAWZJHjJ91yWZaZq9DiEC7L4h718NBPYQ/OsGGpC53XZVEJho2Iq68F1W2WQvC2j5YY2GMVJwvK3iNSdR0/vudn8kT18dvS8bEvsTSJ/j6J2rJUjyXZbqj7cq6qdjXkiFbI4ep9Uov7gYKWa7LfJHLj7JBO1yJT3yBkN1E2RMlVlWp8kDeLzR6HUWUPpfvXZa1F65AngLQLlhTdRj1nECgN0rAii8up9jm8Iu4SUCgqxNLkBISBqGBwxGqgXAlaIhkwwGpNR9THA2iH7g6dTA9or9e5bHG1VQWl39nnB2f1Jt75OrKLpsTtKcVa/Rp34J3wI8K/zBx6fnWbJbdT/MYFuu6j5ndJYJazqGOgykDHSDUjc1UlsRRUqo5mbmAoOdzJsmIrDcQyxkobszFYgrAPPk3tE+JqIjKjvL3W1LiOaWttVWFbfKHIzBhthWX1dbMals31cymb9ha7Y4forKDr5Cw==</diagram></mxfile>

examples/complex/tf-prod/my-module/main.tf

@@ -0,0 +1,15 @@
+module "gen_helm_values" {
+  # source = "git::https://github.com/schollii/terraform-local-gen-files.git"
+  source = "../../../.."
+
+  stack_id     = "prod"
+  module_id    = "my-module"
+  config_roots = ["../my-micro-svc/deploy-config", "../my-micro-svc/test-config"]
+
+  # the keys and values in template_vars are completely arbitrary:
+  tmpl_vars = {
+    cluster_name = "prod"
+    sg_ingress   = "sg-1234567"
+    aws_region   = "us-east-1"
+  }
+}

examples/complex/tf-staging/my-module/main.tf

@@ -0,0 +1,15 @@
+module "gen_helm_values" {
+  # source = "git::https://github.com/schollii/terraform-local-gen-files.git"
+  source = "../../../.."
+
+  stack_id     = "staging"
+  module_id    = "my-module"
+  config_roots = ["../my-micro-svc/deploy-config", "../my-micro-svc/test-config"]
+
+  # the keys and values in template_vars are completely arbitrary:
+  tmpl_vars = {
+    cluster_name = "staging"
+    sg_ingress   = "sg-1234567"
+    aws_region   = "us-east-1"
+  }
+}