docs: update aws and gcp self-managed page with terraform instructions #30846

kay-kim · 2024-12-16T21:04:29Z

https://preview.materialize.com/materialize/30846/self-managed/installation/install-on-aws/

running from module at root. The examples/simple requires couple of more changes (updates to the versions file, some node ami and instance types - to use the ARM one, etc).

https://preview.materialize.com/materialize/30846/self-managed/installation/install-on-gcp/

running from examples/simple, but assumes some changes in the examples/simple module. To test (need to change machine_type), locally added the following config to the main.tf materialize module block:

  gke_config = {
    node_count     = 1
    machine_type   = "e2-standard-4"
    disk_size_gb   = 50
    min_nodes      = 1
    max_nodes      = 2
    node_locations = []
  }

Future updates:

Per Refactor resource name definitions and Add Optional Helm Install Support terraform-aws-materialize#15 -- Sections B & C may soon be deleted + updates to tfvar section to specify different variables

kay-kim · 2024-12-16T21:08:43Z

doc/user/content/self-managed/installation/install-on-aws.md


-{{< /note >}}
+   ```bash
+   cd terraform-aws-materialize


wasn't sure if we wanted people to then go to the examples/simple directory (or whatever this gets renamed to).

kay-kim · 2024-12-16T21:13:58Z

doc/user/content/self-managed/installation/install-on-aws.md

@@ -4,7 +4,16 @@ description: ""
 robots: "noindex, nofollow"
 ---

-The following tutorial deploys Materialize onto AWS.
+Self-managed Materialize requires:


https://preview.materialize.com/materialize/30846/self-managed/installation/install-on-aws/

kay-kim · 2024-12-16T21:21:09Z

doc/user/content/self-managed/installation/install-on-aws.md

+- Install the AWS CLI. For details, see the [AWS
+  documentation](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html).
+
+- Configure with your AWS credentials. For details, see the [AWS


Wasn't sure if we should just give them the steps like aws configure or export AWS_ACCESS_KEY_ID ... But can, if you all prefer.

soft preference for what you have here already! seems much simpler to maintain

kay-kim · 2024-12-16T21:30:44Z

doc/user/content/self-managed/installation/install-on-aws.md

+   db_allocated_storage = 20
+   db_multi_az          = false
+
+   enable_cluster_creator_admin_permissions = true


Should we include:

enable_monitoring = false
metrics_retention_days = 0

makes sense. the monitoring doesn't come cheap, though in that case, I wonder if we want the defaults to be off

kay-kim · 2024-12-16T21:45:11Z

doc/user/content/self-managed/installation/install-on-aws.md

+
+    ```bash
+    terraform plan -out my-plan.tfplan
+    ```


If you prefer, I can also add a tip making sure that people did config their aws cli. (it's part of the prereq step above .... but, eh).

seems fine 👍

kay-kim · 2024-12-16T21:47:23Z

doc/user/content/self-managed/installation/install-on-aws.md

+1. Create a terraform plan and review the changes.
+
+    ```bash
+    terraform plan -out my-plan.tfplan


Wasn't sure if we want to create an output file or just do terraform plan.

kay-kim · 2024-12-16T21:49:25Z

doc/user/content/self-managed/installation/install-on-aws.md

-          cidrs: ["0.0.0.0/0"]
-        internal:
-          enabled: true
+            cidrs: ["0.0.0.0/0"]


gah! cursor and its helfpul indentations. Will update. sigh.

kay-kim · 2024-12-16T21:51:28Z

doc/user/content/self-managed/installation/install-on-aws.md

+
+   node_group_ami_type       = "AL2023_ARM_64_STANDARD"
+   node_group_instance_types = ["r6g.medium"]
+   node_group_desired_size   = 3


can make these more inline with the values in examples/simple.

Actually, I do need to make these like those. Otherwise, Insufficient cpu, 3 Insufficient memory.

kay-kim · 2024-12-16T22:00:12Z

doc/user/content/self-managed/installation/install-on-aws.md

+   - Install `kubectl`. See the [Amazon EKS: install `kubectl`
+     documentation](https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html).
+
+   - Configure `kubectl` to  connect to your EKS cluster, replacing:


Might pull the config out to a separate step to run after people create their environment.

pH14

Looks great! Really like how this is shaping up, thank you @kay-kim

pH14 · 2024-12-23T19:27:38Z

doc/user/content/self-managed/installation/install-on-aws.md

 module](https://github.com/MaterializeInc/terraform-aws-materialize/blob/main/README.md)
-to deploy a sample infrastructure on AWS with the following components:
+for evaluation purposes only. The module deploys a sample infrastructure on AWS
+(region `us-east-1`) with the following components:

 - EKS component
 - Networking component


not sure what "networking component" translates to. I think we can drop "component" from each of these lines:

- A dedicated VPC - A Kubernetes (EKS) cluster - An S3 bucket - An RDS Postgres cluster and database

Updated in patch

pH14 · 2024-12-23T19:28:40Z

doc/user/layouts/shortcodes/self-managed/requirements-list.html

@@ -0,0 +1,5 @@
+- A Kubernetes (v1.29+) environment.


I'd say cluster over environment

Updated in patch

pH14 · 2024-12-23T19:29:13Z

doc/user/layouts/shortcodes/self-managed/terraform-disclaimer.html

+The sample Terraform module is for **evaluation purposes only** and not intended
+for production use. It is provided to help you get started with Materialize for
+evaluation purposes only. Materialize does not support nor recommends this
+module for production use. Materialize does not guarantee tests for changes to


I do love the warning here, but the first three lines are all a little redundant... might be able to pare back

Updated in patch

pH14 · 2024-12-23T19:29:28Z

doc/user/layouts/shortcodes/self-managed/terraform-disclaimer.html

+for production use. It is provided to help you get started with Materialize for
+evaluation purposes only. Materialize does not support nor recommends this
+module for production use. Materialize does not guarantee tests for changes to
+the module.


not sure what "guarantee tests for changes" is

Updated in patch

pH14 · 2024-12-23T19:29:58Z

doc/user/layouts/shortcodes/self-managed/terraform-disclaimer.html

+module for production use. Materialize does not guarantee tests for changes to
+the module.
+
+For simplicity, this tutorial stores various secrets in a file as well as print


nit: the grammar here seems off

Updated in patch

pH14 · 2024-12-23T19:30:36Z

doc/user/content/self-managed/installation/install-on-aws.md

@@ -14,101 +19,237 @@ For testing purposes only. For testing purposes only. For testing purposes only.

 ## Prerequisites

+### Terraform


I can't comment on the line direction, but I think the "testing purposes only" above can go, now that terraform-disclaimer exists

Updated in patch

pH14 · 2024-12-23T19:32:04Z

doc/user/content/self-managed/installation/install-on-aws.md

-```bash
-kubectl get nodes
-```
+   - set your `database_password` to a secure password.


cc @bobbyiliev is this needed? could we auto-generate a password as part of the TF run?

pH14 · 2024-12-23T19:35:38Z

doc/user/content/self-managed/installation/install-on-aws.md

-   ```bash
-   cd materialize
-   ```
+   include the storage configuration your configuration file.  See the


nit: grammar here

Updated in patch

pH14 · 2024-12-23T19:36:47Z

doc/user/content/self-managed/installation/install-on-aws.md

+
+1. {{< warning>}}
+
+   {{< self-managed/terraform-disclaimer >}}


hammering home the disclaimer is important, but imo we don't need it a second time here

👍 Also, I guess this whole section may go away if we get Terraform to deploy materialize itself.

Updated in patch

kay-kim requested a review from a team as a code owner December 16, 2024 21:04

kay-kim requested a review from pH14 December 16, 2024 21:04

kay-kim commented Dec 16, 2024

View reviewed changes

kay-kim added 8 commits December 18, 2024 11:20

docs: update aws self-managed page with terraform instructions

fbcd4c0

fix yaml indentations

34693c6

change environmentd version from latest to v0.127.0

0644859

update node size to avoid insufficient cpu error

3fdf15a

update instance type to 2xlarge and config filenames

4204019

specify the service account name

efc2a15

add tip about emptying s3 bucket

97c8fa1

add gcp terraform instructions

f2b0def

kay-kim force-pushed the docs-self-managed-aws-terraform branch from c696409 to f2b0def Compare December 18, 2024 21:33

kay-kim changed the title ~~docs: update aws self-managed page with terraform instructions~~ docs: update aws and gcp self-managed page with terraform instructions Dec 18, 2024

kay-kim added 8 commits December 18, 2024 16:48

tweak

bd9624b

tweak to use tfvars from root of repo

bdd0fbd

tweaks

1004749

get terraform outputs

e2637c7

tweak

907a94e

fix numbering

3e9d27f

metadata database PostgreSQL

b1998fb

Updates

9e9ba85

pH14 reviewed Dec 23, 2024

View reviewed changes

kay-kim added 2 commits December 23, 2024 16:03

Updates per feedback

a1d37dc

additional tweaks per feedback

d5b8004

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: update aws and gcp self-managed page with terraform instructions #30846

docs: update aws and gcp self-managed page with terraform instructions #30846

kay-kim commented Dec 16, 2024 •

edited

Loading

kay-kim Dec 16, 2024

kay-kim Dec 16, 2024

kay-kim Dec 16, 2024

pH14 Dec 23, 2024

kay-kim Dec 16, 2024

pH14 Dec 23, 2024

kay-kim Dec 16, 2024

pH14 Dec 23, 2024

kay-kim Dec 16, 2024

kay-kim Dec 16, 2024

kay-kim Dec 16, 2024

kay-kim Dec 16, 2024

kay-kim Dec 16, 2024

pH14 left a comment

pH14 Dec 23, 2024

kay-kim Dec 23, 2024

pH14 Dec 23, 2024

kay-kim Dec 23, 2024

pH14 Dec 23, 2024

kay-kim Dec 23, 2024

pH14 Dec 23, 2024

kay-kim Dec 23, 2024

pH14 Dec 23, 2024

kay-kim Dec 23, 2024

pH14 Dec 23, 2024

kay-kim Dec 23, 2024

pH14 Dec 23, 2024

pH14 Dec 23, 2024

kay-kim Dec 23, 2024

pH14 Dec 23, 2024

kay-kim Dec 23, 2024

kay-kim Dec 23, 2024

		@@ -14,101 +19,237 @@ For testing purposes only. For testing purposes only. For testing purposes only.

		## Prerequisites

		### Terraform

docs: update aws and gcp self-managed page with terraform instructions #30846

Are you sure you want to change the base?

docs: update aws and gcp self-managed page with terraform instructions #30846

Conversation

kay-kim commented Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pH14 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kay-kim commented Dec 16, 2024 •

edited

Loading