Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nadia, OpenTofu, cdk8s #32

Merged
merged 8 commits into from
Oct 2, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .terraform-version

This file was deleted.

1 change: 1 addition & 0 deletions .tofu-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1.8.2
11 changes: 5 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,8 @@ less like the AWS Cloud Development Kit (CDK) and more like Terraform.

## Background
Helicopyter uses [CDKTF](https://github.com/hashicorp/terraform-cdk) and is inspired by
[Configerator](https://research.facebook.com/file/877841159827226/holistic-configuration-management-at-facebook.pdf),
[Terraformpy](https://github.com/NerdWalletOSS/terraformpy), and
[Terraform JSON configuration syntax](https://developer.hashicorp.com/terraform/language/syntax/json).
[Configerator](https://research.facebook.com/file/877841159827226/holistic-configuration-management-at-facebook.pdf) and
[Terraformpy](https://github.com/NerdWalletOSS/terraformpy).

## What Helicopyter does (goals)
- Name in the resource-prefix Terraform style, instead of the hash-suffix CDK style. This makes
Expand Down Expand Up @@ -67,7 +66,7 @@ Helicopyter uses [CDKTF](https://github.com/hashicorp/terraform-cdk) and is insp
-
-app.synth()
```
- Enable hand-written Hashicorp Configuration Language (HCL) files and auto-generated HCL/JSON to
- Enable hand-written and auto-generated Hashicorp Configuration Language (HCL) files to
co-exist, allowing incremental adoption.
- Separate object instantiation from synthesis, allowing Python script to import the objects/data
and do completely different things with them.
Expand All @@ -76,7 +75,7 @@ Helicopyter uses [CDKTF](https://github.com/hashicorp/terraform-cdk) and is insp

## What Helicopyter will probably never do (non-goals)
- Support languages other than Python
- Make use of the CDKTF's command line interface. Integration with it is untested and not recommended.
- Make use of CDKTF's command line interface. Integration with it is untested and not recommended.

## What Helicopyter might do in the future
- Support multiple backend configurations per codename
Expand All @@ -88,4 +87,4 @@ Helicopyter uses [CDKTF](https://github.com/hashicorp/terraform-cdk) and is insp
- Why do we need a Node.js server? Can we build dataclasses or Pydantic models out of the type annotations already being
generated?
- Provide helper classes or functions for useful but annoyingly verbose patterns such as local-exec provisioner command
- Backend / state file linter: prod must exist, and region/bucket/workspace_key_prefix/key must follow pattern
- Backend / state file linter, such as prod must exist, and region/bucket/workspace_key_prefix/key must follow pattern
4 changes: 2 additions & 2 deletions deploys/buddies/terraform/.terraform.lock.hcl

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions deploys/buddies/terraform/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
'duncan.tormey': ('DuncanTormey', 'admin'),
'james.braza': ('jamesbraza', 'admin'),
'matt.fowler': ('mattefowler', 'admin'),
'nadia.wallace': ('16NWallace', 'admin'),
}


Expand Down
5 changes: 5 additions & 0 deletions deploys/buddies/terraform/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,8 @@ resource "github_membership" "matt_fowler" {
role = "admin"
username = "mattefowler"
}

resource "github_membership" "nadia_wallace" {
role = "admin"
username = "16NWallace"
}
1 change: 1 addition & 0 deletions deploys/foundation/terraform/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
'airdjang': ('Airflow + Django', ['airflow', 'django', 'python']),
'allowedflare': ('Intranet connectivity for Django and more', ['django', 'python']),
'helicopyter': ('Python-defined infrastructure', ['ansible', 'cdktf', 'python', 'terraform']),
'wellplated': ('Python Django models for liquid handling', ['django', 'python']),
}


Expand Down
118 changes: 118 additions & 0 deletions documentation/djangocon-us-2024.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
## Why Write Down Infrastructure?
* Reproducibility
* Automation
* Disaster recovery

## Why Use Terraform?
* Simplicity
* Wealth of providers
* Good diffing

## Original CDKTF
```python
#!/usr/bin/env python

from constructs import Construct
from cdktf import App, TerraformStack
from cdktf_cdktf_provider_docker.image import Image
from cdktf_cdktf_provider_docker.container import Container
from cdktf_cdktf_provider_docker.provider import DockerProvider


class MyStack(TerraformStack):
def __init__(self, scope: Construct, ns: str):
super().__init__(scope, ns)

DockerProvider(self, 'docker')

docker_image = Image(self, 'nginxImage',
name='nginx:latest',
keep_locally=False)

Container(self, 'nginxContainer',
name='tutorial',
image=docker_image.name,
ports=[{
'internal': 80,
'external': 8000
}])


app = App()
MyStack(app, "learn-cdktf-docker")

app.synth()
```

## Diff from CDKTF to Helicopyter
```diff
--- documentation/learn_cdktf_docker.py 2024-08-05 08:47:30
+++ documentation/learn_helicopyter_docker.py 2024-08-05 08:47:25
@@ -1,28 +1,18 @@
-from cdktf import App, TerraformStack
from cdktf_cdktf_provider_docker.container import Container
from cdktf_cdktf_provider_docker.image import Image
-from cdktf_cdktf_provider_docker.provider import DockerProvider
-from constructs import Construct
+from helicopyter import HeliStack

-class MyStack(TerraformStack):
- def __init__(self, scope: Construct, ns: str):
- super().__init__(scope, ns)
+def synth(stack: HeliStack):
-
- DockerProvider(self, 'docker')
+ stack.provide('docker')

- docker_image = Image(self, 'nginxImage', name='nginx:latest', keep_locally=False)
+ docker_image = stack.push(Image, 'nginxImage', name='nginx:latest', keep_locally=False)

- Container(
- self,
- 'nginxContainer',
- name='tutorial',
- image=docker_image.name,
- ports=[{'internal': 80, 'external': 8000}],
- )
+ stack.push(
+ Container,
+ 'nginxContainer',
+ name='tutorial',
+ image=docker_image.name,
+ ports=[{'internal': 80, 'external': 8000}],
+ )
-
-
-app = App()
-MyStack(app, 'learn-cdktf-docker')
-
-app.synth()
```

## Helicopyter
```python
from cdktf_cdktf_provider_docker.container import Container
from cdktf_cdktf_provider_docker.image import Image
from helicopyter import HeliStack

def synth(stack: HeliStack):
stack.provide('docker')

docker_image = stack.push(Image, 'nginxImage', name='nginx:latest', keep_locally=False)

stack.push(
Container,
'nginxContainer',
name='tutorial',
image=docker_image.name,
ports=[{'internal': 80, 'external': 8000}],
)
```

## Uses
* Containers for Gunicorn web servers and Celery workers
* PostgreSQL database
* Load balancers
* Intranetworking (authenticating proxy)
* Git repository configuration
* Accounts and access to service providers
14 changes: 11 additions & 3 deletions helicopyter.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,17 @@
from subprocess import check_output
from typing import Any, TypeVar

from cdk8s import Chart
from cdktf import App, TerraformElement, TerraformStack
from constructs import Construct, Node
from tap import Tap


class HeliChart(Chart):
def __init__(self, cona: str) -> None:
super().__init__(App(outdir='.'), cona)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imo this will give us relative pathing issues, depending on where one invokes python

Can you do something like pathlib.Path(__file__).parent

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might not need to specify it. It was mindlessly copied over from cdktf, where it was specified to stop the library from creating an unused directory.

self.cona = cona

class HeliStack(TerraformStack):
def __init__(self, cona: str) -> None:
# Something is automatically creating outdir, which is cdktf.out by default
Expand Down Expand Up @@ -77,9 +83,9 @@ def push(
def multisynth(
all_or_conas_or_paths: Iterable[str],
*,
change_directory: Path | None = None,
hashicorp_configuration_language: bool = True,
format_with: str = 'terraform',
change_directory: Path | None,
hashicorp_configuration_language: bool,
format_with: str,
) -> None:
if not all_or_conas_or_paths:
print('No codenames specified. Doing nothing.')
Expand Down Expand Up @@ -146,6 +152,7 @@ def multisynth(
class Parameters(Tap):
conas: list[str] # space-separated COdeNAmes
directory: Path | None = None
format_with: str = 'terraform'
hashicorp_configuration_language: bool = True

def configure(self) -> None: # noqa: D102
Expand All @@ -158,5 +165,6 @@ def configure(self) -> None: # noqa: D102
multisynth(
args.conas,
change_directory=args.directory,
format_with=args.format_with,
hashicorp_configuration_language=args.hashicorp_configuration_language,
)
10 changes: 5 additions & 5 deletions includes.sh
Original file line number Diff line number Diff line change
Expand Up @@ -322,15 +322,15 @@ hta() {
return 1
fi
shift 2
python -m helicopyter "$cona"
TF_WORKSPACE="$envi" terraform -chdir="deploys/$cona/terraform" apply "$@"
python -m helicopyter --format_with="${INSH_TF:-terraform}" "$cona"
TF_WORKSPACE="$envi" ${INSH_TF:-terraform} -chdir="deploys/$cona/terraform" apply "$@"
}

hti() {
: 'Helper for Terraform Init'
local cona="${1?:Please provide a code name as the first argument}"
shift
terraform -chdir="deploys/$cona/terraform" init "$@"
${INSH_TF:-terraform} -chdir="deploys/$cona/terraform" init "$@"
}

htp() {
Expand All @@ -343,8 +343,8 @@ htp() {
return 1
fi
shift 2
python -m helicopyter "$cona"
TF_WORKSPACE="$envi" terraform -chdir="deploys/$cona/terraform" plan "$@"
python -m helicopyter --format_with="${INSH_TF:-terraform}" "$cona"
TF_WORKSPACE="$envi" ${INSH_TF:-terraform} -chdir="deploys/$cona/terraform" plan "$@"
}

pc() {
Expand Down
3 changes: 3 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,8 @@
},
"scripts": {
"prettier": "prettier --write"
},
"dependencies": {
"cdk8s-cli": "^2.198.227"
}
}
Loading
Loading