ci: add basic integration testing framework

[obs-gh-nikhildua]

• Add "integration" folder which contains basic Integration testing framework
• Adds basic CI workflow for helm charts agent - integration tests
• Next iteration of this will contain real python tests against the cluster/helm-charts. Currently this performs a crude smoke test to ensure our helm-chart can be successfully installed in a cluster on any given commit in the PR process

Description

The location (helm-charts/integration) is intended to run integration tests using the terraform test framework both locally and in CI the exact same way.

The tests are run from the location using the terraform test -verbose command.

The tests ran are those located in the helm-charts/integrations/tests location as hcl files

The current agent-default.tftest.hcl file contains a basic integration test that will do the following in sequence

• Setup a local kind cluster (using docker) with one master node and two worker nodes. This uses the integration/modules/setup_kind_cluster module
• Install our agent helm chart charts/agent with default values (integration/modules/deploy_helm/values/default.yaml). This uses the integration/modules/deploy_helm module
• Runs a python script (scripts/test_basic.py) using pytest/kubeclient framework to check if all deployed agent pods are ready

Note: Since the helm chart deployment automatically waits until all pods are in READY state (within the timeout), the initial test in redundant. If a helm chart deployment fails, the python tests themselves will run to provide more information on what failed (eg: why the pods weren't READY within desired timeout)

Other modules (such as setup_aws_*) are not used in integration testing yet but the foundation for future real-cluster testing is setup when a deployment is needed in EKS.

Local Testing:

These tests can also be ran locally. See integration/README.md

[obs-gh-alexlew]

lgtm, not sure why all the AKS stuff is going in with this PR if that parts not ready yet but don't mind it really. just would make more sense for all of that to go in together

[obs-gh-alexlew]

where does the kind cluster run? on an ec2 instance? or on the actual docker runner?

[obs-gh-nikhildua]

Other modules (such as setup_aws_*) are not used in integration testing yet but the foundation for future real-cluster testing is setup when a deployment is needed in EKS.

I merged them in anyways since I had started off building the PoC with that but diverted to kind cluster instead. This just gives us the option to deploy to EKS at any time by anyone when needed without setting up a new cluster.

I can see it being later for more "real-world" e2e testing

[obs-gh-nikhildua]

where does the kind cluster run? on an ec2 instance? or on the actual docker runner?

It runs on the github actions machine itself. it sets up 2 docker containers for worker nodes and 1 container for master node