Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Replace yaml manifests with k8s resources and map implicit dependencies #639

Closed
wants to merge 9 commits into from
Closed

refactor: Replace yaml manifests with k8s resources and map implicit dependencies #639

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
1f55f0c
Managed addon
Jun 7, 2022
abfab09
Split adot manifest
bonclay7 Jun 12, 2022
be3fb17
Fix modules dependencies
bonclay7 Jun 12, 2022
16dac45
Replicate fixes for all samples
bonclay7 Jun 13, 2022
35d069e
Pre-commit fixes
bonclay7 Jun 13, 2022
05c522e
Set region
bonclay7 Jun 14, 2022
89a6c8d
refactor: Replace yaml manifests with k8s resources and map implicit …
bryantbiggs Jun 15, 2022
2060b13
fix: Updates from testing and validation
bryantbiggs Jun 15, 2022
5fa493f
feat: Quick sync with other examples, still need to be validated
bryantbiggs Jun 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.2.0
rev: v4.3.0
hooks:
- id: trailing-whitespace
args: ['--markdown-linebreak-ext=md']
Expand Down
197 changes: 70 additions & 127 deletions examples/observability/adot-amp-grafana-for-haproxy/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,170 +10,113 @@ Amazon Managed Grafana.
This example provides a curated dashboard along with Alerts and Rules on
Amazon Managed Prometheus configured as a default data source on Managed Grafana.

---
#### ⚠️ API Key

**NOTE**
The Grafana API key is currently handled in this example through a variable until [native support is provided](https://github.com/hashicorp/terraform-provider-aws/issues/25100).
Users can store the retrieved key in a `terraform.tfvars` file with the variable name like `grafana_api_key="xxx"`, or set the value through an environment variable
like `export TF_VAR_grafana_api_key="xxx"`when working with the example. However, in a current production environment, users should use an external secret store such as AWS Secrets Manager and use the
[aws_secretsmanager_secret](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/secretsmanager_secret) data source to retrieve the API key.

For the sake of simplicity in this example, we store sensitive information and
credentials in `variables.tf`. This should not be done in a production environment.
Instead, use an external secret store such as AWS Secrets Manager and use the
[aws_secretsmanager_secret](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/secretsmanager_secret) data source to retrieve them.
## Prerequisites

---
Ensure that you have the following tools installed locally:

## How to Deploy
1. [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
2. [kubectl](https://Kubernetes.io/docs/tasks/tools/)
3. [terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli)

### Prerequisites
## Deploy

- Terraform
- An AWS Account
- kubectl
- awscli
- jq
- An existing Amazon Managed Grafana workspace.
This example deploy all the necessary components to start monitoring your HAProxy
applications. You can follow the steps below to build and deploy an example
application to populate the dashboard with metrics.

To provision this example:

1. Provision the Grafana workspace first; we need to retrieve the key after creation before we can proceed with provisioning:

```sh
terraform init
terraform apply -target=module.managed_grafana # required to retrieve API key before we can proceed
```

Enter `yes` at command prompt to apply

#### Generate a Grafana API Key
2. Generate a Grafana API Key

- Give admin access to the SSO user you set up when creating the Amazon Managed Grafana Workspace:
- In the AWS Console, navigate to Amazon Grafana. In the left navigation bar, click **All workspaces**, then click on the workspace name you are using for this example.
- Under **Authentication** within **AWS Single Sign-On (SSO)**, click **Configure users and user groups**
- Check the box next to the SSO user you created and click **Make admin**
- Navigate back to the Grafana Dashboard. If you don't see the gear icon in the left navigation bar, log out and log back in.
- From the workspace in the AWS console, click on the `Grafana workspace URL` to open the workspace
- If you don't see the gear icon in the left navigation bar, log out and log back in.
- Click on the gear icon, then click on the **API keys** tab.
- Click **Add API key**, fill in the _Key name_ field and select _Admin_ as the Role.
- Copy your API key into `dev.tfvars` under `grafana_api_key`
- Add your Grafana endpoint to `dev.tfvars` under `grafana_endpoint`. (ex `https://<workspace-id>.grafana-workspace.<region>.amazonaws.com/`)
- Copy your API key into `terraform.tfvars` under the `grafana_api_key` variable (`grafana_api_key="xxx"`) or set as an environment variable on your CLI (`export TF_VAR_grafana_api_key="xxx"`)

### Deployment Steps
3. Complete provisioning of resources

- Clone this repository:

```
git clone https://github.com/aws-ia/terraform-aws-eks-blueprints.git
```sh
terraform apply
```

- Initialize a working directory
Enter `yes` at command prompt to apply

```
cd examples/observability/eks-cluster-with-adot-amp-grafana-for-haproxy
terraform init
```

- Fill-in the values for the variables in `dev.tfvars`
- Verify the resources created by this execution:
## Validate

```
export AWS_REGION=<ENTER YOUR REGION> # Select your own region
terraform validate
terraform plan -var-file=dev.tfvars
```
The following command will update the `kubeconfig` on your local machine and allow you to interact with your EKS Cluster using `kubectl` to validate the deployment.

- Deploy resources:
1. Run `update-kubeconfig` command:

```
terraform apply -var-file=dev.tfvars --auto-approve
```sh
aws eks --region <REGION> update-kubeconfig --name <CLUSTER_NAME>
```

- Add the cluster to your kubeconfig:

```
aws eks --region $AWS_REGION update-kubeconfig --name aws001-preprod-observability-eks
```

`terraform apply` will provision all the aforementioned resources.

---

**NOTE**

This example deploy all the necessary components to start monitoring your HAProxy
applications. However, you can follow the steps below to build and deploy and example
application.

---

#### Verify that the Resources Deployed Successfully

- Verify that Amazon Managed Prometheus workspace was created successfully:

- Check the status of Amazon Managed Prometheus workspace through the AWS console.

- Check that OpenTelemetry Collector is running successfully inside EKS:
2. Test by listing all the pods running currently:

```
kubectl get pods -A

NAMESPACE NAME READY STATUS RESTARTS AGE
cert-manager cert-manager-7989877dff-jxk57 1/1 Running 0 160m
cert-manager cert-manager-cainjector-7d55bf8f78-jcc6d 1/1 Running 0 160m
cert-manager cert-manager-webhook-577f77586f-m6mlg 1/1 Running 0 160m
kube-system aws-node-kvbdl 1/1 Running 0 3h36m
kube-system aws-node-lv4g4 1/1 Running 0 3h36m
kube-system aws-node-x8zcs 1/1 Running 0 3h36m
kube-system coredns-745979c988-bhtx6 1/1 Running 0 3h42m
kube-system coredns-745979c988-ktdlg 1/1 Running 0 3h42m
kube-system kube-proxy-2wqr2 1/1 Running 0 3h36m
kube-system kube-proxy-7kz4p 1/1 Running 0 3h36m
kube-system kube-proxy-rxkp8 1/1 Running 0 3h36m
opentelemetry-operator-system adot-collector-64c8b46888-q6s98 1/1 Running 0 158m
opentelemetry-operator-system opentelemetry-operator-controller-manager-68f5b47944-pv6x7 2/2 Running 0 158m
```

- Open your Managed Grafana Workspace, head to the configuration page and and verify that Amazon Managed Prometheus was added as a default data source, test its connectivity.

#### Deploy an Example Application

In this section we will deploy sample application and extract metrics using AWS OpenTelemetry collector

- 1. Add the helm incubator repo:

```
helm repo add haproxy-ingress https://haproxy-ingress.github.io/charts
```

- 2. Enter the following command to create a new namespace:

```
kubectl create namespace haproxy-ingress-sample
```

- 3. Enter the following commands to install HAProxy:
adot-collector-haproxy adot-collector-788f78cf45-przds 1/1 Running 0 59s
cert-manager cert-manager-c84fb49b6-6qwr8 1/1 Running 0 6m49s
cert-manager cert-manager-cainjector-7d55bf8f78-w24bv 1/1 Running 0 6m50s
cert-manager cert-manager-webhook-577f77586f-xldrk 1/1 Running 0 6m49s
haproxy-ingress haproxy-ingress-566cc75f8b-dhv6g 1/1 Running 0 7m15s
haproxy-ingress haproxy-ingress-default-backend-5c746cccb9-p2ztq 1/1 Running 0 7m15s
kube-system aws-node-6b6cb 1/1 Running 0 5m52s
kube-system aws-node-mgx8d 1/1 Running 0 5m53s
kube-system aws-node-z8p8r 1/1 Running 0 5m48s
kube-system coredns-85d5b4454c-dr4z5 1/1 Running 0 10m
kube-system coredns-85d5b4454c-zp8qs 1/1 Running 0 10m
kube-system kube-proxy-2n5vj 1/1 Running 0 5m52s
kube-system kube-proxy-rcvw4 1/1 Running 0 5m53s
kube-system kube-proxy-wzml2 1/1 Running 0 5m48s
opentelemetry-operator-system opentelemetry-operator-controller-manager-865fd559cd-7tvmg 2/2 Running 0 65s
```

3. Open your Managed Grafana Workspace, head to the configuration page and and verify that Amazon Managed Prometheus was added as a default data source, test its connectivity.

4. Navigate to the dashboard side panel, click on `Observability` Folder and open the `HAProxy for Kubernetes` Dashboard.

<!-- TODO - this link is dead, is there a replacement link for what the dashboard should look like? -->
<img width="1468" alt="HAProxy-dashboard" src="https://github.com/awsdabra/amg-dashboard-examples/blob/d4275d2e0251963b8783dcc03fd475d6f8783cc7/haproxy_grafana_dashboard.png">

```
helm install haproxy haproxy-ingress/haproxy-ingress \
--namespace haproxy-ingress-sample \
--set defaultBackend.enabled=true \
--set controller.stats.enabled=true \
--set controller.metrics.enabled=true \
--set-string controller.metrics.service.annotations."prometheus\.io/port"="9101" \
--set-string controller.metrics.service.annotations."prometheus\.io/scrape"="true"
```
## Destroy

- 4. Verify if the application is running
To teardown and remove the resources created in this example:

```sh
terraform destroy -target=module.eks_blueprints_kubernetes_addons -auto-approve
terraform destroy -target=module.eks_blueprints -auto-approve
terraform destroy -auto-approve
```
kubectl get pods -n haproxy-ingress-sample

```

#### Vizualize the Application's dashboard

Log back into your Managed Grafana Workspace and navigate to the dashboard side panel, click on `Observability` Folder and open the `HAProxy for Kubernetes` Dashboard.

<img width="1468" alt="HAProxy-dashboard" src="https://github.com/awsdabra/amg-dashboard-examples/blob/d4275d2e0251963b8783dcc03fd475d6f8783cc7/haproxy_grafana_dashboard.png">

## Cleanup

- Run `terraform destroy -var-file=dev.tfvars` to remove all resources except for your Amazon Managed Grafana workspace.
- Delete your Amazon Managed Grafana workspace through the AWS console.

## Troubleshooting

- When running `terraform apply` or `terraform destroy`, the process will sometimes time-out. If that happens, run the command again and the operation will continue where it left off.

- You can explore the OpenTelemtry Collector logs by running the following command:

```
```sh
kubectl get pods -n opentelemetry-operator-system
kubectl logs -f -n opentelemetry-operator-system adot-collector-xxxx
kubectl logs -f -n opentelemetry-operator-system
```
Loading