Skip to content

Commit

Permalink
OSS Update Pass + Style Changes (#362)
Browse files Browse the repository at this point in the history
* style update: casing

* more style updates

* update kured to 1.12.0

* update sed instruction

* update azure monitor config

* fix error without notification controller :(

* fix kured typos

* Update traefik to 2.9.6

* style updates

* Update cluster-manifests/kube-system/container-azm-ms-agentconfig.yaml
  • Loading branch information
ckittel authored Jan 3, 2023
1 parent 7943456 commit ef75c5e
Show file tree
Hide file tree
Showing 23 changed files with 118 additions and 90 deletions.
2 changes: 1 addition & 1 deletion 01-prerequisites.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Prerequisites

This is the starting point for the instructions on deploying the [AKS Baseline reference implementation](./README.md). There is required access and tooling you'll need in order to accomplish this. Follow the instructions below and on the subsequent pages so that you can get your environment ready to proceed with the AKS cluster creation.
This is the starting point for the instructions on deploying the [AKS baseline reference implementation](./README.md). There is required access and tooling you'll need in order to accomplish this. Follow the instructions below and on the subsequent pages so that you can get your environment ready to proceed with the AKS cluster creation.

| :clock10: | These steps are intentionally verbose, intermixed with context, narrative, and guidance. The deployments are all conducted via [Bicep templates](https://learn.microsoft.com/azure/azure-resource-manager/bicep/overview), but they are executed manually via `az cli` commands. We strongly encourage you to dedicate time to walk through these instructions, with a focus on learning. We do not provide any "one click" method to complete all deployments.<br><br>Once you understand the components involved and have identified the shared responsibilities between your team and your greater organization, you are encouraged to build suitable, repeatable deployment processes around your final infrastructure and cluster bootstrapping. The [AKS baseline automation guidance](https://github.com/Azure/aks-baseline-automation#aks-baseline-automation) is a great place to learn how to build your own automation pipelines. That guidance is based on the same architecture foundations presented here in the AKS baseline, and illustrates GitHub Actions based deployments for all components, including workloads. |
|-----------|:--------------------------|
Expand Down
12 changes: 6 additions & 6 deletions 02-ca-certificates.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Generate Your Client-Facing and AKS Ingress Controller TLS Certificates
# Generate your client-facing and AKS ingress controller TLS certificates

Now that you have the [prerequisites](./01-prerequisites.md) met, follow the steps below to create the TLS certificates that Azure Application Gateway will serve for clients connecting to your web app as well as the AKS Ingress Controller. If you already have access to an appropriate certificates, or can procure them from your organization, consider doing so and skipping the certificate generation steps. The following will describe using a self-signed certs for instructive purposes only.

Expand All @@ -10,7 +10,7 @@ Now that you have the [prerequisites](./01-prerequisites.md) met, follow the ste
export DOMAIN_NAME_AKS_BASELINE="contoso.com"
```

1. Generate a client-facing self-signed TLS certificate
1. Generate a client-facing, self-signed TLS certificate.

> :book: Contoso Bicycle needs to procure a CA certificate for the web site. As this is going to be a user-facing site, they purchase an EV cert from their CA. This will serve in front of the Azure Application Gateway. They will also procure another one, a standard cert, to be used with the AKS Ingress Controller. This one is not EV, as it will not be user facing.
Expand All @@ -23,7 +23,7 @@ Now that you have the [prerequisites](./01-prerequisites.md) met, follow the ste
openssl pkcs12 -export -out appgw.pfx -in appgw.crt -inkey appgw.key -passout pass:
```

1. Base64 encode the client-facing certificate
1. Base64 encode the client-facing certificate.

:bulb: No matter if you used a certificate from your organization or you generated one from above, you'll need the certificate (as `.pfx`) to be Base64 encoded for proper storage in Key Vault later.

Expand All @@ -32,15 +32,15 @@ Now that you have the [prerequisites](./01-prerequisites.md) met, follow the ste
echo APP_GATEWAY_LISTENER_CERTIFICATE_AKS_BASELINE: $APP_GATEWAY_LISTENER_CERTIFICATE_AKS_BASELINE
```

1. Generate the wildcard certificate for the AKS Ingress Controller
1. Generate the wildcard certificate for the AKS ingress controller.

> :book: Contoso Bicycle will also procure another TLS certificate, a standard cert, to be used with the AKS Ingress Controller. This one is not EV, as it will not be user facing. Finally the app team decides to use a wildcard certificate of `*.aks-ingress.contoso.com` for the ingress controller.
> :book: Contoso Bicycle will also procure another TLS certificate, a standard cert, to be used with the AKS ingress controller. This one is not EV, as it will not be user facing. Finally the app team decides to use a wildcard certificate of `*.aks-ingress.contoso.com` for the ingress controller.
```bash
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -out traefik-ingress-internal-aks-ingress-tls.crt -keyout traefik-ingress-internal-aks-ingress-tls.key -subj "/CN=*.aks-ingress.${DOMAIN_NAME_AKS_BASELINE}/O=Contoso AKS Ingress"
```

1. Base64 encode the AKS Ingress Controller certificate
1. Base64 encode the AKS ingress controller certificate.

:bulb: No matter if you used a certificate from your organization or you generated one from above, you'll need the public certificate (as `.crt` or `.cer`) to be Base64 encoded for proper storage in Key Vault later.

Expand Down
2 changes: 1 addition & 1 deletion 03-aad.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Prep for Azure Active Directory Integration
# Prep for Azure Active Directory integration

In the prior step, you [generated the user-facing TLS certificate](./02-ca-certificates.md); now we'll prepare Azure AD for Kubernetes role-based access control (RBAC). This will ensure you have an Azure AD security group(s) and user(s) assigned for group-based Kubernetes control plane access.

Expand Down
14 changes: 7 additions & 7 deletions 04-networking.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# Deploy the Hub-Spoke Network Topology
# Deploy the hub-spoke network topology

The prerequisites for the [AKS Baseline cluster](./) are now completed with [Azure AD group and user work](./03-aad.md) performed in the prior steps. Now we will start with our first Azure resource deployment, the network resources.
The prerequisites for the [AKS baseline cluster](./) are now completed with [Azure AD group and user work](./03-aad.md) performed in the prior steps. Now we will start with our first Azure resource deployment, the network resources.

## Subscription and resource group topology

This reference implementation is split across several resource groups in a single subscription. This is to replicate the fact that many organizations will split certain responsibilities into specialized subscriptions (e.g. regional hubs/vwan in a _Connectivity_ subscription and workloads in landing zone subscriptions). We expect you to explore this reference implementation within a single subscription, but when you implement this cluster at your organization, you will need to take what you've learned here and apply it to your expected subscription and resource group topology (such as those [offered by the Cloud Adoption Framework](https://learn.microsoft.com/azure/cloud-adoption-framework/decision-guides/subscriptions/).) This single subscription, multiple resource group model is for simplicity of demonstration purposes only.

## Expected results

### Resource Groups
### Resource groups

The following two resource groups will be created and populated with networking resources in the steps below.

Expand All @@ -19,10 +19,10 @@ The following two resource groups will be created and populated with networking

### Resources

* Regional Azure Firewall in Hub Virtual Network
* Network Spoke for the Cluster
* Network Peering from the Spoke to the Hub
* Force Tunnel UDR for Cluster Subnets to the Hub
* Regional Azure Firewall in hub virtual network
* Network spoke for the cluster
* Network peering from the spoke to the hub
* Force tunnel UDR for cluster subnets to the hub
* Network Security Groups for all subnets that support them

## Steps
Expand Down
6 changes: 3 additions & 3 deletions 05-bootstrap-prep.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ We'll be bootstrapping this cluster with the Flux GitOps agent as installed as a
echo ACR_NAME_AKS_BASELINE: $ACR_NAME_AKS_BASELINE

# Import core image(s) hosted in public container registries to be used during bootstrapping
az acr import --source ghcr.io/kubereboot/kured:1.11.0 -n $ACR_NAME_AKS_BASELINE
az acr import --source ghcr.io/kubereboot/kured:1.12.0 -n $ACR_NAME_AKS_BASELINE
```

> In this walkthrough, there is only one image that is included in the bootstrapping process. It's included as an reference for this process. Your choice to use Kubernetes Reboot Daemon (Kured) or any other images, including helm charts, as part of your bootstrapping is yours to make.
Expand All @@ -69,12 +69,12 @@ We'll be bootstrapping this cluster with the Flux GitOps agent as installed as a
:warning: Without updating these files and using your own fork, you will be deploying your cluster such that it takes dependencies on public container registries. This is generally okay for exploratory/testing, but not suitable for production. Before going to production, ensure _all_ image references you bring to your cluster are from _your_ container registry (link imported in the prior step) or another that you feel confident relying on.

```bash
sed -i "s:docker.io:${ACR_NAME_AKS_BASELINE}.azurecr.io:" ./cluster-manifests/cluster-baseline-settings/kured.yaml
sed -i "s:ghcr.io:${ACR_NAME_AKS_BASELINE}.azurecr.io:" ./cluster-manifests/cluster-baseline-settings/kured.yaml
```

Note, that if you are on macOS, you might need to use the following command instead:
```bash
sed -i '' 's:docker.io:'"${ACR_NAME_AKS_BASELINE}"'.azurecr.io:g' ./cluster-manifests/cluster-baseline-settings/kured.yaml
sed -i '' 's:ghcr.io:'"${ACR_NAME_AKS_BASELINE}"'.azurecr.io:g' ./cluster-manifests/cluster-baseline-settings/kured.yaml
```
Now commit changes to repository.

Expand Down
2 changes: 1 addition & 1 deletion 06-aks-cluster.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Deploy the AKS Cluster
# Deploy the AKS cluster

Now that your [ACR instance is deployed and ready to support cluster bootstrapping](./05-bootstrap-prep.md), the next step in the [AKS baseline reference implementation](./) is deploying the AKS cluster and its remaining adjacent Azure resources.

Expand Down
2 changes: 1 addition & 1 deletion 07-bootstrap-validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ GitOps allows a team to author Kubernetes manifest files, persist them in their
The bootstrapping process that already happened due to the usage of the Flux extension for AKS has set up the following, amoung other things

* the workload's namespace named `a0008`
* Installed kured
* installed kured

```bash
kubectl get namespaces
Expand Down
4 changes: 2 additions & 2 deletions 08-workload-prerequisites.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Workload Prerequisites
# Workload prerequisites

The AKS Cluster has been [bootstrapped](./07-bootstrap-validation.md), wrapping up the infrastructure focus of the [AKS Baseline reference implementation](./). Follow the steps below to import the TLS certificate that the Ingress Controller will serve for Application Gateway to connect to your web app.
The AKS Cluster has been [bootstrapped](./07-bootstrap-validation.md), wrapping up the infrastructure focus of the [AKS baseline reference implementation](./). Follow the steps below to import the TLS certificate that the Ingress Controller will serve for Application Gateway to connect to your web app.

## Steps

Expand Down
4 changes: 2 additions & 2 deletions 09-secret-management-and-ingress-controller.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Configure AKS Ingress Controller with Azure Key Vault integration
# Configure AKS ingress controller with Azure Key Vault integration

Previously you have configured [workload prerequisites](./08-workload-prerequisites.md). These steps configure Traefik, the AKS ingress solution used in this reference implementation, so that it can securely expose the web app to your Application Gateway.

Expand Down Expand Up @@ -58,7 +58,7 @@ Previously you have configured [workload prerequisites](./08-workload-prerequisi
```bash
# Import ingress controller image hosted in public container registries
az acr import --source docker.io/library/traefik:v2.8.1 -n $ACR_NAME_AKS_BASELINE
az acr import --source docker.io/library/traefik:v2.9.6 -n $ACR_NAME_AKS_BASELINE
```
1. Install the Traefik Ingress Controller.
Expand Down
2 changes: 1 addition & 1 deletion 10-workload.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Deploy the Workload (ASP.NET Core Docker web app)
# Deploy the workload (ASP.NET Core Docker web app)

The cluster now has an [Traefik configured with a TLS certificate](./09-secret-management-and-ingress-controller.md). The last step in the process is to deploy the workload, which will demonstrate the system's functions.

Expand Down
24 changes: 12 additions & 12 deletions 11-validation.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# End-to-End Validation
# End-to-end validation

Now that you have a workload deployed, the [ASP.NET Core Docker sample web app](./10-workload.md), you can start validating and exploring this reference implementation of the [AKS Baseline cluster](./). In addition to the workload, there are some observability validation you can perform as well.
Now that you have a workload deployed, the [ASP.NET Core sample web app](./10-workload.md), you can start validating and exploring this reference implementation of the [AKS baseline cluster](./). In addition to the workload, there are some observability validation you can perform as well.

## Validate the Web App
## Validate the web app

This section will help you to validate the workload is exposed correctly and responding to HTTP requests.

### Steps

1. Get Public IP of Application Gateway
1. Get the public IP address of Application Gateway.

> :book: The app team conducts a final acceptance test to be sure that traffic is flowing end-to-end as expected, so they place a request against the Azure Application Gateway endpoint.
Expand All @@ -18,7 +18,7 @@ This section will help you to validate the workload is exposed correctly and res
echo APPGW_PUBLIC_IP: $APPGW_PUBLIC_IP
```

1. Create `A` Record for DNS
1. Create an `A` record for DNS.

> :bulb: You can simulate this via a local hosts file modification. You're welcome to add a real DNS entry for your specific deployment's application domain name, if you have access to do so.
Expand Down Expand Up @@ -83,7 +83,7 @@ Built-in as well as custom policies are applied to the cluster as part of the [c
Error from server (Forbidden): error when creating "STDIN": admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8scustomingresstlshostshavede-e64871e795ce3239cd99] TLS host must have one of defined domain suffixes. Valid domain names are ["contoso.com"]; defined TLS hosts are {"bu0001a0008-00.aks-ingress.invalid-domain.com"}; incompliant hosts are {"bu0001a0008-00.aks-ingress.invalid-domain.com"}.
```
## Validate Web Application Firewall functionality
## Validate web application firewall functionality
Your workload is placed behind a Web Application Firewall (WAF), which has rules designed to stop intentionally malicious activity. You can test this by triggering one of the built-in rules with a request that looks malicious.
Expand All @@ -104,7 +104,7 @@ Your workload is placed behind a Web Application Firewall (WAF), which has rules
| where ResourceProvider == "MICROSOFT.NETWORK" and Category == "ApplicationGatewayFirewallLog"
```
## Validate Cluster Azure Monitor Insights and Logs
## Validate cluster Azure Monitor insights and logs
Monitoring your cluster is critical, especially when you're running a production cluster. Therefore, your AKS cluster is configured to send [diagnostic information](https://learn.microsoft.com/azure/aks/monitor-aks) of categories _cluster-autoscaler_, _kube-controller-manager_, _kube-audit-admin_ and _guard_ to the Log Analytics Workspace deployed as part of the [bootstrapping step](./05-bootstrap-prep.md). Additionally, [Azure Monitor for containers](https://learn.microsoft.com/azure/azure-monitor/insights/container-insights-overview) is configured on your cluster to capture metrics and logs from your workload containers. Azure Monitor is configured to surface cluster logs, here you can see those logs as they are generated.
Expand All @@ -115,13 +115,13 @@ Monitoring your cluster is critical, especially when you're running a production
1. In the Azure Portal, navigate to your AKS cluster resource.
1. Click _Insights_ to see captured data.
You can also execute [queries](https://learn.microsoft.com/azure/azure-monitor/log-query/get-started-portal) on the [cluster logs captured](https://learn.microsoft.com/azure/azure-monitor/insights/container-insights-log-search).
You can also execute [queries](https://learn.microsoft.com/azure/azure-monitor/logs/log-analytics-tutorial) on the [cluster logs captured](https://learn.microsoft.com/azure/azure-monitor/containers/container-insights-log-query).
1. In the Azure Portal, navigate to your AKS cluster resource.
1. Click _Logs_ to see and query log data.
:bulb: There are several examples on the _Kubernetes Services_ category.
## Validate Azure Monitor for containers (Prometheus Metrics)
## Validate Azure Monitor for containers (Prometheus metrics)
Azure Monitor is configured to [scrape Prometheus metrics](https://learn.microsoft.com/azure/azure-monitor/insights/container-insights-prometheus-integration) in your cluster. This reference implementation is configured to collect Prometheus metrics from two namespaces, as configured in [`container-azm-ms-agentconfig.yaml`](./cluster-baseline-settings/container-azm-ms-agentconfig.yaml). There are two pods configured to emit Prometheus metrics:
Expand All @@ -137,15 +137,15 @@ Azure Monitor is configured to [scrape Prometheus metrics](https://learn.microso
1. Find the one of the above queries in the _Containers_ category.
1. You are able to select and execute the saved query over the scraped metrics.
## Validate Workload Logs
## Validate workload logs
The example workload uses the standard dotnet logger interface, which are captured in `ContainerLogs` in Azure Monitor. You could also include additional logging and telemetry frameworks in your workload, such as Application Insights. Here are the steps to view the built-in application logs.
### Steps
1. In the Azure Portal, navigate to your AKS cluster resource group (`rg-bu0001a0008`).
1. Select your Log Analytic Workspace resource and open the _Logs_ blade.
1. Execute the following query
1. Execute the following query.
```
let podInventory = KubePodInventory
Expand Down Expand Up @@ -185,7 +185,7 @@ A series of metric alerts were configured as well in this reference implementati
1. Select your cluster, then _Insights_.
1. Select _Recommended alerts_ to see those enabled. (Feel free to enable/disable as you see fit.)
## Validate Azure Container Registry Image Pulls
## Validate Azure Container Registry image pulls
If you configured your third-party images to be pulled from your Azure Container Registry vs public registries, you can validate that the container registry logs show `Pull` logs for your cluster when you applied your flux configuration.
Expand Down
2 changes: 1 addition & 1 deletion 12-cleanup.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Clean up

After you are done exploring your deployed [AKS Baseline cluster](./), you'll want to delete the created Azure resources to prevent undesired costs from accruing. Follow these steps to delete all resources created as part of this reference implementation.
After you are done exploring your deployed [AKS baseline cluster](./), you'll want to delete the created Azure resources to prevent undesired costs from accruing. Follow these steps to delete all resources created as part of this reference implementation.

## Steps

Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Contributing to the AKS Baseline reference implementation
# Contributing to the AKS baseline reference implementation

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit <https://cla.opensource.microsoft.com/>.

Expand Down
Loading

0 comments on commit ef75c5e

Please sign in to comment.