If you are using the official blueprint these have already been exposed, but if you are building a new blueprint or deployment you will need to manually expose the following
- Prometheus: 9090
- Grafana: 3000
- Zipkin: 9411
While NGINX Service Mesh supports pulling the control and data plane images directly from F5's container registry during install, it's helpful to know how to configure NSM to pull images from your own reposititory (helpful in an air-gapped environment). The below section will help familiarize you with manually pulling/taggin$
Local registry is registry:5000.
The below is a quick walkthrough on using the local registry. It is not required for NSM, simply some background on using a local registry with K8s (which you will be using for your NSM deployment).
$ docker pull nginx
$ docker tag nginx:latest registry:5000/nginx:latest
$ docker push registry:5000/nginx:latest
$ wget https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/application/deployment.yaml
$ vim deployment.yaml
Change image to registry:5000/nginx:latest
$ kubectl apply -f ./deployment.yaml
$ kubectl get pods
$ kubectl describe pod/deplomyment-[GUID]
Look for Successfully pulled image "registry:5000..." to make sure the local reg is working
$ kubectl delete -f ./deployment.yaml
$ export ver=1.1.0
$ gzip -dc nginx-meshctl_linux.gz > nginx-meshctl && chmod 755 nginx-meshctl
$ tar zxvf nginx-mesh-images-$ver.tar.gz
*NOTE: docker load/tag/push is captured here for testing and documentation purposes. These steps are not needed when using F5's container registry to deploy NSM (see Deploy The Mesh below for details).
$ docker load < nginx-mesh-images-$ver/nginx-mesh-api-$ver.tar.gz && \
docker load < nginx-mesh-images-$ver/nginx-mesh-metrics-$ver.tar.gz && \
docker load < nginx-mesh-images-$ver/nginx-mesh-sidecar-$ver.tar.gz && \
docker load < nginx-mesh-images-$ver/nginx-mesh-init-$ver.tar.gz
$ docker tag nginx-mesh-sidecar:$ver registry:5000/nginx-mesh-sidecar:$ver && \
docker tag nginx-mesh-init:$ver registry:5000/nginx-mesh-init:$ver && \
docker tag nginx-mesh-metrics:$ver registry:5000/nginx-mesh-metrics:$ver && \
docker tag nginx-mesh-api:$ver registry:5000/nginx-mesh-api:$ver
$ docker push registry:5000/nginx-mesh-api:$ver && \
docker push registry:5000/nginx-mesh-metrics:$ver && \
docker push registry:5000/nginx-mesh-init:$ver && \
docker push registry:5000/nginx-mesh-sidecar:$ver
$ docker image ls | grep "registry:5000/nginx-mesh"
This walkthrough will disable auto-injection cluster-wide and enable the bookinfo namespace for auto-inect only. We will look at how to change this behavior after NSM is deployed later in the lesson.
You have two options to instruct NSM on how to retrieve images:
To pull from F5's public container registry
$ ./nginx-meshctl deploy \
--disable-auto-inject \
--enabled-namespaces bookinfo \
--mtls-mode strict \
--mtls-trust-domain nginx.mesh \
--registry-server docker-registry.nginx.com/nsm \
--image-tag $ver
Point NSM installer to your registry and assign a tag (please see docs for details about authenticating to a private registry, if needed)
$ ./nginx-meshctl deploy \
--disable-auto-inject \
--enabled-namespaces bookinfo \
--mtls-mode strict \
--mtls-trust-domain nginx.mesh \
--registry-server registry:5000 \
--image-tag $ver
Manually call out each image by name and tag
$ ./nginx-meshctl deploy \
--disable-auto-inject \
--enabled-namespaces bookinfo \
--mtls-mode strict \
--mtls-trust-domain nginx.mesh \
--nginx-mesh-api-image "registry:5000/nginx-mesh-api:$ver" \
--nginx-mesh-sidecar-image "registry:5000/nginx-mesh-sidecar:$ver" \
--nginx-mesh-init-image "registry:5000/nginx-mesh-init:$ver" \
--nginx-mesh-metrics-image "registry:5000/nginx-mesh-metrics:$ver"
Check that install was successful and auto-injection is disabled for the entire cluster by default but enabled for bookinfo
$ ./nginx-meshctl version
$ ./nginx-meshctl config | jq '.injection'
---
{
"disabledNamespaces": [],
"enabledNamespaces": [
"bookinfo"
],
"isAutoInjectEnabled": false
}
---
With the ./nginx-meshctl version command, you won't see a version string for the sidecar versions until you deploy an app that is managed by NSM. After deploying the app you will see a new version string returned for each sidecar version managed by NSM.
$ wget https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/platform/kube/bookinfo.yaml
$ kubectl create namespace bookinfo
$ kubectl apply -f ./bookinfo.yaml -n bookinfo
$ kubectl get pods -n bookinfo
Pod status should eventually show Running and Ready should show 2/2 for each pod
$ kubectl get pods -n bookinfo -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' | sort
*The above command will display a list of each container running in each pod in namespace bookinfo. You want to look for both the bookinfo application container and the NSM sidecar container in each pod. For example, you should see something like `details-v1-74f858558f-69lll: docker.io/istio/examples-bookinfo-details-v1:$
Demo enabled namespaces by deploying bookinfo in a namespace not on configured for NSM auto-injection
$ kubectl create namespace bookinfo-no-inject
$ kubectl apply -f ./bookinfo.yaml -n bookinfo-no-inject
$ kubectl get pods -n bookinfo-no-inject
Pod status should eventually show Running and Ready should show 1/1 for each pod, showing that the mesh did not inject a sidecar into the same deployment
$ kubectl create namespace bookinfo-man-inject
$ ./nginx-meshctl inject < bookinfo.yaml > bookinfo-man-inject.yaml
$ kubectl apply -f ./bookinfo-man-inject.yaml -n bookinfo-man-inject
$ kubectl get pods -n bookinfo-man-inject
Pod status should eventually show Running and Ready should show 2/2 for each pod, showing that we created a manual injection app deployment and deployed that into a namespace that is not enabled for auto-injection but we still manage it from what we've added to the deployment.
$ kubectl get pods -n bookinfo-man-inject -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' | sort
$ kubectl describe pod/$(kubectl get pod -l app=productpage -n bookinfo-man-inject -o jsonpath='{.items[0].metadata.name}') -n bookinfo-man-inject
Similar to the above, the goal is to show that the NSM sidecar proxies have been injected into the bookinfo app in a namespace that is not configured for auto-inject. This means that NSM is still managing traffic for the bookinfo app in the unmanaged namespace bookinfo-man-inject.
$ diff bookinfo.yaml bookinfo-man-inject.yaml
The diff will show that the new deployment, bookinfo-man-inject.yaml, contains new specifications for injecting the NSM sidecar into each application. You are able to deploy this new NSM-managed application deployment in any cluster where NSM is running, regardless of namespace.
$ ./nginx-meshctl services
Will return each service in the form of namespace/service <ip_address> <port>
$ kubectl exec -it $(kubectl get pod -l app=ratings -n bookinfo -o jsonpath='{.items[0].metadata.name}') -n bookinfo -c ratings -- curl -I productpage:9080/productpage
*When NSM is configured with mTLS strict, the easist way to test that traffic flowing through each sidecar is to exec into one application container, a ratings pod in this case, and connect to another app service also managed by the mesh, productpage. You will see the X-Mesh-Request-ID header is appended by the ingr$
NOTE: When in mTLS strict mode, you can't connect directly to productpages through the mesh unless you're deploying N+ KIC. For more information on configuring NGINX Plus KIC for ingress into NSM, please see https://docs.nginx.com/nginx-service-mesh/tutorials/kic/deploy-with-kic/
$ kubectl exec -it $(kubectl get pod -l app=ratings -n bookinfo -o jsonpath='{.items[0].metadata.name}') -n bookinfo -c ratings -- bash -c 'while true; do curl -I productpage:9080/productpage?u=normal; done'
From one of your UDF K8s nodes, the above command will exec into a container and generate simple GET traffic into the mesh which will start traffic flowing through the various microservice containers within bookinfo and be available to your visualization tools like Grafana.
NSM ships with a default Grafana dashboard that's used when NSM installs Grafana as part of the control plane. If you are running a dedicated Grafana instance, example dashboards can be found at https://github.com/nginxinc/nginx-service-mesh/tree/main/examples/grafana
$ kubectl port-forward -n nginx-mesh --address=0.0.0.0 svc/grafana 3000:3000&
$ while true; do ./nginx-meshctl top pods -n bookinfo; sleep 1; done
- Call
https://<your_k8s_cli_image>:3000to review the Grafana dashboard
The first command will expose the Grafana service port so that it can be access remotely via the previously exposed port in UDF Access Methods. The second command will loop the nginx-meshctl top command to show that NSM is seeing and distributing the inter-service traffic between application pods.
*NOTE: While it is possible to run both while traffic generators from the same K8s instance by running each in the background (which can be managed using standard *nix tools like jobs and/or fg), it is generally easier to run the nginx-meshctl top loop from the node where you've installed NSM and the traffic generato$
$ kubectl delete pod/$(kubectl get pod -l app=reviews,version=v3 -n bookinfo -o jsonpath='{.items[0].metadata.name}') -n bookinfo
$ kubectl get pods -n bookinfo
From another node in the cluster, delete the reviews pod so that K8s has to re-generate the pod. NSM will detect the re-gen and pick up traffic management as soon as the pod becomes available. Continue watching the nginx-meshctl top loop and Grafana to see NSM redistribute traffic as pods become available.
$ kubectl scale deployments ratings-v1 --replicas=3 -n bookinfo
$ kubectl get pods -n bookinfo -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' | sort
Continue watching the nginx-meshctl top loop and Grafana to see NSM redistribute traffic as pods become available
$ kubectl scale deployments ratings-v1 --replicas=1 -n bookinfo
$ kubectl get pods -n bookinfo -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' | sort
Continue watching the nginx-meshctl top loop and Grafana to see NSM redistribute traffic as pods become available
$ kubectl exec $(kubectl get pod -l app=ratings -n bookinfo -o jsonpath='{.items[0].metadata.name}') -c nginx-mesh-sidecar -n bookinfo -it -- cat /etc/nginx/nginx.conf
Add two new namespaces: One with auto-inject enabled and one without (for testing manual injection), both used to test the mesh.
$ kubectl create namespace test-tools-injected
$ kubectl create namespace test-tools
$ ./nginx-meshctl config | jq '.injection.enabledNamespaces[]'
Enable API and test by querying the auto-injection namespace allowlist (same result, different method to retrieve/verify)
$ kubectl port-forward -n nginx-mesh svc/nginx-mesh-api 8443:443&
$ curl -ks https://localhost:8443/api/config | jq '.injection.enabledNamespaces[]'
$ cat <<EOF > update-namespaces.json
{
"op": "replace",
"field": {
"injection": {
"isAutoInjectEnabled": false,
"enabledNamespaces": ["bookinfo", "test-tools-injected"]
}
}
}
EOF
$ curl -kX PATCH --header "Content-Type: application/json" https://localhost:8443/api/config -d @update-namespaces.json
$ curl -ks https://localhost:8443/api/config | jq '.injection.enabledNamespaces[]'
Or use CLI: $ ./nginx-meshctl config | jq '.injection.enabledNamespaces[]'
NOTE: Removing the mesh removes the control plane but not the data plane; sidecars stay in place but are reconfigured as transparent sidecars with no security or routing policies.
$ ./nginx-meshctl remove -y
$ kubectl get pods -n bookinfo -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' | sort
$ kubectl exec -it $(kubectl get pod -l app=ratings -n bookinfo -o jsonpath='{.items[0].metadata.name}') -n bookinfo -c ratings -- curl -I productpage:9080/productpage
You will no longer see the mesh-inserted header: X-Mesh-Request-ID
$ kubectl scale deployments ratings-v1 --replicas=3 -n bookinfo
$ kubectl get pods -n bookinfo -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' | sort
New ratings pods will not contain the sidecar
$ for deps in $(kubectl get deployments --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -n bookinfo); do kubectl rollout restart deployment/$deps -n bookinfo; done
$ kubectl get pods -n bookinfo
You'll see the sidecar-attached pods terminating and new sidecar-less pods coming up.