(Fix) Updating helm charts to support version 3 windows integration #754

RagalahariP · 2023-05-18T07:20:12Z

Updating helm charts to support version 3 windows integration

Testing : Tested in local EKS cluster with the following node config

kubectl get nodes -o wide
NAME                          STATUS   ROLES    AGE     VERSION                INTERNAL-IP   EXTERNAL-IP      OS-IMAGE                         KERNEL-VERSION                 CONTAINER-RUNTIME
ip-10-0-11-231.ec2.internal   Ready    <none>   3d21h   v1.23.17-eks-a59e1f0   10.0.11.231   18.208.247.135   Windows Server 2022 Datacenter   10.0.20348.1668                docker://20.10.21
ip-10-0-110-39.ec2.internal   Ready    <none>   22d     v1.23.17-eks-a59e1f0   10.0.110.39   <none>           Windows Server 2019 Datacenter   10.0.17763.4252                docker://20.10.21
ip-10-0-27-190.ec2.internal   Ready    <none>   22d     v1.23.17-eks-a59e1f0   10.0.27.190   3.95.250.93      Amazon Linux 2                   5.4.238-148.347.amzn2.x86_64   docker://20.10.17

Dashboard: https://onenr.io/0BQ18La75jx

Original PR (with comments): #744

nrepai · 2023-05-18T17:15:28Z

charts/newrelic-infrastructure/values.yaml

+  enableWindows: false
+  # --  List of Windows versions present in the cluster. Our Kubernetes integration supported Windows versions LTSC 2019 (1809), 20H2, and LTSC 2022
+  windowsOsList: []
+  #  - version: 2019


is it intended for those to be commented out?

@nrepai Yes, WindowsOsList is required if enableWindows is set to True.

@nrepai I have updated comment.

juanjjaramillo

Looks good to me. My only concern is that in the description it seems you only tested using Kubernetes v1.23:

kubectl get nodes -o wide
NAME                          STATUS   ROLES    AGE     VERSION                INTERNAL-IP   EXTERNAL-IP      OS-IMAGE                         KERNEL-VERSION                 CONTAINER-RUNTIME
ip-10-0-11-231.ec2.internal   Ready    <none>   3d21h   v1.23.17-eks-a59e1f0   10.0.11.231   18.208.247.135   Windows Server 2022 Datacenter   10.0.20348.1668                docker://20.10.21
ip-10-0-110-39.ec2.internal   Ready    <none>   22d     v1.23.17-eks-a59e1f0   10.0.110.39   <none>           Windows Server 2019 Datacenter   10.0.17763.4252                docker://20.10.21
ip-10-0-27-190.ec2.internal   Ready    <none>   22d     v1.23.17-eks-a59e1f0   10.0.27.190   3.95.250.93      Amazon Linux 2                   5.4.238-148.347.amzn2.x86_64   docker://20.10.17

This version is no longer supported. If possible, can we test using a v1.26 cluster?

…inux nodes. And collecting metrics from Windows nodes will continue the same approach in V2

juanjjaramillo · 2023-05-26T17:47:27Z

The added selectors look good to me and match others that we have included already. Anyone else has a different opinion?

xqi-nr · 2023-05-31T23:40:31Z

charts/newrelic-infrastructure/README.md

@@ -161,13 +161,15 @@ integrations that you have configured.
 | kubelet.config.retries | int | `3` | Number of retries after timeout expired |
 | kubelet.config.scraperMaxReruns | int | `4` | Max number of scraper rerun when scraper runtime error happens |
 | kubelet.config.timeout | string | `"10s"` | Timeout for the kubelet APIs contacted by the integration |
+| kubelet.enableWindows | bool | `false` | Enable Windows monitoring and update `windowsOsList` with the list of Windows versions present in the cluster. |


make sure this readme is auto-generated by helm-docs, otherwise, it will be overwritten

xqi-nr · 2023-06-01T16:59:00Z

charts/newrelic-infrastructure/templates/clusterrole.yaml

@@ -18,6 +18,7 @@ rules:
      - "services"
      - "nodes"
      - "namespaces"
+      - "pods" # Needed for our Windows integration to be able to access KSM on a Linux node.


@RagalahariP From v2 to v3, the pods resource is removed for linux support for some reason as you mentioned during the meeting. The reason may be reducing the security surface.

What I mean during the meeting is if we can adding it back for windows only, that will be better .

@xqi-nr after you left the meeting we agreed that this comment is a good compromise. @RagalahariP will contact @paologallinaharbur to request his review of this line change since he was involved in the project to upgrade V2 to V3 and may have historical context to validate this change

Hello! Yes, I'll take a look!

In general, we have never added support for Windows due to the lack of a canary environment, tooling, and proper pipelines to build the images.
Consider to specify clearly that this is experimental and might change in the future.

By the way, answering this particular point, it was a permission not needed anymore and therefore it was dropped.
You can reintroduce it without any issues, however if you want to ask for as few permission as possible you can add this specific line behind a "if windows.enabled=true" instead of a comment

paologallinaharbur

I was expecting a different approach.
If merged you will end-up in a mixed scenario with both V2 a V3 images running (V2 on Windows and V3 on Linux).

This could lead to multiple issues if not well configured, for example, KSM could be scraped form two different places (from a linux and a Windows node).

I would have expected to see V3 Windows images

paologallinaharbur · 2023-06-05T13:34:10Z

charts/newrelic-infrastructure/templates/kubelet/daemonset.yaml

      nodeSelector:
+        kubernetes.io/os: linux 


are we sure that kubernetes.io/os: linux is present on all user nodes?

paologallinaharbur · 2023-06-05T13:35:03Z

charts/newrelic-infrastructure/templates/kubelet/daemonset-windows.yaml

+      serviceAccountName: {{ include "newrelic.common.serviceAccount.name" $ }}
+      containers:
+        - name: agent
+          image: newrelic/infrastructure-k8s:{{ .imageTag }}


This is a V2 image, why not installing the nri-bundle-v2 next to the V3 then? In this way we are mixing them up in the same chart!

paologallinaharbur · 2023-06-05T13:36:00Z

charts/newrelic-infrastructure/README.md

@@ -161,13 +161,15 @@ integrations that you have configured.
 | kubelet.config.retries | int | `3` | Number of retries after timeout expired |
 | kubelet.config.scraperMaxReruns | int | `4` | Max number of scraper rerun when scraper runtime error happens |
 | kubelet.config.timeout | string | `"10s"` | Timeout for the kubelet APIs contacted by the integration |
+| kubelet.enableWindows | bool | `false` | Enable Windows monitoring and update `windowsOsList` with the list of Windows versions present in the cluster. |


We are adding the V2 image under the Kubelet component. However, the V2 is a bundle of kubelet+KSM+controlPlane component

paologallinaharbur · 2023-06-05T13:36:37Z

charts/newrelic-infrastructure/README.md

 | kubelet.enabled | bool | `true` | Enable kubelet monitoring. Advanced users only. Setting this to `false` is not supported and will break the New Relic experience. |
 | kubelet.extraEnv | list | `[]` | Add user environment variables to the agent |
 | kubelet.extraEnvFrom | list | `[]` | Add user environment from configMaps or secrets as variables to the agent |
 | kubelet.extraVolumeMounts | list | `[]` | Defines where to mount volumes specified with `extraVolumes` |
 | kubelet.extraVolumes | list | `[]` | Volumes to mount in the containers |
 | kubelet.hostNetwork | bool | Not set | Sets pod's hostNetwork. When set bypasses global/common variable |
 | kubelet.tolerations | list | Schedules in all tainted nodes | Tolerations for the control plane DaemonSet. |
+| kubelet.windowsOsList | list | `[]` | `windowsOsList` is only required if `enableWindows` is true. It contains list of Windows versions present in the cluster. Our Kubernetes integration supported Windows versions LTSC 2019 (1809), 20H2, and LTSC 2022 |


"Our Kubernetes integration supported Windows versions"

I would always specify that it is alpha

paologallinaharbur · 2023-06-05T13:46:15Z

charts/newrelic-infrastructure/templates/kubelet/daemonset-windows.yaml

+  template:
+    metadata:
+      annotations:
+        checksum/nri-kubernetes: {{ include (print $.Template.BasePath "/kubelet/scraper-configmap.yaml") $ | sha256sum }}


Nri-kubernetes in V2 branch does not read any configuration, that was introduced in V3.

I wonder if you actually compiled the V3 branch and pushed it to the old repo newrelic/infrastructure-k8s, is that the case?

There are some discrepancies:

kubelet/scraper-configmap.yaml is V3

NRIA_PASSTHROUGH_ENVIRONMENT is V2

app.kubernetes.io/component: kubelet is V3

having just one container is definitely V2

how a user could change the config mounted? or set a proxy?

paologallinaharbur · 2023-06-05T13:50:40Z

charts/newrelic-infrastructure/templates/kubelet/daemonset-windows.yaml

I may not understand the approach. However, this seems a V2 component being merged into the V3 chart.

The image used is newrelic/infrastructure-k8s, the is just one container, and the nri-kubernetes process seems controlled by the agent. Maybe this is the fastest way to get Windows support, but there is a big risk of tech dept

juanjjaramillo

@paologallinaharbur has raised valid and serious issues that need to be addressed before we can merge the PR. I'm just marking the PR such that we do not merge it by mistake before his comments are fully addressed

RagalahariP added 2 commits May 18, 2023 12:39

Updating helm charts to support version 3 windows integration

101cf07

Updating README with helm-docs

ba8abf7

RagalahariP requested a review from a team as a code owner May 18, 2023 07:20

nrepai approved these changes May 18, 2023

View reviewed changes

htroisi mentioned this pull request May 19, 2023

Adding V3 compatible windows integration #744

Closed

Updated comments

0230975

RagalahariP requested a review from nrepai May 22, 2023 18:08

juanjjaramillo approved these changes May 25, 2023

View reviewed changes

RagalahariP added 2 commits May 25, 2023 17:00

Adding node selector for KSM so that it will ensure to run for only L…

ec2094a

…inux nodes. And collecting metrics from Windows nodes will continue the same approach in V2

Fixing test cases

a399c40

RagalahariP requested a review from juanjjaramillo May 25, 2023 11:51

xqi-nr reviewed May 31, 2023

View reviewed changes

Updating comment in clusterRole

7d860af

xqi-nr reviewed Jun 1, 2023

View reviewed changes

paologallinaharbur suggested changes Jun 5, 2023

View reviewed changes

paologallinaharbur reviewed Jun 5, 2023

View reviewed changes

juanjjaramillo suggested changes Jun 5, 2023

View reviewed changes

juanjjaramillo requested a review from a team as a code owner October 6, 2023 00:05

juanjjaramillo force-pushed the main branch from 2553378 to 0659806 Compare October 6, 2023 00:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(Fix) Updating helm charts to support version 3 windows integration #754

(Fix) Updating helm charts to support version 3 windows integration #754

RagalahariP commented May 18, 2023 •

edited by htroisi

Loading

nrepai May 18, 2023

RagalahariP May 22, 2023 •

edited

Loading

RagalahariP May 22, 2023

juanjjaramillo left a comment

juanjjaramillo commented May 26, 2023

xqi-nr May 31, 2023

xqi-nr Jun 1, 2023 •

edited

Loading

juanjjaramillo Jun 2, 2023

paologallinaharbur Jun 5, 2023

paologallinaharbur Jun 5, 2023

paologallinaharbur left a comment

paologallinaharbur Jun 5, 2023

paologallinaharbur Jun 5, 2023 •

edited

Loading

paologallinaharbur Jun 5, 2023

paologallinaharbur Jun 5, 2023 •

edited

Loading

paologallinaharbur Jun 5, 2023

paologallinaharbur Jun 5, 2023

paologallinaharbur Jun 5, 2023

juanjjaramillo left a comment

(Fix) Updating helm charts to support version 3 windows integration #754

Are you sure you want to change the base?

(Fix) Updating helm charts to support version 3 windows integration #754

Conversation

RagalahariP commented May 18, 2023 • edited by htroisi Loading

Choose a reason for hiding this comment

RagalahariP May 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juanjjaramillo left a comment

Choose a reason for hiding this comment

juanjjaramillo commented May 26, 2023

Choose a reason for hiding this comment

xqi-nr Jun 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paologallinaharbur left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paologallinaharbur Jun 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paologallinaharbur Jun 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juanjjaramillo left a comment

Choose a reason for hiding this comment

RagalahariP commented May 18, 2023 •

edited by htroisi

Loading

RagalahariP May 22, 2023 •

edited

Loading

xqi-nr Jun 1, 2023 •

edited

Loading

paologallinaharbur Jun 5, 2023 •

edited

Loading

paologallinaharbur Jun 5, 2023 •

edited

Loading