customize node config

[unacao]

Is there any way to customize the kubelet config such as KUBELET_HARD_EVICTION_THRESHOLD in the AKS cluster?

[slack]

Right now no, beyond EVICTION_THRESHOLD what would you like to configure?

[unacao]

possibly the configuration mentioned here #245

[sfrolich]

I have some rather large images in ACR that timeout pulling the images. Some info here:
https://blog.openai.com/scaling-kubernetes-to-2500-nodes/
#245

changing to KUBELET_OPTS=--image-pull-progress-deadline=10m
on all my agent nodes solved my problem.

Being able to edit these should be made easier as I'm sure as more and more customers use AKS, each customer's scenario will require changing kubelet options.

[kautsig]

This seems like a duplicate of #245

[doubleyewdee]

I'm actually curious about whether and how we could use a different option (--cpu-manager-policy) passed in to KUBELET_OPTS as part of our node provisioning. Is there a generic way to do this or is the plan to expose specific functionality through other AKS knobs and have those turn on the appropriate flags indirectly for users?

[jnoller]

@doubleyewdee It would be the latter - depending on the complexity of the option (some aren't well suited for AKS as a managed solution). AKS would expose the knobs and then handle application behind the scenes.

[AlverezYari]

+1 to need to set my eviction limits

[zerda]

How about Dynamic Kubelet Configuration ?

I'm using --authentication-token-webhook=true to enable service account tokens to be used to authenticate to the kubelet.

It's required by kube-prometheus to scrape the kubelet metrics.

[henrylilei]

@jnoller I understand that certain configs are not suitable for managed AKS, but is it possible to have a blacklist/whitelist on the dynamic configurations? So that we can at least setup the cluster and VMSS better suited to the latency sensitive production traffic.

Some of the latency and perf related settings for AKS that I wish to be able to set when I have Cluster AutoScaler enabled,

• cpu-manager-policy
• cpu-cfs-quota
• pod-max-pids

Thank you for your consideration.

[faustos]

I would need to set ConfigMapAndSecretChangeDetectionStrategy

[jayush]

What is the latest status of this ask?

[jeff1985]

Custom kubelet config would also allow to solve #686 which is failing an important deployment on my cluster

[henrylilei]

After couple of trial-and-error, I settled with updating the kubelet config using Kured, please check out this gist. You need to change the init-containers script to fit your specific needs. It works with VMSS and HPA/CA, but this feels like a hack to me and I would love to see AKS to support this natively.

[kortatu]

How about Dynamic Kubelet Configuration ?

I'm using --authentication-token-webhook=true to enable service account tokens to be used to authenticate to the kubelet.

It's required by kube-prometheus to scrape the kubelet metrics.

Dynamic Kubelet configuration is great but first you need to chante this kubelet option --dynamic-config-dir as stated here https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/
So again, we need a way of specifying some options in Azure AKS nodes at creation time.

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[LudoNieuw]

I'm curious about an update to this issue. I am myself facing issues with scraping the kubelet from :2550/metrics (http scrape) but am getting a 'connection refused' message thrown back...

[m1o1]

I might be wrong about this but I believe the kubelet binds to localhost, so it's inaccessible to prometheus

[LudoNieuw]

I might be wrong about this but I believe the kubelet binds to localhost, so it's inaccessible to prometheus
Prometheus is in most cases deployed inside the cluster (also my situation), so it should be able to access it, but it turns out the http port is removed without any public mention in recent AKS rollouts...

[palma21]

AKS will soon provide for more bootstrap options to configure the node OS and kubelet for a nodepool.

[slawekww]

More options should be available for other K8S services: allow to configure additional parameters for kube-api, kube-controller-manager etc. If AKS shall play production grade cluster, we need to have capabilities to setup parameters according to guidelines.

[slawekww]

It is possible to update kubelet (any node config actually) via DaemonSet like example:
https://docs.nginx.com/nginx-service-mesh/setup/aks/

[palma21]

@slawekww any specific requests/use cases for setting parameters on those components? please do open a specific issue with those, happy to discuss there.

[arunmahadevan]

Looking for options to customize the kueblet root dir. Is this supported or planned to be supported?

--root-dir string     Default: `/var/lib/kubelet`
  | Directory path for managing kubelet files (volume mounts, etc).

[aidapsibr]

@palma21 Hey, we are looking to evaluate this preview, is there any samples you can share?

[ohadschn]

I would add setting logging verbosity (--v) to the list - could be invaluable for test clusters. Opened #2165 to track.
Related Kubernetes issue: kubernetes/kubernetes#22723.

[palma21]

https://docs.microsoft.com/en-us/azure/aks/custom-node-configuration

[venkatalolla]

@palma21 when will the custom-node-configuration is going to be GA?
https://docs.microsoft.com/en-us/azure/aks/custom-node-configuration

[sonnyhcl]

As claimed in Support for custom log options, containerLogMaxSizeMB/containerLogMaxFiles should be avaiable as a valid node config. While I tried updating my AKS cluster and doesn't work.

So I have similiar question on when containerLog related config goes GA.

[abarqawi]

@palma21 any ETA please ?

@palma21 when will the custom-node-configuration is going to be GA? https://docs.microsoft.com/en-us/azure/aks/custom-node-configuration

[MarcosMMartinez]

Is there a plan to allow for custom config of --kube-config=cpu=xxxx,memory=xxxx for customers that prefer to have more simplistic QoS configs for deployments?

[ohadschn]

For reference, I believe these are the relevant shipped features:

• https://docs.microsoft.com/en-us/azure/aks/custom-node-configuration
• https://docs.microsoft.com/en-us/azure/aks/support-policies#shared-responsibility