Logging Verbosity setting for kube* components

[ohadschn]

The following logs are available for cluster debugging:

Master

• /var/log/kube-apiserver.log - API Server, responsible for serving the API
• /var/log/kube-scheduler.log - Scheduler, responsible for making scheduling decisions
• /var/log/kube-controller-manager.log - Controller that manages replication controllers

Worker Nodes

• /var/log/kubelet.log - Kubelet, responsible for running containers on the node
• /var/log/kube-proxy.log - Kube Proxy, responsible for service load balancing

It is my understanding that they all support a -v verbosity flag (klog convention), which can be immensely useful in test clusters. Unfortunately,

• Master nodes cannot be interacted with, so the verbosity of the components they host is impossible to change.
• Worker-level system pods can be hacked to enable this (Kured, DaemonSet) but there is no built-in, proper solution.

There is also the related matter of container logs settings (number of log files, max log file size, etc): #1062

[ghost]

Hi ohadschn, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

1. If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
2. Please abide by the AKS repo Guidelines and Code of Conduct.
3. If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
4. Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
5. Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
6. If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[yuriwoof]

Same here.
It is necessary to get the more detailed log....

$ kubectl describe pvc gluster-pvc
Name:          gluster-pvc
Namespace:     default
StorageClass:  gluster-heketi-external
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-class: gluster-heketi-external
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason              Age                 From                         Message
  ----     ------              ----                ----                         -------
  Warning  ProvisioningFailed  51s (x205 over 9h)  persistentvolume-controller  Failed to provision volume with StorageClass "gluster-heketi-external": failed to create volume: failed to create volume: see kube-controller-manager.log for details

[pawelpabich]

We just hit a case that is pretty much impossible without access to detailed kube-scheduler logs. Having read only access to all logs would be amazing.

[pawelpabich]

kube-scheduler supports this flag: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/

[Kaarthis-zz]

@yuri

Same here.
It is necessary to get the more detailed log....

$ kubectl describe pvc gluster-pvc
Name:          gluster-pvc
Namespace:     default
StorageClass:  gluster-heketi-external
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-class: gluster-heketi-external
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason              Age                 From                         Message
  ----     ------              ----                ----                         -------
  Warning  ProvisioningFailed  51s (x205 over 9h)  persistentvolume-controller  Failed to provision volume with StorageClass "gluster-heketi-external": failed to create volume: failed to create volume: see kube-controller-manager.log for details

Could you please elaborate what led you to this issue - 'ProvisioningFailed' , please describe the problem use case here and can you also let us know which log details are you most interested in - Any Support cases opened on this prior would be helpful for us .

[yuriwoof]

Hi @kaarthis you may mention another Yuri...
Anyway, in my scenario, I set up GluserFS cluster and create Stroage Class & PVC.
But I couldn't investigate the reason why ProvisioningFailed occur (it's limited in log level)

Any Support cases opened on this prior would be helpful for us .

Let me loop in the MS internal email thread.

[Kaarthis-zz]

Got it @yuriwoof - which logs did you access for this and may i know if through the support case having increased verbosity helped you at all ,if so how ?

[Kaarthis-zz]

@yuriwoof any luck with the MS Internal thread ? also happy to sync with you if you are available to discuss this.

[Kaarthis-zz]

We are evaluating this for long term impact especially balancing component CCP stability vs benefits of increased logging. Closing this for now as we have communicated with Octopus group in the thread.