Problems with log configurations

[pierluigilenoci]

What steps did you take and what happened:

• We have configured the pod to generate logs at verbosity 1 (WARNING) but still log type 0 (INFO)

What did you expect to happen:

• No more logs are generated for level 0 (INFO) if the level is 1 (WARNING)

Anything else you would like to add:

Chart configuration:

logVerbosity: 1
logFormatJSON: true
secrets-store-csi-driver:
  logLevel:
    debug: false
  logVerbosity: 1
  logFormatJSON: true
  livenessProbe:
    logLevel: 1
  enableSecretRotation: true
  rotationPollInterval: 2m

Log with verbosity 1 (WARNING) in JSON format

{"ts":1612521013516.2664,"msg":"fetching object from key vault","v":0,"objectName":"[redacted]","objectType":"secret","keyvault":"[redacted]","pod":"[redacted]/[redacted]"}
{"ts":1612521013516.3215,"msg":"using pod identity to retrieve token","v":0,"pod":"[redacted]/[redacted]"}
{"ts":1612521013526.9822,"msg":"successfully acquired access token","v":0,"accessToken":"eyJ0##### REDACTED #####7sQw","clientID":"7cbd##### REDACTED #####5d53","pod":"[redacted]/[redacted]"}
{"ts":1612521013549.5703,"msg":"successfully wrote file","v":0,"file":"[redacted]","pod":"[redacted]/[redacted]"}

verbosity is 0

Ref: kubernetes-sigs/secrets-store-csi-driver#442

Which access mode did you use to access the Azure Key Vault instance:

Pod Identity https://github.com/Azure/aad-pod-identity

Environment:

• Secrets Store CSI Driver version: 0.0.19
• Azure Key Vault provider version: 0.0.12
• Kubernetes version: Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"c86ad89b8715ed17fd55b87cbb2888ccc6fa9878", GitTreeState:"clean", BuildDate:"2020-09-25T01:53:27Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
• Cluster type: AKS

[pierluigilenoci]

@aramase could you please take a look?

[pierluigilenoci]

I also tried to set level 2 (ERROR):

Chart configuration:

logVerbosity: 2
logFormatJSON: true
secrets-store-csi-driver:
  logLevel:
    debug: false
  logVerbosity: 2
  logFormatJSON: true
  livenessProbe:
    logLevel: 2
  enableSecretRotation: true
  rotationPollInterval: 2m

provider-azure-installer logs with verbosity 2 (ERROR) in JSON format

{"ts":1612524964758.1375,"msg":"fetching object from key vault","v":0,"objectName":"[redacted]","objectType":"secret","keyvault":"[redacted]","pod":"[redacted]/[redacted]"}
{"ts":1612524964758.1567,"msg":"vaultName: [redacted]\n","v":2}
{"ts":1612524964758.2212,"msg":"using pod identity to retrieve token","v":0,"pod":"[redacted]/[redacted]"}
{"ts":1612524964767.8184,"msg":"successfully acquired access token","v":0,"accessToken":"eyJ0##### REDACTED #####pyAw","clientID":"132a##### REDACTED #####fdc1","pod":"[redacted]/[redacted]"}
{"ts":1612524965008.1086,"msg":"successfully wrote file","v":0,"file":"[redacted]","pod":"[redacted]/[redacted]"}

Other extracted logs that did not appear before:

{"ts":1612524965988.1472,"msg":"GRPC call: /grpc.health.v1.Health/Check\n","v":2}
{"ts":1612524965988.2139,"msg":"GRPC request: {}\n","v":2}
{"ts":1612524965988.2312,"msg":"GRPC response: {\"status\":1}\n","v":2}
{"ts":1612524973189.5605,"msg":"GRPC call: /grpc.health.v1.Health/Check\n","v":2}
{"ts":1612524973189.6375,"msg":"GRPC request: {}\n","v":2}
{"ts":1612524973189.6663,"msg":"GRPC response: {\"status\":1}\n","v":2}
{"ts":1612524976088.028,"msg":"GRPC call: /grpc.health.v1.Health/Check\n","v":2}
{"ts":1612524976088.1003,"msg":"GRPC request: {}\n","v":2}
{"ts":1612524976088.1223,"msg":"GRPC response: {\"status\":1}\n","v":2}
{"ts":1612524983188.8538,"msg":"GRPC call: /grpc.health.v1.Health/Check\n","v":2}
{"ts":1612524983188.923,"msg":"GRPC request: {}\n","v":2}
{"ts":1612524983188.943,"msg":"GRPC response: {\"status\":1}\n","v":2}
{"ts":1612524986087.3892,"msg":"GRPC call: /grpc.health.v1.Health/Check\n","v":2}

With verbosity at level 2 (ERROR), other messages appeared that did not appear before, those of level 2. It seems that the filter is implemented backward.

@aramase can you help me better understand how error levels work?

I was referring to the levels of glog/klog:

The numbers of severity levels INFO, WARNING, ERROR, and FATAL are 0, 1, 2, and 3, respectively

Ref: http://rpg.ifi.uzh.ch/docs/glog.html

[aramase]

This project uses klog for logging. This is the same logging framework used by Kubernetes.

No more logs are generated for level 0 (INFO) if the level is 1 (WARNING)

These are not severity levels. The logs levels indicate the klog v level logging:

-v=0
	Enable V-leveled logging at the specified level.

Changing the loglevel to higher values will enable debug logs that are marked with higher level.

[pierluigilenoci]

So the fix on secrets-store-csi-driver will fix also this side? Cool.
Thank you again.

[dkirrane]

Looking for the same.
What are the exact Helm verbosity & loglevel values required to only log WARNING and ERROR for all containers deployed by the Chart?

[pierluigilenoci]

@dkirrane great question!

[aramase]

Closing this issue in favor of kubernetes/klog#212. There is no option to only pick individual log levels. There is stderrthreshold which when set will log everything above the threshold. This however only takes effect when logs are not printed to stderr which would mean no logs will be available in kubectl logs if enabled.