Observing many logs in the format "Connecting to %s"

[tomfotherby]

We are using livenessprobe v2.2.0 and see many logs in stderr that look like this:

1 connection.go:153] Connecting to unix:///csi/csi.sock

I tracked the line of code that generates these log messages to the golang connection library:

klog.V(5).Infof("Connecting to %s", address)

Source: https://github.com/kubernetes-csi/csi-lib-utils/blob/v0.9.1/connection/connection.go#L153 . Although the V(5) was added only 3 months ago, 29/Dec/20 in kubernetes-csi/csi-lib-utils@75fbafd.

We configure the livenessprobe sidecar like this:

      - args:
        - --csi-address=/csi/csi.sock
        - --health-port=9808
        - --v=3
        image: k8s.gcr.io/sig-storage/livenessprobe:v2.2.0
        imagePullPolicy: IfNotPresent
        name: liveness-probe
        resources:
          limits:
            cpu: 50m
            memory: 32Mi
          requests:
            cpu: 10m
            memory: 8Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /csi
          name: socket-dir

My understanding was that the V(5) in the Connection logging whould mean that if we used --v=5 we would see the log, but since we use --v=3, we shouldn't see it.

Are we doing something wrong? or do we simply need to wait until v2.3 to pickup the new V(5) added to the connection library logging?

---

I noticed in the v2.2 CHANGELOG, there was a PR to reduce the default log level of the livenessprobe-sidecar to '4' (#88).

---

Currently we get about 20,000 logs like this per hour, which is wasting space in ElasticSearch. As a workaround we can filter out these logs with a logstash filter, for example:

    filter {
      if [kubernetes][container][name] == "liveness-probe" {
        if "Connecting to unix:///csi/csi.sock" in [message] {
          drop{}
        }
      }
    }

[sylr]

You are using k8s.gcr.io/sig-storage/livenessprobe:v2.2.0 as I am and the fix you mention in kubernetes-csi/csi-lib-utils@75fbafd in not present in csi-lib-utils@v0.9.0 which is the version used in livenessprobe:v2.2.0.

Master has been upgraded to csi-lib-utils@v0.9.1 but has yet been released.

@pohly would it be possible to release a new version ? I'd really like to get rid of this spurious logging as well.

[pohly]

A new release of livenessprobe might be prepared for the Kubernetes 1.21 update of the sidecars: https://github.com/orgs/kubernetes-csi/projects/38#card-58114079

/cc @jsafrane

Jan, you wanted to check livenessprobe dependencies. Do we need to update those before doing a new release?

[jsafrane]

I checked livenessprobe and it's fine, we don't need to update deps.

[pohly]

Will we do a new release anyway?

[msau42]

Yeah let's do a release to get this logging fix.

[ialidzhikov]

livenessprobe@v2.3.0 is out with a fix for this issue. @tomfotherby , can you check the new version and verify that it is fixing the issue?

[tomfotherby]

@ialidzhikov - I tested it on a small staging cluster. It works a treat:

[ialidzhikov]

@tomfotherby, thank you for testing. Can we then close this issue?

[tomfotherby]

Yes please