restore state in unit tests

[pohly]

/kind feature

Describe the solution you'd like

There are several unit tests which temporarily modify global klog state, for example to capture log output. There have been several cases where this led to race conditions:

• kubelet: Race condition in nodeshutdown unit test kubernetes#108040
• Prevent race condition with bytes buffer in vsphere unit tests kubernetes#98944
• fix: race detected in TestErrConnKilled kubernetes#108553

Besides the race conditions related to byte.Buffer there's also the question on how to restore state after a test. In some recent tests I added defer klog.StopFlushDaemon, but that is a new call and many tests don't have it.

I think we should add a klog.CaptureState than can be used like this:

func TestSomething(t *testing.T) {
    defer klog.CaptureState(t)()
}

CaptureState captures the current state and returns a function that can restore that state:

• all flags
• output buffer
• flush daemon

Anything else you would like to add:

Many different tests uses a custom struct with mutex locking around byte.Buffer. This is unnecessarily complex and can be simplified as in kubernetes/kubernetes#108553 by setting a second byte.Buffer before reading the first one.

Long-term, contextual logging can completely replace SetBuffer: kubernetes/kubernetes#98944

[pohly]

/assign