Handle k8s.io/client-go warnings

[ChrsMark]

With #18817 we upgraded the dependency to k8s.io/client-go which resulted in having some warnings of the client now coming in the output of Beats whenever a k8s client is trying to get initialised. A common case for this is running Metricbeat which has add_kubernetes_metadata enabled by default which will result in output like this one:

./metricbeat modules disable system
W0604 10:46:01.769645    8048 client_config.go:559] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0604 10:46:01.769661    8048 client_config.go:566] error creating inClusterConfig, falling back to default config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
Module system is already disabled

This is normal in terms of functionality when the k8s configuration is not set up, however it is quite annoying and we should handle these outputs properly.

Related to kubernetes/client-go#18 kubernetes/client-go#610

cc: @exekias

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[exekias]

Thanks for opening, I see from the related issues that we cannot really disable client-go logging.

I wonder if, as an alternative, there is a way to check if inClusterConfig will work before starting the client, that way we could avoid the warning.

[ChrsMark]

That sounds good @exekias!

Better we could "re-implement" the BuildConfigFromFlags on our side in order to avoid this logging. Something like this:

func buildConfig(kubeconfigPath string) (*restclient.Config, error) {
	if kubeconfigPath == "" {
		kubeconfig, err := restclient.InClusterConfig()
		if err == nil {
			return kubeconfig, nil
		}
	}
	return clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
		&clientcmd.ClientConfigLoadingRules{ExplicitPath: kubeconfigPath},
		&clientcmd.ConfigOverrides{ClusterInfo: clientcmdapi.Cluster{Server: ""}}).ClientConfig()
}

instead of calling client.BuildConfigFromFlags() at

beats/libbeat/common/kubernetes/util.go

Line 52 in 5bfcc9c

cfg, err := clientcmd.BuildConfigFromFlags("", kubeconfig)

:

func BuildConfigFromFlags(masterUrl, kubeconfigPath string) (*restclient.Config, error) {
	if kubeconfigPath == "" && masterUrl == "" {
		klog.Warningf("Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.")
		kubeconfig, err := restclient.InClusterConfig()
		if err == nil {
			return kubeconfig, nil
		}
		klog.Warning("error creating inClusterConfig, falling back to default config: ", err)
	}
	return NewNonInteractiveDeferredLoadingClientConfig(
		&ClientConfigLoadingRules{ExplicitPath: kubeconfigPath},
		&ConfigOverrides{ClusterInfo: clientcmdapi.Cluster{Server: masterUrl}}).ClientConfig()
}

[exekias]

That sounds good to me, good find! I only expect further warnings if you are actually in a kubernetes scenario, where these warnings are still not ideal, but more reasonable