[BUG] tunnel-agent crashes when the local certificate can not be loaded correctly

[SataQiu]

What happened:
For some reason, the local certificate file may be corrupted.
At this point, the tunnel-agent cannot automatically restore the damaged certificate file and continues to crash.

What you expected to happen:
The tunnel-agent can restore the damaged certificate file automatically and work well.

How to reproduce it (as minimally and precisely as possible):

1. clean the contents of the certificate file

echo "" > /var/lib/yurttunnel-agent/pki/yurttunnel-agent-2021-07-06-03-37-18.pem

2. restart the tunnel-agent Pod
3. check the Pod logs

I0706 06:18:13.923142       1 start.go:48] yurttunnel-agent version: projectinfo.Info{GitVersion:"v0.4.0", GitCommit:"9426d63", BuildDate:"2021-07-05T09:55:37Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
I0706 06:18:13.923190       1 options.go:136] ipv4=172.16.247.51&host=izbp1ikl8dfbhoc2661jufz is set for agent identifies
I0706 06:18:13.923195       1 options.go:141] neither --kube-config nor --apiserver-addr is set, will use /etc/kubernetes/kubelet.conf as the kubeconfig
I0706 06:18:13.923199       1 options.go:145] create the clientset based on the kubeconfig(/etc/kubernetes/kubelet.conf).
I0706 06:18:13.943353       1 start.go:84] yurttunnel-server address: 122.43.234.97:32502
I0706 06:18:13.943407       1 certificate_store.go:130] Loading cert/key pair from "/var/lib/yurttunnel-agent/pki/yurttunnel-agent-current.pem".
Error: failed to initialize server certificate manager: could not convert data from "/var/lib/yurttunnel-agent/pki/yurttunnel-agent-current.pem" into cert/key pair: tls: failed to find any PEM data in certificate input
Usage:
   [flags]

Flags:
      --add_dir_header                   If true, adds the file directory to the header of the log messages
      --agent-identifiers string         The identifiers of the agent, which will be used by the server when choosing agent.
      --alsologtostderr                  log to standard error as well as files
      --apiserver-addr string            A reachable address of the apiserver.
  -h, --help                             help for this command
      --kube-config string               Path to the kubeconfig file.
      --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                   If non-empty, write log files in this directory
      --log_file string                  If non-empty, use this log file
      --log_file_max_size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --logtostderr                      log to standard error instead of files (default true)
      --meta-host string                 The ip address on which listen for --meta-port port. (default "127.0.0.1")
      --meta-port string                 The port on which to serve HTTP requests like profling, metrics (default "10266")
      --node-ip string                   The host IP of the edge node.
      --node-name string                 The name of the edge node.
      --skip_headers                     If true, avoid header prefixes in the log messages
      --skip_log_headers                 If true, avoid headers when opening log files
      --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
      --tunnelserver-addr string         The address of yurttunnel-server
  -v, --v Level                          number for the log level verbosity (default 0)
      --version                          print the version information.
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging

F0706 06:18:13.943805       1 agent.go:36] yurttunnel-agent failed: failed to initialize server certificate manager: could not convert data from "/var/lib/yurttunnel-agent/pki/yurttunnel-agent-current.pem" into cert/key pair: tls: failed to find any PEM data in certificate input
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc0000c0001, 0xc0003a63c0, 0x118, 0x134)
        /go/pkg/mod/k8s.io/klog/v2@v2.0.0/klog.go:972 +0xb8
k8s.io/klog/v2.(*loggingT).output(0x2330b60, 0xc000000003, 0x0, 0x0, 0xc0003ba150, 0x2299df7, 0x8, 0x24, 0x0)
        /go/pkg/mod/k8s.io/klog/v2@v2.0.0/klog.go:921 +0x19d
k8s.io/klog/v2.(*loggingT).printf(0x2330b60, 0x3, 0x0, 0x0, 0x164ec4b, 0xd, 0xc0001a3f30, 0x2, 0x2)
        /go/pkg/mod/k8s.io/klog/v2@v2.0.0/klog.go:733 +0x17a
k8s.io/klog/v2.Fatalf(...)
        /go/pkg/mod/k8s.io/klog/v2@v2.0.0/klog.go:1427
main.main()
        /opt/src/cmd/yurt-tunnel-agent/agent.go:36 +0x218

goroutine 18 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x2330a80)
        /go/pkg/mod/k8s.io/klog@v1.0.0/klog.go:1010 +0x8b
created by k8s.io/klog.init.0
        /go/pkg/mod/k8s.io/klog@v1.0.0/klog.go:411 +0xd6

goroutine 19 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x2330b60)
        /go/pkg/mod/k8s.io/klog/v2@v2.0.0/klog.go:1107 +0x8b
created by k8s.io/klog/v2.init.0
        /go/pkg/mod/k8s.io/klog/v2@v2.0.0/klog.go:416 +0xd6

goroutine 8 [IO wait]:
internal/poll.runtime_pollWait(0x7f33dbd05fb8, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc00047c098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00047c080, 0xc00036b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc00047c080, 0xc00036b000, 0x1000, 0x1000, 0x4312ec, 0xc000014b20, 0x45b600)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000c0018, 0xc00036b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:184 +0x68
net/http.(*persistConn).Read(0xc00047a000, 0xc00036b000, 0x1000, 0x1000, 0xc00009a180, 0xc000014c20, 0x404d15)
        /usr/local/go/src/net/http/transport.go:1758 +0x75
bufio.(*Reader).fill(0xc000370060)
        /usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc000370060, 0x1, 0x0, 0x0, 0x1, 0xc000426100, 0x0)
        /usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc00047a000)
        /usr/local/go/src/net/http/transport.go:1911 +0x1d6
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1580 +0xb0d

goroutine 9 [select]:
net/http.(*persistConn).writeLoop(0xc00047a000)
        /usr/local/go/src/net/http/transport.go:2210 +0x123
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1581 +0xb32

Anything else we need to know?:

Environment:

• OpenYurt version: v0.4.0
• Kubernetes version (use kubectl version): v1.14.8
• OS (e.g: cat /etc/os-release): CentOS 7
• Kernel (e.g. uname -a): 3.10.0-1062.18.1.el7.x86_64
• Install tools:
• Others:

others

/kind bug

[rambohe-ch]

@SataQiu Very appreciate for considering certificate unexpected situation and making a solution for the bug.