add logging and separate muxer for metrics

[jpeeler]

This is to hopefully help debug a strange issue I've seen where https metrics are not being served upon initial pod launch. (However, if the pod is restarted, metrics are served as expected.) These changes might be good enough to keep even if they don't fix the problem, especially the mux change.

(BZ #1689836)

[jpeeler]

Ok, definitely leaving in these changes!

time="2019-04-11T00:09:02Z" level=error msg="Metrics (https) serving failed: open /var/run/secrets/serving-cert/tls.crt: no such file or directory"

Will try making the secret non-optional tomorrow, and maybe that will cause the pod to fail until the secret is ready. Best as I can tell, the secrets end up in the pod over a minute after OLM is started. Extracted from inside the OLM pod:

sh-4.2$ ps aux
USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
1001          1  0.8  0.4  77864 67128 ?        Ssl  00:09   0:10 /bin/olm -writeStatusName operator-lifecycle-manager -tls-cert /var/run/secrets/serving-cert/tls.crt -tls-key /var/run/secrets/serving-cert/tls.key
1001         23  0.0  0.0  11828  2900 pts/0    Ss   00:27   0:00 /bin/sh
1001         34  0.0  0.0  51752  3528 pts/0    R+   00:28   0:00 ps aux
sh-4.2$ ls -l /var/run/secrets/serving-cert --full-time
total 0
lrwxrwxrwx. 1 root root 14 2019-04-11 00:10:29.460383912 +0000 tls.crt -> ..data/tls.crt
lrwxrwxrwx. 1 root root 14 2019-04-11 00:10:29.460383912 +0000 tls.key -> ..data/tls.key

sh-4.2$ ps -p 1 -o lstart
                 STARTED
Thu Apr 11 00:09:01 2019

Confirms metrics aren't being served, (but health is as expected):

sh-4.2$ lsof -i -P -n
lsof: no pwd entry for UID 1001
lsof: no pwd entry for UID 1001
COMMAND PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
lsof: no pwd entry for UID 1001
olm       1     1001    7u  IPv6  58483      0t0  TCP *:8080 (LISTEN)
lsof: no pwd entry for UID 1001
olm       1     1001    9u  IPv4 225600      0t0  TCP 10.128.0.9:33000->172.30.0.1:443 (ESTABLISHED)

[jpeeler]

/retest
e2e-aws passed... looks like e2e-aws-olm had a test issue

[ecordell]

Can we use fsnotify to know when the files have actually been mounted and start the serving then?

[ecordell]

/lgtm

[jpeeler]

/test unit
Just saw the metrics test pass, so I think this is ready for LGTM again.

[jpeeler]

/retest

[ecordell]

/lgtm
/retest

[ecordell]

/retest

[jpeeler]

/retest

[jpeeler]

/hold
I just realized I need to slightly modify the metrics test to not fail when metrics are being served over HTTP.

[jpeeler]

/retest

[jpeeler]

/retest
CI is not behaving today.

[ecordell]

/retest

[ecordell]

/lgtm
/hold cancel

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ecordell, jpeeler

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ecordell,jpeeler]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jpeeler]

/retest

[jpeeler]

/retest

[jpeeler]

/retest

[ecordell]

/retest

[jpeeler]

/retest

[ecordell]

/retest