TCP healthchecks (SYN/FIN) cause noise in logs with TLS enabled

[benbuzbee]

Nomad version

Nomad v0.10.3 (65af1b9ecff5b55a1dd6e10b8c3224f896d6c9fa

Operating system and Environment details

$ uname -a
Linux ip-192-168-69-193 4.15.0-1058-aws #60-Ubuntu SMP Wed Jan 15 22:35:20 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"

Issue

If TLS is enabled, then even with log_level = "WARN", nomad will log any time a TLS handshake fails.

This causes extreme noise in, for example, the case where we have a load balancer performing TCP healthchecks against nomad. In this case the TCP connection is closed after it is successfully established, and nomad logs an error. In our specific case, we are using an amazon network load balancer to load balance to API servers with TCP healthchecks so we do not need to configure the load balancer health checks to use mTLS and our logs are spammed like so:

Mar 22 22:56:01 ip-192-168-1-4 nomad[2606]:     2020/03/22 22:56:01.042365 http: TLS handshake error from 192.168.6.255:10068: EOF
Mar 22 22:56:02 ip-192-168-1-4 nomad[2606]:     2020/03/22 22:56:02.279376 http: TLS handshake error from 192.168.6.255:65209: EOF
Mar 22 22:56:04 ip-192-168-1-4 nomad[2606]:     2020/03/22 22:56:04.776674 http: TLS handshake error from 192.168.6.255:6788: EOF
Mar 22 22:56:05 ip-192-168-1-4 nomad[2606]:     2020/03/22 22:56:05.438893 http: TLS handshake error from 192.168.6.255:48011: EOF
Mar 22 22:56:05 ip-192-168-1-4 nomad[2606]:     2020/03/22 22:56:05.457150 http: TLS handshake error from 192.168.6.255:3633: EOF
Mar 22 22:56:05 ip-192-168-1-4 nomad[2606]:     2020/03/22 22:56:05.704746 http: TLS handshake error from 192.168.6.255:59939: EOF

Even with the least verbose log level we have this problem. At a minimum I feel like this shouldn't log on log level WARN (also it's not logging as json even though log_json=true.
Truthfully I feel like this shouldn't log at all - this is a common scenario, and also provides an extra easy way to dos nomad by causing it to log to disk like mad - but at least we should be able to not log EOF, or not log on WARN, to unblock this scenario.

[notnoop]

I agree, the logging seems unnecessary. We should downgrade these warnings to TRACE or DEBUG level if we decide to keep them.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.