isConnectionDead logs warnings when the connection should simply be closed

[bh-tt]

We recently encountered an increased number of warnings from hikari, all about failing to validate a connection before borrowing it from the pool. The log line in question is HikariPool-2 - Failed to validate connection org.mariadb.jdbc.Connection@294613b0 ((conn=55230) Connection.setNetworkTimeout cannot be called on a closed connection). Possibly consider using a shorter maxLifetime value.

This appears to come from Poolbase.isConnectionDead, which does 2 setNetworkTimeout calls (both before validation to the validationTimeout and afterward to reset the network timeout back to what it was). If the connection was closed due to errors (at least for the mariadb 3.x driver) the setNetworkTimeout call itself results in an SQLException, which is then logged as a warning. The result is a large number (thousands/day) of warnings about the connection being closed from a method that is supposed to determine whether the connection is closed.

This method should not log warnings (maybe info or debug?) when a return value of false is an expected result (e.g. the connection should not be used, evict it).

I think there are 2 ways to fix this:

• lower the log level, which is low-impact
• do not call setNetworkTimeout before and after the validation, as these calls will throw an SQLException if the connection is closed, while isValid simply returns false. I see this validation was added in HikariCP not adhering connectionTimeout #867 (9e9a975), to guard against network timeouts turning connect timeouts into socket timeouts which are much longer. It seems the mariadb driver mostly ignores this timeout value (see https://github.com/mariadb-corporation/mariadb-connector-j/blob/7f2ff7eb1cfbd74f511eb235ae7fda8936c22810/src/main/java/org/mariadb/jdbc/Connection.java#L704), and if they do it many other drivers will likely do so too. This does not seem like a good option.

[bh-tt]

Well we found the underlying error: very high system load, likely causing network issues. I'll close this.