SharedTimer race condition

[mmimica]

Driver version

12.2.0.jre11

Problem description

There is a race-condition in SharedTimer class.

The class holds a singleton instance of a ScheduledThreadPoolExecutor and keeps a reference count. It disposes the executor instance when reference count reaches 0. Because of ineffective use of double-lock check pattern in SharedTimer#getTimer the reference counter is prone to race-conditions.

The problem escalates with dawn of "Connection Resiliency" feature which is enabled by default. It can be mitigated by setting connectRetryCount connection property to 0.

[mmimica]

Introduced by #1931

[Jeffery-Wasty]

Hi @mmimica,

Thank you for finding this, we'll take a look at the issue and the PR and get back to you on how we'd like to proceed.

[jubax]

I think I stumbled over this problem in production since upgrading from 11.2.2 to 12.2.0. See the stack trace below:

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@297df4d2[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@40b78b8e[Wrapped task = com.microsoft.sqlserver.jdbc.TDSTimeoutTask@445ae498]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@6aafff45[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562) ~[?:?]
	at com.microsoft.sqlserver.jdbc.SharedTimer.schedule(SharedTimer.java:144) ~[mssql-jdbc-12.2.0.jre11.jar:?]
	at com.microsoft.sqlserver.jdbc.SharedTimer.schedule(SharedTimer.java:134) ~[mssql-jdbc-12.2.0.jre11.jar:?]
	at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:8074) ~[mssql-jdbc-12.2.0.jre11.jar:?]
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:918) ~[mssql-jdbc-12.2.0.jre11.jar:?]
	at com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute(SQLServerStatement.java:814) ~[mssql-jdbc-12.2.0.jre11.jar:?]
	at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7685) ~[mssql-jdbc-12.2.0.jre11.jar:?]
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:4048) ~[mssql-jdbc-12.2.0.jre11.jar:?]
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:272) ~[mssql-jdbc-12.2.0.jre11.jar:?]
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:246) ~[mssql-jdbc-12.2.0.jre11.jar:?]
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeQueryInternal(SQLServerStatement.java:743) ~[mssql-jdbc-12.2.0.jre11.jar:?]
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.isValid(SQLServerConnection.java:7077) ~[mssql-jdbc-12.2.0.jre11.jar:?]
	at com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(PoolBase.java:147) ~[HikariCP-java7-2.4.13.jar:?]
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:169) ~[HikariCP-java7-2.4.13.jar:?]
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145) ~[HikariCP-java7-2.4.13.jar:?]
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85) ~[HikariCP-java7-2.4.13.jar:?]
    ... (cut of proprietary classes from trace)

[mmimica]

Yes, this is the usual consequence. Setting connectRetryCount=0 didn't really help as I initially said.