Redeploying an Eureka server does not stop all threads and timers

[windmueller]

Our web application uses spring-cloud-netflix-eureka-server in version 2.1.1.RELEASE. When it is redeployed without restarting the application server, we get messages like

org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web
application [MyWebApp] created a ThreadLocal with key of type [java.lang.ThreadLocal]
(value [java.lang.ThreadLocal@f049ac9]) and a value of type [com.netflix.eureka.Version]
(value [V2]) but failed to remove it when the web application was stopped. Threads are going
to be renewed over time to try and avoid a probable memory leak.

and

org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web
application [MyWebApp] appears to have started a thread named [Eureka-CacheFillTimer]
but has failed to stop it. This is very likely to create a memory leak.

The latter is shown for

• ReplicaAwareInstanceRegistry - RenewalThresholdUpdater
• Eureka-JerseyClient-Conn-Cleaner
• Eureka-CacheFillTimer

For example, the CacheFillTimer is started in ResponseCacheImpl as a daemon but never stopped. This results in errors like

org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading
Illegal access: this web application instance has been stopped already. Could not load [ch.qos.logback.classic.spi.ThrowableProxy].
The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [ch.qos.logback.classic.spi.ThrowableProxy].

These issues can cause resource problems on the application server. It would be great if Eureka stops all timers and threads on shutting the application for redeployment.

[troshko111]

Thanks for the report @stovocor, this sounds reasonable and if anyone's interested in addressing this we can discuss the fix here.

Considering that we never redeploy in place and the workaround is simple (server restart), we will not be fixing this ourselves.

[sepgh]

Which version to use for this fix?
is 2020.0.0-M1 ok? Or Hoxton.RC2 works too?

[troshko111]

Releases -> Search for the commit msg / hash you're interested in.
In this case you want https://github.com/Netflix/eureka/releases/tag/v1.9.14 and above.

is 2020.0.0-M1 ok? Or Hoxton.RC2 works too?

No idea what these strings are, they are not Eureka versions.

[spencergibb]

Those are spring cloud release trains. Hoxton.SR5 is the latest that you shsould be using in production. https://search.maven.org/artifact/org.springframework.cloud/spring-cloud-netflix-dependencies/2.2.3.RELEASE/pom