ClassNotFoundException when deserializing session

[anton-johansson]

I'm running Tomcat 8 within a container in a Kubernetes cluster.

Under /usr/local/tomcat/lib/ I have the following (other than the standard JARs):

• hazelcast-3.11.1.jar
• hazelcast-kubernetes-1.3.1.jar
• hazelcast-tomcat8-sessionmanager-1.1.3.jar

Under /usr/local/tomcat/webapps/ROOT/WEB-INF/lib/ I have the dependencies of my webapp.

I'm using NGINX Ingress Controller for Kubernetes with Sticky sessions enabled. To simulate a failover, I'm removing my INGRESSCOOKIE. This gives me a new backend, and the session manager notices it and tries to fetch the session on the new backend. I get the following logs:

16-Jan-2019 15:14:45.237 INFO [http-apr-8080-exec-1] com.hazelcast.session.HazelcastSessionManager.findSession Sticky Session is currently enabled.Some failover occured so reading session from Hazelcast map:null
16-Jan-2019 15:14:45.244 SEVERE [http-apr-8080-exec-1] org.apache.coyote.http11.AbstractHttp11Processor.process Error processing request
 com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: org.acegisecurity.context.SecurityContextImpl
        at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:86)
        at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:75)
        at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)
        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:269)
        at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:574)
        at com.hazelcast.session.HazelcastSession.deserializeMap(HazelcastSession.java:141)
        at com.hazelcast.session.HazelcastSession.readData(HazelcastSession.java:127)
        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:160)
        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:106)
        at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:51)
        at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)
        at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:187)
        at com.hazelcast.map.impl.proxy.MapProxySupport.toObject(MapProxySupport.java:1245)
        at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:120)
        at com.hazelcast.session.HazelcastSessionManager.findSession(HazelcastSessionManager.java:231)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:2951)
        at org.apache.catalina.connector.Request.getSession(Request.java:2343)
        at org.apache.catalina.valves.CrawlerSessionManagerValve.invoke(CrawlerSessionManagerValve.java:189)
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:676)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1156)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
        at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2464)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.acegisecurity.context.SecurityContextImpl
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:288)
        at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:252)
        at com.hazelcast.nio.IOUtil$ClassLoaderAwareObjectInputStream.resolveClass(IOUtil.java:646)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1866)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1749)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2040)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
        at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:82)
        ... 26 more

It does look like an incorrect class loader is being used. The session manager exists in the generic Tomcat class loader and the classes that is put into the session exists in the webapp class loader. Pure guesses, but how do I make sure that the correct class loader is being used?

[mesutcelik]

Hi @anton-johansson ,
Are you using client/server mode or P2P?
As far as I see, we fixed similar issue for client-server mode #13

Can you please try to deploy hazelcast cluster into your k8s cluster and switch to client-server mode to see if you are experiencing any problem. You can simply install hazelcast via hazelcast helm chart

If you don't see any issue then we probably need to set WebAppClassloader in Hazelcast Config.

[anton-johansson]

Hi! Sorry I left that information out. I'm using P2P. Hazelcast is running within the application itself.

Would it be any different using Client/Server? I'm sure the deserialization process is the same either way?

Hm, how do I set the WebAppClassLoader? I'd have to do that programmatically, right? Not possible using a pure hazelcast.xml configuration?

[mesutcelik]

Actually Tomcat Manager is setting it for client/server. see the code here
I see that there is no set Classloader for P2P here before passing config object to Hazelcast.getOrCreateHazelcastInstance(config)
so it could be a bug in P2P mode.

[anton-johansson]

Hmm, that's interesting. I'll see if I can fix the issue and maybe submit a pull request.

[anton-johansson]

I've started working on a PR, see above. However, I'm having some issues. Care to take a look?

[helge79]

We also experienced this issue and solved it with #33, i.e. we initialize an own instance in a ServletContextListener and hand this over to HazelcastSessionManager.

@anton-johansson You mentioned in #56 your solution does not work. I am not quite familiar of the lifecycle events in Tomcat but to me it looks like P2PLifecycleListener may be called before you are able to configure the classloader in HazelcastSessionManager. The instance would therefore already exist and your changes would have no effect

When looking at ClientServerLifecycleListener and comparing it to P2PLifecycleListener I noticed the latter one creates an instance of Hazelcast while the first one just initializes the Config. Is this intended that way, @mesutcelik? Thanks for clarification.

[Rainfords]

When will this fix be in the next release? v1.14, I built a snapshot from master and it fixed my issue!