Tyrus send partial message more than buffer size will cause exception in catalina OutputBuffer. In original glassfish 4.1 (not nightly build) works fine. Tyrus 1.10 with original glassfish 4.1 works fine too.

[HighTower1991]

java.lang.IllegalStateException: Cannot set WriteListener for non-async or non-upgrade request
at org.apache.catalina.connector.OutputBuffer.setWriteListener(OutputBuffer.java:536)
at org.apache.catalina.connector.CoyoteOutputStream.setWriteListener(CoyoteOutputStream.java:223)
at org.glassfish.tyrus.servlet.TyrusServletWriter.write(TyrusServletWriter.java:140)
at org.glassfish.tyrus.core.ProtocolHandler.write(ProtocolHandler.java:486)
at org.glassfish.tyrus.core.ProtocolHandler.send(ProtocolHandler.java:274)
at org.glassfish.tyrus.core.ProtocolHandler.send(ProtocolHandler.java:269)
at org.glassfish.tyrus.core.ProtocolHandler.stream(ProtocolHandler.java:402)
at org.glassfish.tyrus.core.TyrusWebSocket.sendBinary(TyrusWebSocket.java:406)
at org.glassfish.tyrus.core.TyrusWebSocket.sendBinary(TyrusWebSocket.java:392)
at org.glassfish.tyrus.core.TyrusRemoteEndpoint$Basic.sendBinary(TyrusRemoteEndpoint.java:141)

[HighTower1991]

Test case may be simple. Even if send not patial message but more big than buffer size from server endpoint then client session close with buffer overflow and on server will be the same exception. I use annotated endpoint on client and server and send response message to client by session.getAsyncEndpoint().sendBinary and the same exception occur.

[peteral]

I am getting the same exception, however from my investigations it looks to be happening when sending an object message through Async when the Session is currently occupied delivering a previous telegram to client (OutputStream still busy).

Stack trace:

Caused by: java.lang.IllegalStateException: Cannot set WriteListener for non-async or non-upgrade request
    at org.apache.catalina.connector.OutputBuffer.setWriteListener(OutputBuffer.java:536)
    at org.apache.catalina.connector.CoyoteOutputStream.setWriteListener(CoyoteOutputStream.java:223)
    at org.glassfish.tyrus.servlet.TyrusServletWriter.write(TyrusServletWriter.java:140)
    at org.glassfish.tyrus.core.ProtocolHandler.write(ProtocolHandler.java:486)
    at org.glassfish.tyrus.core.ProtocolHandler.send(ProtocolHandler.java:274)
    at org.glassfish.tyrus.core.ProtocolHandler.send(ProtocolHandler.java:269)
    at org.glassfish.tyrus.core.ProtocolHandler.send(ProtocolHandler.java:320)
    at org.glassfish.tyrus.core.TyrusWebSocket.sendText(TyrusWebSocket.java:306)
    at org.glassfish.tyrus.core.TyrusRemoteEndpoint.sendSyncObject(TyrusRemoteEndpoint.java:400)
    at org.glassfish.tyrus.core.TyrusRemoteEndpoint$Async.sendAsync(TyrusRemoteEndpoint.java:295)
    at org.glassfish.tyrus.core.TyrusRemoteEndpoint$Async.sendObject(TyrusRemoteEndpoint.java:256)
    at eisenmann.logic.vstore.hmi.server.SessionRepositoryBean.sendObject(SessionRepositoryBean.java:212)

Tried to work around the problem by using Basic and Locks in EJB Thread pool to only use a Session when idle, but sometimes tyrus keeps waiting for a future forewer and this blocks the whole EJB thread pool and completely blocks the application server:

__ejb-thread-pool47" thread-id: 393 thread-state: WAITING Waiting on lock: java.util.concurrent.CountDownLatch$Sync@1b5a87c2
     at: sun.misc.Unsafe.park(Native Method)
     at: java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
     at: java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
     at: java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
     at: java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
     at: java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
     at: org.glassfish.tyrus.core.TyrusFuture.get(TyrusFuture.java:77)
     at: org.glassfish.tyrus.core.TyrusRemoteEndpoint$Basic.sendObject(TyrusRemoteEndpoint.java:177)
     at: eisenmann.logic.vstore.hmi.server.SessionRepositoryBean.sendObject(SessionRepositoryBean.java:212)

Not sure how to reproduce the problem for I only get it in production environment. Was not able to simulate it yet in the stress tests. Probably because they only run via loopback and don't cover the diversity of the real production environment. Will try to make a stress test including real network communication, maybe this helps.

We have following configuration:

• HMI system visualising production plants, JEE7 / HTML5 + GWT
• Clients Chrome and Firefox around 100
• Observed machines causing transactions in the system around 250
• There is a relatively high amount of data transferred via the websocket (current status of the installation, client registers to "variables" and gets informed when they change)
• also having problems with glassfish in this area although they manifest a little differenty, that's why I started evaluating payara (commercial support also looks interesting after oracle gave it up for glassfish)
• currently I have the most stable environment in this area with glassfish 4.0 but this version has tons of other issues

Not sure what to try next, any idea any1?

[HighTower1991]

Make simple test server and client: https://github.com/HighTower1991/PayaraTest

[smillidge]

Thanks that will help a lot

[smillidge]

Hi I tried to reproduce this with your test case on the latest Payara 4.1.153 but I do not get an error. I get;

On the server

[#|2015-08-08T21:31:11.481+0100|INFO|Payara 4.1||_ThreadID=102;_ThreadName=Thread-8;_TimeMillis=1439065871481;_LevelValue=800;|
  Hello from client: java.nio.HeapByteBuffer[pos=0 lim=0 cap=0]|#]

On the client

Hello from server: java.nio.HeapByteBuffer[pos=0 lim=5000000 cap=5000000]

[smillidge]

Hi I can't reproduce on Payara 4.1.152.1 either. Is there anything special that needs to be done?

[smillidge]

I've reproduced this by putting a loop in the client to send multiple messages.

[smillidge]

@peteral I suspect you are only seeing this with object messages as they exceed a size threshold. Hopefully the referenced pull request will fix your issue as well.

[peteral]

@smillidge it's possible, we do use object messages and the content is quite big. I am on vacation now but will test the pull request when I am back in office (end of August).

Thanks a lot guys...