Errors when stopping the container

[j3rem1e]

Hello,

After upgrading to Karaf 4.4 and Pax web 8, most of my pax-exam tests shows some warnings when the container stop.
Do you have an idea where I should look to fix it ?

I didn't registered a WebSocketUpgradeFilter myself.

Thanks

09:13:24.042 [e93-518b21981322]|BundleWebApplication            | Problem undeploying Web Application "/gui" for bundle com.jb.client.web/24.0.0.SNAPSHOT: javax.servlet.ServletException: org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter is defined twice for the same context attribute key 'org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter'.  Make sure you have different init-param 'contextAttributeKey' values set

java.lang.RuntimeException: javax.servlet.ServletException: org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter is defined twice for the same context attribute key 'org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter'.  Make sure you have different init-param 'contextAttributeKey' values set
	at org.ops4j.pax.web.service.spi.model.ServerModel.runSilently(ServerModel.java:657)
	at org.ops4j.pax.web.service.internal.HttpServiceEnabled$WebAppWebContainer.sendBatch(HttpServiceEnabled.java:2647)
at org.ops4j.pax.web.extender.war.internal.model.BundleWebApplication.undeploy(BundleWebApplication.java:791)
	at org.ops4j.pax.web.extender.war.internal.model.BundleWebApplication.stop(BundleWebApplication.java:347)
	at org.ops4j.pax.web.extender.war.internal.WarExtenderContext$WabExtension.destroy(WarExtenderContext.java:796)
	at org.apache.felix.utils.extender.AbstractExtender$2.run(AbstractExtender.java:285)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.apache.felix.utils.extender.AbstractExtender.destroyExtension(AbstractExtender.java:307)
	at org.apache.felix.utils.extender.AbstractExtender.bundleChanged(AbstractExtender.java:181)
	at org.ops4j.pax.web.extender.war.internal.Activator.bundleChanged(Activator.java:149)
	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:949)
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151)
	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEventPrivileged(EquinoxEventPublisher.java:229)
	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:138)
	at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:130)
	at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor.publishModuleEvent(EquinoxContainerAdaptor.java:217)
	at org.eclipse.osgi.container.Module.publishEvent(Module.java:499)
	at org.eclipse.osgi.container.Module.doStop(Module.java:658)
	at org.eclipse.osgi.container.Module.stop(Module.java:521)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.decStartLevel(ModuleContainer.java:1888)
	at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1763)
	at org.eclipse.osgi.container.SystemModule.stopWorker(SystemModule.java:275)
	at org.eclipse.osgi.internal.framework.EquinoxBundle$SystemBundle$EquinoxSystemModule.stopWorker(EquinoxBundle.java:208)
	at org.eclipse.osgi.container.Module.doStop(Module.java:660)
	at org.eclipse.osgi.container.Module.stop(Module.java:521)
	at org.eclipse.osgi.container.SystemModule.stop(SystemModule.java:207)
	at org.eclipse.osgi.internal.framework.EquinoxBundle$SystemBundle$EquinoxSystemModule$1.run(EquinoxBundle.java:226)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.RuntimeException: javax.servlet.ServletException: org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter is defined twice for the same context attribute key 'org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter'.  Make sure you have different init-param 'contextAttributeKey' values set
	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:804)
	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1529)
	at org.eclipse.jetty.servlet.ServletHandler.addFilter(ServletHandler.java:1149)
	at org.ops4j.pax.web.service.jetty.internal.JettyServerWrapper.visitFilterStateChange(JettyServerWrapper.java:1598)
	at org.ops4j.pax.web.service.spi.task.FilterStateChange.accept(FilterStateChange.java:84)
	at org.ops4j.pax.web.service.spi.task.Batch.accept(Batch.java:430)
	at org.ops4j.pax.web.service.jetty.internal.JettyServerController.sendBatch(JettyServerController.java:168)
	at org.ops4j.pax.web.service.internal.HttpServiceEnabled$WebAppWebContainer.lambda$sendBatch$0(HttpServiceEnabled.java:2678)
	at org.ops4j.pax.web.service.spi.model.ServerModel.lambda$run$2(ServerModel.java:541)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	... 1 more
	Suppressed: java.lang.Throwable
		at org.ops4j.pax.web.service.spi.model.ServerModel.run(ServerModel.java:535)
		at org.ops4j.pax.web.service.spi.model.ServerModel.runSilently(ServerModel.java:652)
		at org.ops4j.pax.web.service.internal.HttpServiceEnabled$WebAppWebContainer.sendBatch(HttpServiceEnabled.java:2647)
		at org.ops4j.pax.web.extender.war.internal.model.BundleWebApplication.undeploy(BundleWebApplication.java:791)
		at org.ops4j.pax.web.extender.war.internal.model.BundleWebApplication.stop(BundleWebApplication.java:347)
		at org.ops4j.pax.web.extender.war.internal.WarExtenderContext$WabExtension.destroy(WarExtenderContext.java:796)
		at org.apache.felix.utils.extender.AbstractExtender$2.run(AbstractExtender.java:285)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at org.apache.felix.utils.extender.AbstractExtender.destroyExtension(AbstractExtender.java:307)
		at org.apache.felix.utils.extender.AbstractExtender.bundleChanged(AbstractExtender.java:181)
		at org.ops4j.pax.web.extender.war.internal.Activator.bundleChanged(Activator.java:149)
		at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:949)
		at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
		at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151)
		at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEventPrivileged(EquinoxEventPublisher.java:229)
		at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:138)
		at org.eclipse.osgi.internal.framework.EquinoxEventPublisher.publishBundleEvent(EquinoxEventPublisher.java:130)
		at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor.publishModuleEvent(EquinoxContainerAdaptor.java:217)
		at org.eclipse.osgi.container.Module.publishEvent(Module.java:499)
		at org.eclipse.osgi.container.Module.doStop(Module.java:658)
		at org.eclipse.osgi.container.Module.stop(Module.java:521)
		at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.decStartLevel(ModuleContainer.java:1888)
		at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1763)
		at org.eclipse.osgi.container.SystemModule.stopWorker(SystemModule.java:275)
		at org.eclipse.osgi.internal.framework.EquinoxBundle$SystemBundle$EquinoxSystemModule.stopWorker(EquinoxBundle.java:208)
		at org.eclipse.osgi.container.Module.doStop(Module.java:660)
		at org.eclipse.osgi.container.Module.stop(Module.java:521)
		at org.eclipse.osgi.container.SystemModule.stop(SystemModule.java:207)
		at org.eclipse.osgi.internal.framework.EquinoxBundle$SystemBundle$EquinoxSystemModule$1.run(EquinoxBundle.java:226)
		... 1 more
Caused by: javax.servlet.ServletException: org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter is defined twice for the same context attribute key 'org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter'.  Make sure you have different init-param 'contextAttributeKey' values set
	at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.setToAttribute(WebSocketUpgradeFilter.java:479)
	at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.init(WebSocketUpgradeFilter.java:453)
	at org.ops4j.pax.web.service.spi.servlet.OsgiInitializedFilter.init(OsgiInitializedFilter.java:115)
	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:140)
	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:799)
	... 15 more
09:13:24.054 [ove HttpService)]|WebContainerManager             | Can't get a WebContainer service from {org.osgi.service.http.HttpService, org.ops4j.pax.web.service.WebContainer}={org.ops4j.pax.web.log.ncsa.extended=true, org.ops4j.pax.web.ssl.keystore.password=********, service.scope=bundle, org.ops4j.pax.web.ssl.key.password=********, org.ops4j.pax.web.session.cookie.name=JSESSIONID, org.osgi.service.http.connector.name=default, org.ops4j.pax.web.enc.iterationcount=1000, org.ops4j.pax.web.log.ncsa.file=yyyy_mm_dd.request.log, org.ops4j.pax.web.server.eventDispatcherThreadCount=1, org.osgi.service.http.checkForwardedHeaders=false, org.ops4j.pax.web.enc.suffix=), org.ops4j.pax.web.digestAuth.maxNonceAge=60000, org.ops4j.pax.web.ssl.ciphersuites.included=, org.ops4j.pax.web.session.url=jsessionid, org.ops4j.pax.web.formAuth.errorRedirect=false, org.ops4j.pax.web.ssl.ciphersuites.excluded=^.*_(MD5|SHA|SHA1)$,^TLS_RSA_.*$,^SSL_.*$,^.*_NULL_.*$,^.*_anon_.*, org.ops4j.pax.web.enc.prefix=ENC(, org.ops4j.pax.web.server.maxThreads=200, org.ops4j.pax.web.validatePeerCerts=false, service.id=154, org.ops4j.pax.web.session.cookie.maxAge=-1, org.ops4j.pax.web.ssl.truststore.password=********, org.ops4j.pax.web.ssl.session.cacheSize=-1, org.ops4j.pax.web.ssl.truststore.type=JKS, org.ops4j.pax.web.ssl.keystore.type=JKS, org.ops4j.pax.web.ssl.session.enabled=true, org.ops4j.pax.web.ssl.protocol=TLSv1.3, org.osgi.service.http.port=8201, org.ops4j.pax.web.log.ncsa.file.date.format=yyyy-MM-dd, org.ops4j.pax.web.ssl.renegotiationLimit=-1, org.osgi.service.http.secure.enabled=false, org.osgi.service.http.enabled=true, org.ops4j.pax.web.server.idleTimeout=60000, org.ops4j.pax.web.log.ncsa.retaindays=90, org.ops4j.pax.web.log.ncsa.logtimezone=GMT, org.ops4j.pax.web.enc.algorithm=PBEWithHmacSHA256AndAES_128, org.ops4j.pax.web.validateCerts=false, org.ops4j.pax.web.config.files=/home/jenkins/workspace/itests/target/paxexam/08fb7d64-e330-4d88-adcc-14f7c2975101/etc/jetty.xml, org.ops4j.pax.web.ssl.renegotiationAllowed=true, org.ops4j.pax.web.digestAuth.maxNonceCount=1024, org.ops4j.pax.web.enableOCSP=false, org.ops4j.pax.web.ssl.clientauth.needed=false, org.ops4j.pax.web.enc.enabled=false, org.osgi.service.http.port.secure=8443, javax.servlet.context.tempdir=/home/jenkins/workspace/itests/target/paxexam/08fb7d64-e330-4d88-adcc-14f7c2975101/data/pax-web/tmp, org.ops4j.pax.web.enableCRLDP=false, org.ops4j.pax.web.server.connector.idleTimeout=30000, org.ops4j.pax.web.session.timeout=30, org.ops4j.pax.web.ssl.clientauth.wanted=false, org.ops4j.pax.web.ssl.protocols.excluded=SSL,SSLv2,SSLv2Hello,SSLv3, service.bundleid=344, org.ops4j.pax.web.server.minThreads=0, org.ops4j.pax.web.session.cookie.secure=false, org.ops4j.pax.web.enc.masterpassword=********, org.ops4j.pax.web.log.ncsa.append=true, org.osgi.service.http.secure.connector.name=secureDefault, org.ops4j.pax.web.ssl.session.timeout=-1, org.ops4j.pax.web.listening.addresses=0.0.0.0, org.ops4j.pax.web.log.ncsa.buffered=true, org.ops4j.pax.web.session.cookie.sameSite=unset, org.ops4j.pax.web.log.ncsa.enabled=false, org.ops4j.pax.web.ssl.protocols.included=, org.ops4j.pax.web.session.cookie.httpOnly=true, org.ops4j.pax.web.server.showStacks=false}

[grgrzybek]

This filter is picked up automatically from javax-websocket-server-impl-9.4.54.v20240208.jar!/META-INF/services/javax.servlet.ServletContainerInitializer which is added to the context and run when the context (re)starts.
There's some complex logic involved to clean things up when context is restarted (so SCIs can run as if it was their first time), but still we may miss something.

Unfortunately I'd need a bit more than just a stack trace... if you could share your Pax Exam tests it'd help a lot identifying the problem.

[j3rem1e]

PaxExam here launch a custom karaf distribution which embedded a WAB. this war is not used in the test, and it publishes only resources, no servlet or websocket. I can reproduce the log when I shutdown the Karaf instance without pax exam.

[grgrzybek]

What's the WAB I could check? And what's the karaf-maven-plugin configuration (may be trimmed to leave the essentials)?