Concern about 500 Error

[cesareomacias]

Hi everyone!

I've deployed apicurio/apicurio-registry-sql:2.6.5.Final into an EC2 with Docker and I saw these errors (just 10 occurrences) and UI and API always working fine:

WARN <_> [io.apicurio.registry.metrics.health.liveness.ResponseErrorLivenessCheck] (executor-thread-84) Liveness problem suspected in ResponseErrorLivenessCheck because of an exception: : java.lang.IllegalStateException: UT010005: Cannot call getOutputStream(), getWriter() already called INFO <_> [io.apicurio.registry.metrics.health.liveness.ResponseErrorLivenessCheck] (executor-thread-84) After this event, the error counter is 1 (out of the maximum 5 allowed). WARN <_> [io.apicurio.registry.metrics.health.liveness.ResponseErrorLivenessCheck] (executor-thread-174) Liveness problem suspected in ResponseErrorLivenessCheck because of an exception: : java.lang.IllegalStateException: UT010005: Cannot call getOutputStream(), getWriter() already called ERROR <_> [io.apicurio.registry.services.http.RegistryExceptionMapperService] (executor-thread-174) [500 ERROR DETECTED] : UT010005: Cannot call getOutputStream(), getWriter() already called: java.lang.IllegalStateException: UT010005: Cannot call getOutputStream(), getWriter() already called

I would like to understand the reason or if I can do something.

Thank you for your help!

César.

[EricWittmann]

Is that the only information relevant in the logs? Is there no full stack trace? Does this happen on startup?

[cesareomacias]

Hi,

Let me share a full trace of the logs with you. The issue occurs during runtime, not just at start/restart. Additionally, the 500 error is now appearing as a 404 error for various endpoints, even for non-users.

`2024-11-13 07:48:23 INFO <> [io.apicurio.common.apps.logging.audit.AuditLogService] (executor-thread-11068) apicurio.audit action="request" result="failure" src_ip="null" path="/6003d62d7f921837125fc3292f6622f5.txt" response_code="404" method="GET" user=""
2024-11-13 07:48:24 INFO <> [io.apicurio.common.apps.logging.audit.AuditLogService] (executor-thread-11068) apicurio.audit action="request" result="failure" src_ip="null" path="/index.php/Pan/ShareUrl/downloadSharedFile" response_code="404" method="GET" user=""
2024-11-13 07:48:24 INFO <> [io.apicurio.common.apps.logging.audit.AuditLogService] (executor-thread-11068) apicurio.audit action="request" result="failure" src_ip="null" path="/index.php/component/chronoforums2/profiles/avatar/u1" response_code="404" method="GET" user=""
2024-11-13 07:48:24 INFO <> [io.apicurio.common.apps.logging.audit.AuditLogService] (executor-thread-11068) apicurio.audit action="request" result="failure" src_ip="null" path="/webjars/swagger-ui/4.4.1-1/index.html" response_code="404" method="GET" user=""
2024-11-13 07:48:24 INFO <> [io.apicurio.common.apps.logging.audit.AuditLogService] (executor-thread-11066) apicurio.audit action="request" result="failure" src_ip="null" path="/pages/setup.php" response_code="404" method="GET" user=""
2024-11-13 07:48:24 INFO <> [io.apicurio.common.apps.logging.audit.AuditLogService] (executor-thread-11068) apicurio.audit action="request" result="failure" src_ip="null" path="/.\./.\./.\./.\./.\./.\./etc/passwd" response_code="404" method="GET" user=""
2024-11-13 07:48:24 INFO <> [io.apicurio.common.apps.logging.audit.AuditLogService] (executor-thread-11066) apicurio.audit action="request" result="failure" src_ip="null" path="/webjars/swagger-ui/3.0.21/index.html" response_code="404" method="GET" user=""
2024-11-13 08:00:06 WARN <> [io.apicurio.registry.metrics.health.liveness.ResponseErrorLivenessCheck] (executor-thread-11077) Liveness problem suspected in ResponseErrorLivenessCheck because of an exception: : java.lang.IllegalStateException: UT010005: Cannot call getOutputStream(), getWriter() already called
at io.undertow.servlet.spec.HttpServletResponseImpl.getOutputStream(HttpServletResponseImpl.java:307)
at io.apicurio.registry.ui.servlets.BaseHrefFilter.doFilter(BaseHrefFilter.java:78)
at io.apicurio.registry.ui.servlets.BaseHrefFilter_ClientProxy.doFilter(Unknown Source)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.apicurio.registry.ui.servlets.HSTSFilter.doFilter(HSTSFilter.java:62)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.apicurio.registry.ui.servlets.ResourceCacheControlFilter.doFilter(ResourceCacheControlFilter.java:92)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.apicurio.registry.rest.RegistryApplicationServletFilter.doFilter(RegistryApplicationServletFilter.java:144)
at io.apicurio.registry.rest.RegistryApplicationServletFilter_ClientProxy.doFilter(Unknown Source)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.apicurio.registry.ui.servlets.RedirectFilter.doFilter(RedirectFilter.java:96)
at io.apicurio.registry.ui.servlets.RedirectFilter_ClientProxy.doFilter(Unknown Source)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:63)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:67)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:133)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:65)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:247)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:111)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:108)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$9$1.call(UndertowDeploymentRecorder.java:626)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227)
at io.undertow.servlet.handlers.ServletInitialHandler.handleRequest(ServletInitialHandler.java:152)
at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$1.handleRequest(UndertowDeploymentRecorder.java:125)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:284)
at io.undertow.server.DefaultExchangeHandler.handle(DefaultExchangeHandler.java:18)
at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$5$2.run(UndertowDeploymentRecorder.java:441)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)

2024-11-13 08:00:06 INFO <> [io.apicurio.registry.metrics.health.liveness.ResponseErrorLivenessCheck] (executor-thread-11077) After this event, the error counter is 18 (out of the maximum 5 allowed).
2024-11-13 08:00:06 ERROR <> [io.apicurio.registry.services.http.RegistryExceptionMapperService] (executor-thread-11077) [500 ERROR DETECTED] : UT010005: Cannot call getOutputStream(), getWriter() already called: java.lang.IllegalStateException: UT010005: Cannot call getOutputStream(), getWriter() already called
at io.undertow.servlet.spec.HttpServletResponseImpl.getOutputStream(HttpServletResponseImpl.java:307)
at io.apicurio.registry.ui.servlets.BaseHrefFilter.doFilter(BaseHrefFilter.java:78)
at io.apicurio.registry.ui.servlets.BaseHrefFilter_ClientProxy.doFilter(Unknown Source)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.apicurio.registry.ui.servlets.HSTSFilter.doFilter(HSTSFilter.java:62)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.apicurio.registry.ui.servlets.ResourceCacheControlFilter.doFilter(ResourceCacheControlFilter.java:92)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.apicurio.registry.rest.RegistryApplicationServletFilter.doFilter(RegistryApplicationServletFilter.java:144)
at io.apicurio.registry.rest.RegistryApplicationServletFilter_ClientProxy.doFilter(Unknown Source)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.apicurio.registry.ui.servlets.RedirectFilter.doFilter(RedirectFilter.java:96)
at io.apicurio.registry.ui.servlets.RedirectFilter_ClientProxy.doFilter(Unknown Source)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:63)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:67)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:133)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:65)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:247)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:111)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:108)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$9$1.call(UndertowDeploymentRecorder.java:626)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227)
at io.undertow.servlet.handlers.ServletInitialHandler.handleRequest(ServletInitialHandler.java:152)
at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$1.handleRequest(UndertowDeploymentRecorder.java:125)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:284)
at io.undertow.server.DefaultExchangeHandler.handle(DefaultExchangeHandler.java:18)
at io.quarkus.undertow.runtime.UndertowDeploymentRecorder$5$2.run(UndertowDeploymentRecorder.java:441)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:840)

2024-11-13 08:00:19 INFO <> [io.apicurio.common.apps.logging.audit.AuditLogService] (executor-thread-11087) apicurio.audit action="exportData" result="success" src_ip="172.31.35.45" x_forwarded_for="52.208.21.11" principal_id="e535c061-0b8f-4675-8cd0-267d6e852365" for_browser="false"
2024-11-13 08:03:51 INFO <> [io.apicurio.common.apps.logging.audit.AuditLogService] (executor-thread-11089) apicurio.audit action="request" result="failure" src_ip="null" path="/.env" response_code="404" method="GET" user=""
2024-11-13 08:15:04 INFO <> [io.apicurio.common.apps.logging.audit.AuditLogService] (executor-thread-11089) apicurio.audit action="request" result="failure" src_ip="null" path="/.env" response_code="404" method="GET" user=""
2024-11-13 08:17:05 INFO <> [io.apicurio.common.apps.logging.audit.AuditLogService] (executor-thread-11089) apicurio.audit action="request" result="failure" src_ip="null" path="/Core/Skin/Login.aspx" response_code="404" method="HEAD" user=""`

Regards

[carlesarnal]

Are you performing some sort of penetration testing? The routes you have for the requests seem to indicate that:

/.\./.\./.\./.\./.\./.\./etc/passwd
/.env
/Core/Skin/Login.aspx
/index.php/Pan/ShareUrl/downloadSharedFile

If not, better check what's going one with your server, because that is for sure not normal. None of those requests are performed by the UI or any regular client application.

[cesareomacias]

Hey @carlesarnal

Thank you for the information; I just needed to verify. I was checking, and it’s indeed strange because those requests started appearing suddenly. I’m going to put a WAF in front and see what happens.

Thank you for your help! Best.