Configuring load balancing service name in deployment.toml of APIM for analytics will fail Alert Configurations in Publisher

[maneeshaheshan]

Description:
When we are defining a load balancer service name with port 7444 as below for store API, it will fail intermittently when the request route to the passive node when trying to configure alerts in publisher node.

    [apim.analytics]
    enable = "true"
    store_api_url = "https://{{ template "am-pattern-3.resource.prefix" . }}-am-analytics-worker-service:7444"

So this is the error we are getting in the UI

Following error also throwing in the backend

[2020-08-31 10:27:13,138] ERROR - AlertsApiServiceImpl |2a206be8-dfc2-41ee-87d7-21b22458665c|Error while retrieving alert configurations org.wso2.carbon.apimgt.api.APIManagementException: Error while connecting to the server at org.wso2.carbon.apimgt.impl.utils.APIUtil.handleException_aroundBody78(APIUtil.java:1935) ~[org.wso2.carbon.apimgt.impl_6.6.163.jar:?] at org.wso2.carbon.apimgt.impl.utils.APIUtil.handleException(APIUtil.java:1933) ~[org.wso2.carbon.apimgt.impl_6.6.163.jar:?] at org.wso2.carbon.apimgt.impl.utils.APIUtil.executeQueryOnStreamProcessor_aroundBody624(APIUtil.java:9221) ~[org.wso2.carbon.apimgt.impl_6.6.163.jar:?] at org.wso2.carbon.apimgt.impl.utils.APIUtil.executeQueryOnStreamProcessor(APIUtil.java:9174) ~[org.wso2.carbon.apimgt.impl_6.6.163.jar:?] at org.wso2.carbon.apimgt.impl.alertmgt.PublisherAlertConfigurator.getAlertConfiguration_aroundBody6(PublisherAlertConfigurator.java:98) ~[org.wso2.carbon.apimgt.impl_6.6.163.jar:?] at org.wso2.carbon.apimgt.impl.alertmgt.PublisherAlertConfigurator.getAlertConfiguration(PublisherAlertConfigurator.java:91) ~[org.wso2.carbon.apimgt.impl_6.6.163.jar:?] at org.wso2.carbon.apimgt.rest.api.publisher.v1.impl.AlertsApiServiceImpl.getAllAlertConfigs(AlertsApiServiceImpl.java:90) [classes/:?] at org.wso2.carbon.apimgt.rest.api.publisher.v1.AlertsApi.getAllAlertConfigs(AlertsApi.java:89) [classes/:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:179) [cxf-core-3.2.8.jar:3.2.8] at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) [cxf-core-3.2.8.jar:3.2.8] at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:193) [cxf-rt-frontend-jaxrs-3.2.8.jar:3.2.8] at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:103) [cxf-rt-frontend-jaxrs-3.2.8.jar:3.2.8] at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59) [cxf-core-3.2.8.jar:3.2.8] at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96) [cxf-core-3.2.8.jar:3.2.8] at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308) [cxf-core-3.2.8.jar:3.2.8] at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-core-3.2.8.jar:3.2.8] at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:267) [cxf-rt-transports-http-3.2.8.jar:3.2.8] at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234) [cxf-rt-transports-http-3.2.8.jar:3.2.8] at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208) [cxf-rt-transports-http-3.2.8.jar:3.2.8] at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160) [cxf-rt-transports-http-3.2.8.jar:3.2.8] at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:216) [cxf-rt-transports-http-3.2.8.jar:3.2.8] at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:301) [cxf-rt-transports-http-3.2.8.jar:3.2.8] at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:225) [cxf-rt-transports-http-3.2.8.jar:3.2.8] at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) [tomcat-servlet-api_9.0.31.wso2v1.jar:?] at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:276) [cxf-rt-transports-http-3.2.8.jar:3.2.8] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat_9.0.31.wso2v1.jar:?] at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:86) [org.wso2.carbon.identity.context.rewrite.valve_1.4.0.jar:?] at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:110) [org.wso2.carbon.identity.authz.valve_1.4.0.jar:?] at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:75) [org.wso2.carbon.identity.auth.valve_1.4.0.jar:?] at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99) [org.wso2.carbon.tomcat.ext_4.6.0.jar:?] at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:49) [org.wso2.carbon.tomcat.ext_4.6.0.jar:?] at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62) [org.wso2.carbon.tomcat.ext_4.6.0.jar:?] at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:145) [org.wso2.carbon.tomcat.ext_4.6.0.jar:?] at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690) [tomcat_9.0.31.wso2v1.jar:?] at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57) [org.wso2.carbon.tomcat.ext_4.6.0.jar:?] at org.wso2.carbon.tomcat.ext.valves.RequestCorrelationIdValve.invoke(RequestCorrelationIdValve.java:119) [org.wso2.carbon.tomcat.ext_4.6.0.jar:?] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590) [tomcat_9.0.31.wso2v1.jar:?] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat_9.0.31.wso2v1.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat_9.0.31.wso2v1.jar:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[?:?] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[?:?] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[?:?] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[?:?] at java.net.Socket.connect(Socket.java:609) ~[?:?] at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:285) ~[?:?] at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:524) ~[httpclient_4.3.6.wso2v2.jar:?] at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:403) ~[httpclient_4.3.6.wso2v2.jar:?] at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177) ~[httpclient_4.3.6.wso2v2.jar:?] at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144) ~[httpclient_4.3.6.wso2v2.jar:?] at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131) ~[httpclient_4.3.6.wso2v2.jar:?] at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611) ~[httpclient_4.3.6.wso2v2.jar:?] at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446) ~[httpclient_4.3.6.wso2v2.jar:?] at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) ~[httpclient_4.3.6.wso2v2.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient_4.3.6.wso2v2.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) ~[httpclient_4.3.6.wso2v2.jar:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) ~[httpclient_4.3.6.wso2v2.jar:?] at org.wso2.carbon.apimgt.impl.utils.APIUtil.executeQueryOnStreamProcessor_aroundBody624(APIUtil.java:9203) ~[org.wso2.carbon.apimgt.impl_6.6.163.jar:?] ... 57 more

Suggested Labels:
kubernetes-apim, 3.1.0.3

Affected Product Version:
v3.1.0.3

[chirangaalwis]

Fixed.