Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[3.2.0] Move Key Manager Deployment From Stateless to Stateful #436

Closed
4 tasks done
chirangaalwis opened this issue Aug 12, 2020 · 2 comments · Fixed by #439
Closed
4 tasks done

[3.2.0] Move Key Manager Deployment From Stateless to Stateful #436

chirangaalwis opened this issue Aug 12, 2020 · 2 comments · Fixed by #439
Assignees
@chirangaalwis
Labels
Severity/Critical Type/Task
Milestone
3.2.0.1

Comments

@chirangaalwis
Copy link
Member

chirangaalwis commented Aug 12, 2020 •
edited
Loading

Description:
As of now, Key Manager profile of WSO2 API Manager is considered stateless. Thus, we use a Kubernetes Deployment resource to define its deployment.

But due to persisting IAM specific database duplicate entry issues during the server startup when starting multiple instances in parallel, it has been suggested to use Kubernetes StatefulSets to start the instances sequentially in a distributed deployment.

[2020-08-11 08:57:18,907] ERROR - IdPManagementServiceComponent Error while activating Identity Provider Management bundle
java.lang.Exception: Error when adding Resident Identity Provider entry for super tenant 
	at org.wso2.carbon.idp.mgt.internal.IdPManagementServiceComponent.addSuperTenantIdp(IdPManagementServiceComponent.java:428) ~[org.wso2.carbon.idp.mgt_5.17.5.jar:?]
	at org.wso2.carbon.idp.mgt.internal.IdPManagementServiceComponent.activate(IdPManagementServiceComponent.java:234) [org.wso2.carbon.idp.mgt_5.17.5.jar:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
	at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
	at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
	at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
	at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
	at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
	at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:113) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:985) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:866) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:525) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:544) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.wso2.carbon.identity.core.internal.IdentityCoreServiceComponent.activate(IdentityCoreServiceComponent.java:171) [org.wso2.carbon.identity.core_5.17.5.jar:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
	at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
	at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
	at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
	at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
	at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
	at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222) [org.eclipse.equinox.ds_1.4.400.v20160226-2036.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:113) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:985) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:866) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:525) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:544) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:529) [org.wso2.carbon.core_4.6.0.jar:?]
	at org.wso2.carbon.core.init.CarbonServerManager.removePendingItem(CarbonServerManager.java:305) [org.wso2.carbon.core_4.6.0.jar:?]
	at org.wso2.carbon.core.init.PreAxis2ConfigItemListener.bundleChanged(PreAxis2ConfigItemListener.java:118) [org.wso2.carbon.core_4.6.0.jar:?]
	at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:973) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
	at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345) [org.eclipse.osgi_3.14.0.v20190517-1309.jar:?]
Caused by: org.wso2.carbon.idp.mgt.IdentityProviderManagementException: Error occurred while adding Identity Provider for tenant -1234
	at org.wso2.carbon.idp.mgt.dao.IdPManagementDAO.addIdPWithResourceId(IdPManagementDAO.java:2290) ~[org.wso2.carbon.idp.mgt_5.17.5.jar:?]
	at org.wso2.carbon.idp.mgt.dao.CacheBackedIdPMgtDAO.addIdP(CacheBackedIdPMgtDAO.java:399) ~[org.wso2.carbon.idp.mgt_5.17.5.jar:?]
	at org.wso2.carbon.idp.mgt.IdentityProviderManager.addResidentIdP(IdentityProviderManager.java:917) ~[org.wso2.carbon.idp.mgt_5.17.5.jar:?]
	at org.wso2.carbon.idp.mgt.internal.IdPManagementServiceComponent.addSuperTenantIdp(IdPManagementServiceComponent.java:426) ~[org.wso2.carbon.idp.mgt_5.17.5.jar:?]
	... 50 more
Caused by: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '-1234-LOCAL' for key 'TENANT_ID'
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:117) ~[mysql-connector-java-8.0.17.jar:8.0.17]
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97) ~[mysql-connector-java-8.0.17.jar:8.0.17]
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[mysql-connector-java-8.0.17.jar:8.0.17]
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953) ~[mysql-connector-java-8.0.17.jar:8.0.17]
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1092) ~[mysql-connector-java-8.0.17.jar:8.0.17]
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1040) ~[mysql-connector-java-8.0.17.jar:8.0.17]
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1340) ~[mysql-connector-java-8.0.17.jar:8.0.17]
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:1025) ~[mysql-connector-java-8.0.17.jar:8.0.17]
	at jdk.internal.reflect.GeneratedMethodAccessor72.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114) ~[jdbc-pool_9.0.16.wso2v1.jar:?]
	at com.sun.proxy.$Proxy53.executeUpdate(Unknown Source) ~[?:?]
	at org.wso2.carbon.idp.mgt.dao.IdPManagementDAO.addIdPWithResourceId(IdPManagementDAO.java:2213) ~[org.wso2.carbon.idp.mgt_5.17.5.jar:?]
	at org.wso2.carbon.idp.mgt.dao.CacheBackedIdPMgtDAO.addIdP(CacheBackedIdPMgtDAO.java:399) ~[org.wso2.carbon.idp.mgt_5.17.5.jar:?]
	at org.wso2.carbon.idp.mgt.IdentityProviderManager.addResidentIdP(IdentityProviderManager.java:917) ~[org.wso2.carbon.idp.mgt_5.17.5.jar:?]
	at org.wso2.carbon.idp.mgt.internal.IdPManagementServiceComponent.addSuperTenantIdp(IdPManagementServiceComponent.java:426) ~[org.wso2.carbon.idp.mgt_5.17.5.jar:?]
	... 50 more

Sub Tasks:

  • Kubernetes and Helm resources for deployment pattern 2
  • Kubernetes and Helm resources for deployment pattern 3
  • Remove unnecessary input value options from values.yaml files
  • Code/Peer review

Affected Product Version:
Helm resources for WSO2 API Management version 3.2.0.1 and above

Related Issues:
wso2/kubernetes-is#7
@chirangaalwis chirangaalwis added this to the 3.2.0.1 milestone Aug 12, 2020
@chirangaalwis chirangaalwis self-assigned this Aug 12, 2020
This was referenced Aug 12, 2020
Open
Merged
@chirangaalwis
Copy link
Member Author

This issue is addressed in pattern 1 Helm resources using a Kubernetes InitContainer defined in API Manager instance 2 deployment, which waits for instance 1 to startup (see 8a1a06f).

We are unable to use Kubernetes StatefulSets due to #397.

@chirangaalwis chirangaalwis mentioned this issue Aug 14, 2020
Merged
@chirangaalwis chirangaalwis linked a pull request Aug 14, 2020 that will close this issue
[3.2.0] Introduce Kubernetes and Helm Resources for WSO2 API Management Pattern 3 #439
Merged
@chirangaalwis
Copy link
Member Author

Fixed in referenced PRs. Thus, closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chirangaalwis chirangaalwis

Labels
Severity/Critical Type/Task
Projects
None yet
Milestone
3.2.0.1
1 participant
@chirangaalwis