Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yikes! KeeperErrorCode = Unimplemented for /kafka-manager/mutex Try again. #748

Open
lifa1225 opened this issue Mar 11, 2020 · 18 comments
Open

Comments

@lifa1225
Copy link

hi,how to fix it?
image
image

@bjosve
Copy link

bjosve commented Mar 11, 2020

I had the same error. When I replaced the config files with the old 2.0.0.2 config files everything worked again. Maybe something is broken with the new "KMScheduleLeaderElectionFeature"?!

@j-mie
Copy link

j-mie commented Mar 11, 2020

I can confirm removing KMScheduleLeaderElectionFeature from application.features did not fix this for us on CMAK 3.0.0.4 (from latest master) and Kafka 2.4.0

Getting this in the logs:

Mar 11 13:55:44 ngkafka-1 cmak[1194]: 2020-03-11 13:55:44,760 - [INFO] k.m.a.KafkaManagerActor - Updating internal state...
Mar 11 13:55:54 ngkafka-1 cmak[1194]: 2020-03-11 13:55:54,759 - [INFO] k.m.a.KafkaManagerActor - Updating internal state...
Mar 11 13:55:57 ngkafka-1 cmak[1194]: 2020-03-11 13:55:57,157 - [INFO] o.a.z.ClientCnxn - Unable to read additional data from server sessionid 0x30000a360a60001, likely server has closed socket, closing socket connection and attempting reconnect
Mar 11 13:55:57 ngkafka-1 cmak[1194]: 2020-03-11 13:55:57,169 - [ERROR] k.m.KafkaManager - Failed on input : KMAddCluster(ClusterConfig(vantage,CuratorConfig(ngkafka-1:2181,ngkafka-2:2181,ngkafka-3:2181,ngkafka-4:2181,ngkafka-5:2181,10,100,1000),true,2.4.0,false,None,None,false,false,false,false,false,false,Some(ClusterTuning(Some(30),Some(2),Some(100),Some(2),Some(100),Some(2),Some(100),Some(30),Some(5),Some(2),Some(1000),Some(2),Some(1000),Some(2),Some(1000),Some(30000),Some(1000000),Some(7))),PLAINTEXT,None,None))
Mar 11 13:55:57 ngkafka-1 cmak[1194]: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /kafka-manager/mutex
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.zookeeper.KeeperException.create(KeeperException.java:106)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:291)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:746)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
Mar 11 13:55:57 ngkafka-1 cmak[1194]: 2020-03-11 13:55:57,178 - [ERROR] k.m.ApiError$ - error : KeeperErrorCode = Unimplemented for /kafka-manager/mutex
Mar 11 13:55:57 ngkafka-1 cmak[1194]: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /kafka-manager/mutex
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.zookeeper.KeeperException.create(KeeperException.java:106)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:291)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:746)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
Mar 11 13:55:57 ngkafka-1 cmak[1194]:         at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
Mar 11 13:55:58 ngkafka-1 cmak[1194]: 2020-03-11 13:55:58,066 - [INFO] o.a.z.ClientCnxn - Opening socket connection to server ngkafka-2/10.20.4.90:2181. Will not attempt to authenticate using SASL (unknown error)
Mar 11 13:55:58 ngkafka-1 cmak[1194]: 2020-03-11 13:55:58,067 - [INFO] o.a.z.ClientCnxn - Socket connection established, initiating session, client: /10.20.4.89:51838, server: ngkafka-2/10.20.4.90:2181
Mar 11 13:55:58 ngkafka-1 cmak[1194]: 2020-03-11 13:55:58,070 - [INFO] o.a.z.ClientCnxn - Session establishment complete on server ngkafka-2/10.20.4.90:2181, sessionid = 0x30000a360a60001, negotiated timeout = 40000
Mar 11 13:55:58 ngkafka-1 cmak[1194]: 2020-03-11 13:55:58,072 - [INFO] k.m.a.KafkaManagerActor - Updating internal state...
Mar 11 13:55:58 ngkafka-1 cmak[1194]: 2020-03-11 13:55:58,073 - [INFO] k.m.a.KafkaManagerActor - Pruning clusters...
Mar 11 13:56:04 ngkafka-1 cmak[1194]: 2020-03-11 13:56:04,761 - [INFO] k.m.a.KafkaManagerActor - Updating internal state...
Mar 11 13:56:14 ngkafka-1 cmak[1194]: 2020-03-11 13:56:14,759 - [INFO] k.m.a.KafkaManagerActor - Updating internal state...

In our Zookeeper logs I'm seeing:

2020-03-11 13:59:54,825 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@756] - Received packet at server of unknown type 19
2020-03-11 13:59:54,826 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1056] - Closed socket connection for client /10.20.4.89:48398 which had sessionid 0x40000a3500a0001

@bjosve
Copy link

bjosve commented Mar 11, 2020

I can confirm removing KMScheduleLeaderElectionFeature from application.features did not fix this for us on CMAK 3.0.0.4 (from latest master) and Kafka 2.4.0

We are running CMAK 3.0.0.4 (docker image hlebalbau/kafka-manager) and Kafka 2.0.1 (docker image wurstmeister/kafka). I replaced both "application.conf" and "routes" with the old config and no more errors.

@j-mie
Copy link

j-mie commented Mar 11, 2020

Updating our Zookeeper version from 3.4.14 to 3.5.7 fixed this for us

@priyanka460
Copy link

@jamie-34254 how to update zookeeper version or how to replace config file with old one?

@priyanka460
Copy link

I had the same error. When I replaced the config files with the old 2.0.0.2 config files everything worked again. Maybe something is broken with the new "KMScheduleLeaderElectionFeature"?!

how to do that?

@priyanka460
Copy link

I can confirm removing KMScheduleLeaderElectionFeature from application.features did not fix this for us on CMAK 3.0.0.4 (from latest master) and Kafka 2.4.0

We are running CMAK 3.0.0.4 (docker image hlebalbau/kafka-manager) and Kafka 2.0.1 (docker image wurstmeister/kafka). I replaced both "application.conf" and "routes" with the old config and no more errors.

from where you found old config file

@gl1ridae
Copy link

I have the same issue, and seems like it's related with unsupported zookeeper version. Since latest CMAK use Apache Curator 2.12.0 and it supported zookeeper 3.5.x version by default, and 3.4.x only with modifications in build.

It will be nice to add Zookeeper requirements in README.

@godwhoa
Copy link

godwhoa commented Apr 12, 2020

Patch for anyone who wants it

diff --git a/build.sbt b/build.sbt
index 9ac7221..27528fc 100644
--- a/build.sbt
+++ b/build.sbt
@@ -33,21 +33,22 @@ libraryDependencies ++= Seq(
   "org.webjars" % "underscorejs" % "1.9.0",
   "org.webjars" % "dustjs-linkedin" % "2.7.2",
   "org.webjars" % "octicons" % "4.3.0",
-  "org.apache.curator" % "curator-framework" % "2.12.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.curator" % "curator-framework" % "2.12.0" exclude("org.apache.zookeeper","zookeeper") exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
   "org.apache.curator" % "curator-recipes" % "2.12.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.zookeeper" % "zookeeper" % "3.4.14",
   "org.json4s" %% "json4s-jackson" % "3.6.5",
   "org.json4s" %% "json4s-scalaz" % "3.6.5",
   "org.slf4j" % "log4j-over-slf4j" % "1.7.25",
   "com.adrianhurt" %% "play-bootstrap" % "1.4-P26-B4" exclude("com.typesafe.play", "*"),
   "org.clapper" %% "grizzled-slf4j" % "1.3.3",
-  "org.apache.kafka" %% "kafka" % "2.4.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.kafka" %% "kafka" % "2.4.0" exclude("org.apache.zookeeper","zookeeper") exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
   "org.apache.kafka" % "kafka-streams" % "2.2.0",
   "com.beachape" %% "enumeratum" % "1.5.13",
   "com.github.ben-manes.caffeine" % "caffeine" % "2.6.2",
   "com.typesafe.play" %% "play-logback" % "2.6.21",
   "org.scalatest" %% "scalatest" % "3.0.5" % "test",
   "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.2" % "test",
-  "org.apache.curator" % "curator-test" % "2.12.0" % "test",
+  "org.apache.curator" % "curator-test" % "2.12.0" % "test" exclude("org.apache.zookeeper","zookeeper") force(),
   "org.mockito" % "mockito-core" % "1.10.19" % "test",
   "com.yammer.metrics" % "metrics-core" % "2.2.0" force(),
   "com.unboundid" % "unboundid-ldapsdk" % "4.0.9"

@rohitranjanef
Copy link

I have the same problem. I have installed zookeeper and kafka with docker images.
with zookeeper v3.5.8 I get KeeperErrorCode = Unimplemented for /kafka-manager/mutex when I try to create a cluster.
Any help would be appreciated.

@zhigang
Copy link

zhigang commented Jun 15, 2020

create /kafka-manager/mutex/locks and /kafka-manager/mutex/leases to fixed this, like this:
#731 (comment)

@TangZhongham
Copy link

Patch for anyone who wants it

diff --git a/build.sbt b/build.sbt
index 9ac7221..27528fc 100644
--- a/build.sbt
+++ b/build.sbt
@@ -33,21 +33,22 @@ libraryDependencies ++= Seq(
   "org.webjars" % "underscorejs" % "1.9.0",
   "org.webjars" % "dustjs-linkedin" % "2.7.2",
   "org.webjars" % "octicons" % "4.3.0",
-  "org.apache.curator" % "curator-framework" % "2.12.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.curator" % "curator-framework" % "2.12.0" exclude("org.apache.zookeeper","zookeeper") exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
   "org.apache.curator" % "curator-recipes" % "2.12.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.zookeeper" % "zookeeper" % "3.4.14",
   "org.json4s" %% "json4s-jackson" % "3.6.5",
   "org.json4s" %% "json4s-scalaz" % "3.6.5",
   "org.slf4j" % "log4j-over-slf4j" % "1.7.25",
   "com.adrianhurt" %% "play-bootstrap" % "1.4-P26-B4" exclude("com.typesafe.play", "*"),
   "org.clapper" %% "grizzled-slf4j" % "1.3.3",
-  "org.apache.kafka" %% "kafka" % "2.4.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.kafka" %% "kafka" % "2.4.0" exclude("org.apache.zookeeper","zookeeper") exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
   "org.apache.kafka" % "kafka-streams" % "2.2.0",
   "com.beachape" %% "enumeratum" % "1.5.13",
   "com.github.ben-manes.caffeine" % "caffeine" % "2.6.2",
   "com.typesafe.play" %% "play-logback" % "2.6.21",
   "org.scalatest" %% "scalatest" % "3.0.5" % "test",
   "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.2" % "test",
-  "org.apache.curator" % "curator-test" % "2.12.0" % "test",
+  "org.apache.curator" % "curator-test" % "2.12.0" % "test" exclude("org.apache.zookeeper","zookeeper") force(),
   "org.mockito" % "mockito-core" % "1.10.19" % "test",
   "com.yammer.metrics" % "metrics-core" % "2.2.0" force(),
   "com.unboundid" % "unboundid-ldapsdk" % "4.0.9"

it works! thanks a lot

@wyisterl
Copy link

Patch for anyone who wants it

diff --git a/build.sbt b/build.sbt
index 9ac7221..27528fc 100644
--- a/build.sbt
+++ b/build.sbt
@@ -33,21 +33,22 @@ libraryDependencies ++= Seq(
   "org.webjars" % "underscorejs" % "1.9.0",
   "org.webjars" % "dustjs-linkedin" % "2.7.2",
   "org.webjars" % "octicons" % "4.3.0",
-  "org.apache.curator" % "curator-framework" % "2.12.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.curator" % "curator-framework" % "2.12.0" exclude("org.apache.zookeeper","zookeeper") exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
   "org.apache.curator" % "curator-recipes" % "2.12.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.zookeeper" % "zookeeper" % "3.4.14",
   "org.json4s" %% "json4s-jackson" % "3.6.5",
   "org.json4s" %% "json4s-scalaz" % "3.6.5",
   "org.slf4j" % "log4j-over-slf4j" % "1.7.25",
   "com.adrianhurt" %% "play-bootstrap" % "1.4-P26-B4" exclude("com.typesafe.play", "*"),
   "org.clapper" %% "grizzled-slf4j" % "1.3.3",
-  "org.apache.kafka" %% "kafka" % "2.4.0" exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
+  "org.apache.kafka" %% "kafka" % "2.4.0" exclude("org.apache.zookeeper","zookeeper") exclude("log4j","log4j") exclude("org.slf4j", "slf4j-log4j12") force(),
   "org.apache.kafka" % "kafka-streams" % "2.2.0",
   "com.beachape" %% "enumeratum" % "1.5.13",
   "com.github.ben-manes.caffeine" % "caffeine" % "2.6.2",
   "com.typesafe.play" %% "play-logback" % "2.6.21",
   "org.scalatest" %% "scalatest" % "3.0.5" % "test",
   "org.scalatestplus.play" %% "scalatestplus-play" % "3.1.2" % "test",
-  "org.apache.curator" % "curator-test" % "2.12.0" % "test",
+  "org.apache.curator" % "curator-test" % "2.12.0" % "test" exclude("org.apache.zookeeper","zookeeper") force(),
   "org.mockito" % "mockito-core" % "1.10.19" % "test",
   "com.yammer.metrics" % "metrics-core" % "2.2.0" force(),
   "com.unboundid" % "unboundid-ldapsdk" % "4.0.9"

It works, but maybe need some attention to "org.apache.zookeeper" % "zookeeper" % "3.4.14",,3.4.14 refering your zk version maybe? I modify my zk version 3.4.10 instead, and recompile it. Thanks a lot!

@Zhangwusheng
Copy link

create /kafka-manager/mutex/locks and /kafka-manager/mutex/leases to fixed this, like this:
#731 (comment)

works for me . thank you

@jaypalsodha
Copy link

org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /debug

I am facing the same issue. Above resolutions shows that we are forcing zookeeper version 3.4.14 to be used. But my requirement is to use zookeeper version 3.5.7 with curator-framework version 2.12.0 or latest.

Any help would be appreciated.

ERROR org.apache.curator.framework.imps.CuratorFrameworkImpl - Ensure path threw exception org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /debug at org.apache.zookeeper.KeeperException.create(KeeperException.java:106) [CCB-globalCronExecutor-3-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x17fb5659c8f0056, likely server has closed socket, closing socket connection and attempting reconnect at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538) at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:291) at org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109) at org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83) at org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:594) at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:158) at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32) at com.santaba.server.util.cloud.Register2CloudZKTask._CreateDiagPathInZookeeper(Register2CloudZKTask.java:160) at com.santaba.server.util.cloud.Register2CloudZKTask._run(Register2CloudZKTask.java:177) at com.santaba.server.cron.Cron.run(Cron.java:72) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) [CCB-globalCronExecutor-3-EventThread] INFO org.apache.curator.framework.state.ConnectionStateManager - State change: SUSPENDED [2022-03-23 06:44:02.013 UTC] [MSG] [WARN] [CCB-globalCronExecutor-3:santaba:cron:] [Register2CloudZKTask._CreateDiagPathInZookeeper:163] something wrong when create the path of diagnosis, CONTEXT=connectString=zookeeper:2181, rootPath=debug, errMsg=KeeperErrorCode = ConnectionLoss for /debug

@wyisterl
Copy link

wyisterl commented Mar 23, 2022 via email

@papillon88
Copy link

"org.apache.zookeeper" % "zookeeper" % "3.4.14"

I removed that line and it still worked

@wyisterl
Copy link

wyisterl commented Jun 2, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests