ClassNotFoundException: io.netty.channel.epoll.NativeStaticallyReferencedJniMethods

[sautran]

I am using the kubernete deployment and go through the installation step.
After deployment, I am seeing the following error in api-server as well as the worker pod

Jan 24, 2019 6:27:39 PM org.apache.distributedlog.impl.BKNamespaceDriver getDefaultEventLoopGroup
WARNING: Could not use Netty Epoll event loop for bookie server:
java.lang.UnsatisfiedLinkError: failed to load the required native library
at dlshade.io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:78)
at dlshade.io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:38)
at org.apache.distributedlog.impl.BKNamespaceDriver.getDefaultEventLoopGroup(BKNamespaceDriver.java:259)
at org.apache.distributedlog.impl.BKNamespaceDriver.initializeBookKeeperClients(BKNamespaceDriver.java:270)
at org.apache.distributedlog.impl.BKNamespaceDriver.initialize(BKNamespaceDriver.java:208)
at org.apache.distributedlog.api.namespace.NamespaceBuilder.build(NamespaceBuilder.java:238)
at com.twitter.heron.downloader.DLDownloader.download(DLDownloader.java:66)
at com.twitter.heron.downloader.DownloadRunner.main(DownloadRunner.java:42)
Caused by: java.lang.NoClassDefFoundError: io/netty/channel/epoll/NativeStaticallyReferencedJniMethods
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at dlshade.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
at dlshade.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:278)
at dlshade.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:231)
at dlshade.io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:191)
at dlshade.io.netty.channel.epoll.Native.(Native.java:61)
at dlshade.io.netty.channel.epoll.Epoll.(Epoll.java:33)
... 7 more
Caused by: java.lang.ClassNotFoundException: io.netty.channel.epoll.NativeStaticallyReferencedJniMethods
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 18 more

Jan 24, 2019 6:27:39 PM org.apache.distributedlog.impl.BKNamespaceDriver createBKCBuilder

[sautran]

i am running minikube on mac

[sautran]

Look like heron packaged with old distributedlog core jar (0.5). in this release, it package io.netty with dlshade which will cause issue when loading native library.

[sautran]

Here is the log in the api server
[2019-01-25 18:30:18 +0000] [INFO] org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase: ZooKeeper client is connected now.
[2019-01-25 18:30:18 +0000] [WARNING] org.apache.distributedlog.impl.BKNamespaceDriver: Could not use Netty Epoll event loop for bookie server:
java.lang.UnsatisfiedLinkError: failed to load the required native library
at dlshade.io.netty.channel.epoll.Epoll.ensureAvailability(Epoll.java:78)
at dlshade.io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:38)
at org.apache.distributedlog.impl.BKNamespaceDriver.getDefaultEventLoopGroup(BKNamespaceDriver.java:259)
at org.apache.distributedlog.impl.BKNamespaceDriver.initializeBookKeeperClients(BKNamespaceDriver.java:270)
at org.apache.distributedlog.impl.BKNamespaceDriver.initialize(BKNamespaceDriver.java:208)
at org.apache.distributedlog.api.namespace.NamespaceBuilder.build(NamespaceBuilder.java:238)
at com.twitter.heron.uploader.dlog.DLUploader.initializeNamespace(DLUploader.java:137)
at com.twitter.heron.uploader.dlog.DLUploader.initialize(DLUploader.java:94)
at com.twitter.heron.scheduler.SubmitterMain.submitTopology(SubmitterMain.java:420)
at com.twitter.heron.apiserver.actions.SubmitTopologyAction.execute(SubmitTopologyAction.java:33)
at com.twitter.heron.apiserver.resources.TopologyResource.submit(TopologyResource.java:223)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: io/netty/channel/epoll/NativeStaticallyReferencedJniMethods
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at dlshade.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
at dlshade.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:278)
at dlshade.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:231)
at dlshade.io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:191)
at dlshade.io.netty.channel.epoll.Native.(Native.java:61)
at dlshade.io.netty.channel.epoll.Epoll.(Epoll.java:33)
... 59 more
Caused by: java.lang.ClassNotFoundException: io.netty.channel.epoll.NativeStaticallyReferencedJniMethods
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 70 more

[sautran]

adding -D io.netty.packagePrefix=dlshade. didn't solve the problem
heron-apiserver
--base-template kubernetes
--cluster kubernetes
-D io.netty.packagePrefix=dlshade.
-D heron.statemgr.connection.string=zookeeper:2181