Shading of Netty breaks on Linux because of native bindings #178

raphw · 2016-07-18T10:42:14Z

The shading does not work for Linux due to native libraries for custom epoll transport not being shaded.

The problem is being described here: netty/netty#4820

The following stack-trace shows the issue:

java.lang.NoClassDefFoundError: io/netty/channel/epoll/NativeStaticallyReferencedJniMethods

    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at org.testcontainers.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:193)
    at org.testcontainers.shaded.io.netty.channel.epoll.Native.<clinit>(Native.java:60)
    at org.testcontainers.shaded.io.netty.channel.epoll.IovArray.<clinit>(IovArray.java:57)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:57)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:78)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:30)
    at org.testcontainers.shaded.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:77)
    at org.testcontainers.shaded.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:48)
    at org.testcontainers.shaded.io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:57)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:63)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:51)
    at org.testcontainers.shaded.com.github.dockerjava.netty.DockerCmdExecFactoryImpl$UnixDomainSocketInitializer.init(DockerCmdExecFactoryImpl.java:221)
    at org.testcontainers.shaded.com.github.dockerjava.netty.DockerCmdExecFactoryImpl.init(DockerCmdExecFactoryImpl.java:197)
    at org.testcontainers.shaded.com.github.dockerjava.core.DockerClientImpl.withDockerCmdExecFactory(DockerClientImpl.java:159)
    at org.testcontainers.shaded.com.github.dockerjava.core.DockerClientBuilder.build(DockerClientBuilder.java:45)
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.getClientForConfig(DockerClientProviderStrategy.java:103)
    at org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy.test(EnvironmentAndSystemPropertyClientProviderStrategy.java:18)
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.getFirstValidStrategy(DockerClientProviderStrategy.java:54)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:82)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:70)
    at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:105)
    at org.testcontainers.containers.FixedHostPortGenericContainer.<init>(FixedHostPortGenericContainer.java:16)

The text was updated successfully, but these errors were encountered:

dbalakirev · 2016-07-18T10:46:26Z

Same for me. Latest code from today.

rnorth · 2016-07-18T13:29:57Z

Ouch - sorry. I'll check the PR and will aim to release tonight.

Could you perhaps let me know what configuration you're running under, e.g. OS, Docker installation method etc? Clearly I should have caught this before, and need to add a different permutation to pre-release testing (on top of Mac and two different Linux CI services 😟 )!

raphw · 2016-07-18T13:40:54Z

I run a recent version of Ubuntu:

Linux 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I assume that most infrastructure is on virtual servers. Mac does not have Unix sockets so I guess the outcome is different. I tested a custom build of Testcontainers with my project and the Netty problems and Guava problems I have had went away.

rnorth · 2016-07-19T21:48:14Z

Just released 1.1.2 - will be rolling out to Maven Central soon.

valdisrigdon · 2016-07-20T13:52:00Z

Testing v1.1.2 and I'm seeing this now on an AWS machine:

Caused by: java.lang.UnsatisfiedLinkError: no netty-transport-native-epoll in java.library.path
    at org.testcontainers.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:168)
    at org.testcontainers.shaded.io.netty.channel.epoll.Native.<clinit>(Native.java:60)
    at org.testcontainers.shaded.io.netty.channel.epoll.IovArray.<clinit>(IovArray.java:57)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:57)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:78)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:30)
    at org.testcontainers.shaded.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:77)
    at org.testcontainers.shaded.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:48)
    at org.testcontainers.shaded.io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:57)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:63)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:51)
    at org.testcontainers.shaded.com.github.dockerjava.netty.DockerCmdExecFactoryImpl$UnixDomainSocketInitializer.init(DockerCmdExecFactoryImpl.java:221)
    at org.testcontainers.shaded.com.github.dockerjava.netty.DockerCmdExecFactoryImpl.init(DockerCmdExecFactoryImpl.java:197)
    at org.testcontainers.shaded.com.github.dockerjava.core.DockerClientImpl.withDockerCmdExecFactory(DockerClientImpl.java:159)
    at org.testcontainers.shaded.com.github.dockerjava.core.DockerClientBuilder.build(DockerClientBuilder.java:45)
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.getClientForConfig(DockerClientProviderStrategy.java:103)
    at org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy.test(EnvironmentAndSystemPropertyClientProviderStrategy.java:18)
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.getFirstValidStrategy(DockerClientProviderStrategy.java:54)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:82)

rnorth · 2016-07-20T19:10:22Z

OK, sorry about this. Yet more digging to do! ☹️

Richard

raphw · 2016-07-20T19:39:14Z

It appears as if the system property is not applied in the above code. Maybe, the library is loaded before the property initializer is applied? I wonder if Amazon executes initializers lazily. Can you try setting:

-Dorg.testcontainers.shaded.io.netty.packagePrefix=org.testcontainers.shaded.

on startup?

rnorth · 2016-07-20T19:51:17Z

That makes sense - I'm curious to see this. I'm trying to think what could
cause this difference in behaviour. @valdisrigdon I can't recall if you
said you were on Oracle/Open JDK...

If it emerges that this is the cause we at least have the option of calling
the necessary setProperty in Testcontainers itself. That would be worst
case, but we're not without options.

@raphw thanks for your continuing involvement in this issue!
On Wed, 20 Jul 2016 at 20:39 Rafael Winterhalter notifications@github.com
wrote:

It appears as if the system property is not applied in the above code.
Maybe, the library is loaded before the property initializer is applied? I
wonder if Amazon executes initializers lazily. Can you try setting:

-Dorg.testcontainers.shaded.io.netty.packagePrefix=org.testcontainers.shaded.

on startup?

—
You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub
#178 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAIET5DJGr8NhoqMIgZALovI3bUGn2TIks5qXnligaJpZM4JOl9j
.

Richard

dbalakirev · 2016-07-22T11:02:24Z

1.1.2 works for me as well. Many thanks!

irmac · 2016-07-26T14:08:45Z

I'm also seeing the same error as @valdisrigdon running on my laptop with testcontainers
1.1.2 and

Linux version 4.3.5-300.fc23.x86_64 (mockbuild@bkernel02.phx2.fedoraproject.org) (gcc version 5.3.1 20151207 (Red Hat 5.3.1-2) (GCC) ) #1 SMP Mon Feb 1 03:18:41 UTC 2016

openjdk version "1.8.0_92"

OpenJDK Runtime Environment (build 1.8.0_92-b14)

OpenJDK 64-Bit Server VM (build 25.92-b14, mixed mode)

java.lang.UnsatisfiedLinkError\: no netty-transport-native-epoll in java.library.path

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1864)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)
    at org.testcontainers.shaded.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:168)
    at org.testcontainers.shaded.io.netty.channel.epoll.Native.<clinit>(Native.java:60)
    at org.testcontainers.shaded.io.netty.channel.epoll.IovArray.<clinit>(IovArray.java:57)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:57)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:78)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:30)
    at org.testcontainers.shaded.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:77)
    at org.testcontainers.shaded.io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:48)
    at org.testcontainers.shaded.io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:57)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:63)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:51)
    at org.testcontainers.shaded.com.github.dockerjava.netty.DockerCmdExecFactoryImpl$UnixDomainSocketInitializer.init(DockerCmdExecFactoryImpl.java:221)
    at org.testcontainers.shaded.com.github.dockerjava.netty.DockerCmdExecFactoryImpl.init(DockerCmdExecFactoryImpl.java:197)
    at org.testcontainers.shaded.com.github.dockerjava.core.DockerClientImpl.withDockerCmdExecFactory(DockerClientImpl.java:159)
    at org.testcontainers.shaded.com.github.dockerjava.core.DockerClientBuilder.build(DockerClientBuilder.java:45)
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.getClientForConfig(DockerClientProviderStrategy.java:103)
    at org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy.test(EnvironmentAndSystemPropertyClientProviderStrategy.java:18)
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.getFirstValidStrategy(DockerClientProviderStrategy.java:54)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:82)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:70)
    at org.testcontainers.containers.DockerComposeContainer.<init>(DockerComposeContainer.java:72)
    at org.testcontainers.containers.DockerComposeContainer.<init>(DockerComposeContainer.java:62)
    at MetadataRepositoryIT.<clinit>(MetadataRepositoryIT.java:12)
    at sun.misc.Unsafe.ensureClassInitialized(Native Method)
    at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
    at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:142)
    at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
    at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
    at java.lang.reflect.Field.get(Field.java:393)
    at org.junit.runners.model.FrameworkField.get(FrameworkField.java:73)
    at org.junit.runners.model.TestClass.getAnnotatedFieldValues(TestClass.java:230)
    at org.junit.runners.ParentRunner.classRules(ParentRunner.java:255)
    at org.junit.runners.ParentRunner.withClassRules(ParentRunner.java:244)
    at org.junit.runners.ParentRunner.classBlock(ParentRunner.java:194)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:362)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

`

raphw · 2016-07-26T14:18:03Z

@irmac How do you trigger DockerComposeContainer without loading GenericContainer, i.e. how does your MetadataRepositoryIT look like? To me, it seems like there are muliple entry points. The problem is that Testcontainers.org must set this system property before the library is loaded but from the both reports, it seems like there are multiple ways of doing so.

I assume you have a field DockerComposeContainer in your class. Can you change its type to GenericContainer and try if this helps? Alternatively, it might help to trigger the field from a @BeforeClass method rather than statically.

valdisrigdon · 2016-07-26T14:24:13Z

I'm triggering a call to org.testcontainers.DockerClientFactory.instance().client(true). If I explicitly set the System property before this, it works as expected.

rnorth · 2016-07-26T14:24:20Z

Ah, this makes sense if going via DockerComposeContainer. This stopped subclassing GenericContainer in the 1.1.0 release, so it seems probable that the route to obtaining a DockerClient instance is not initialising the GenericContainer class.

I think the System.setProperty(...) static call should probably be transplanted over to DockerClientFactory, which is the common element in the path to obtaining the client instance.

irmac · 2016-07-26T14:25:46Z

Here's the test class...

import org.junit.ClassRule;
import org.junit.Test;
import org.testcontainers.containers.DockerComposeContainer;

import java.io.File;

public class MetadataRepositoryIT {

    public static final Integer MONGO_PORT = 27017;

    @ClassRule
    public static DockerComposeContainer environment =
            new DockerComposeContainer(new File("src/test/resources/docker-compose-test.yml"))
            .withExposedService("mongodb_1", MONGO_PORT);

    private final String mongoURL = environment.getServiceHost("mongodb_1",MONGO_PORT)
            + ":" +
            environment.getServicePort("mongodb_1", MONGO_PORT);

    @Test
    public void testDockerComposeTestContainersIT() throws Exception {
        System.out.println(mongoURL);
    }
}

irmac · 2016-07-26T15:01:03Z

@raphw @rnorth Thanks for the help chaps, I understand the problem now. I've got a local workaround in place following the suggestion of @rnorth.

Unless someone else is working on this (in which case, great!) I'd like to submit a pull request for this issue. Is there consensus on whether the property setting logic should be duplicated or extracted somewhere shared ( perhaps a default method ?)

rnorth · 2016-07-26T15:20:10Z

@irmac thanks, that'd be great. Logically I'd have thought moving the static initialiser to DockerClientFactory should be safe for anything relating to shaded netty. It might be a good idea to test and poke holes in this theory, though, just in case there's another entry point.

If that fails, a static initialiser on an interface that both GenericContainer and DockerComposeContainer implement would probably be the next valid way to do it.

I haven't looked yet, but I'm assuming nobody knows of another mechanism for setting the prefix (e.g. ServiceLoader, anything else?)

raphw · 2016-07-26T17:59:01Z

@rnorth No, there is no way. I already looked into alternatives when doing the initial PR as I found it clumsy myself but this is the way to go. If there is a single entry point to accessing Netty; this is where the call goes.

rnorth · 2016-07-26T20:45:26Z

@irmac I'm working on some other PRs this evening so happy to do this while I'm on a roll - I hope this is OK!

irmac · 2016-07-26T20:49:17Z

@rnorth That's great - I'm afraid I've lost any development time this evening!

…ated package name is available for DockerCompose usage scenarios. Fixes #178

raphw mentioned this issue Jul 18, 2016

Properly shade Netty including its native adapters. Include Guava when shading dependencies. #180

Closed

This was referenced Jul 19, 2016

For docker-java v3+, netty should be a shaded dependency #157

Closed

Correct shading for netty native linux lib; add guava shading and fix bouncycastle certs #181

Merged

rnorth modified the milestone: 1.1.2 Jul 19, 2016

rnorth closed this as completed Jul 19, 2016

rnorth reopened this Jul 26, 2016

rnorth added a commit that referenced this issue Jul 26, 2016

Change location of system property setting to ensure that netty reloc…

860c1fb

…ated package name is available for DockerCompose usage scenarios. Fixes #178

rnorth added a commit that referenced this issue Jul 26, 2016

Change location of system property setting to ensure that netty reloc…

8462da2

…ated package name is available for DockerCompose usage scenarios. Fixes #178

rnorth mentioned this issue Jul 26, 2016

Change location of system property setting to ensure that netty reloc… #187

Merged

rnorth closed this as completed in #187 Jul 26, 2016

bsideup mentioned this issue Oct 21, 2019

Need an advice on how to shade ByteBuddy's DLLs raphw/byte-buddy#747

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shading of Netty breaks on Linux because of native bindings #178

Shading of Netty breaks on Linux because of native bindings #178

raphw commented Jul 18, 2016 •

edited

Loading

dbalakirev commented Jul 18, 2016

rnorth commented Jul 18, 2016

raphw commented Jul 18, 2016 •

edited

Loading

rnorth commented Jul 19, 2016

valdisrigdon commented Jul 20, 2016

rnorth commented Jul 20, 2016

raphw commented Jul 20, 2016

rnorth commented Jul 20, 2016

dbalakirev commented Jul 22, 2016

irmac commented Jul 26, 2016 •

edited

Loading

raphw commented Jul 26, 2016 •

edited

Loading

valdisrigdon commented Jul 26, 2016

rnorth commented Jul 26, 2016

irmac commented Jul 26, 2016

irmac commented Jul 26, 2016

rnorth commented Jul 26, 2016

raphw commented Jul 26, 2016

rnorth commented Jul 26, 2016

irmac commented Jul 26, 2016

Shading of Netty breaks on Linux because of native bindings #178

Shading of Netty breaks on Linux because of native bindings #178

Comments

raphw commented Jul 18, 2016 • edited Loading

dbalakirev commented Jul 18, 2016

rnorth commented Jul 18, 2016

raphw commented Jul 18, 2016 • edited Loading

rnorth commented Jul 19, 2016

valdisrigdon commented Jul 20, 2016

rnorth commented Jul 20, 2016

OK, sorry about this. Yet more digging to do! ☹️

raphw commented Jul 20, 2016

rnorth commented Jul 20, 2016

dbalakirev commented Jul 22, 2016

irmac commented Jul 26, 2016 • edited Loading

raphw commented Jul 26, 2016 • edited Loading

valdisrigdon commented Jul 26, 2016

rnorth commented Jul 26, 2016

irmac commented Jul 26, 2016

irmac commented Jul 26, 2016

rnorth commented Jul 26, 2016

raphw commented Jul 26, 2016

rnorth commented Jul 26, 2016

irmac commented Jul 26, 2016

raphw commented Jul 18, 2016 •

edited

Loading

raphw commented Jul 18, 2016 •

edited

Loading

irmac commented Jul 26, 2016 •

edited

Loading

raphw commented Jul 26, 2016 •

edited

Loading