Shading issues (BouncyCastle, Jetty)

[wpoch]

When using the provided distribution I'm facing BouncyCastle and Jetty shading issues with my current application code; I cloned the repo removed the shading and everything worked as expected.

Would you consider releasing a version without shading the dependencies? I can work on the PR with a custom profile.

Congrats for the great lib!

[asafm]

I have a dependency on Jetty - might collide. What problems did you have?

…

On Tue, 9 May 2017 at 5:07 Walter Poch ***@***.***> wrote: When using the provided distribution I'm facing BouncyCastle and Jetty shading issues with my current application code; I cloned the repo removed the shading and everything worked as expected. Would you consider releasing a version without shading the dependencies? I can work on the PR with a custom profile. Congrats for the great lib! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#338>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA8Y8WrFb1PO9GeEtSXpjrNZK6DLFxMWks5r38pZgaJpZM4NUuFT> .

[rnorth]

Hi
Sorry you've had problems with the shaded JARs - it's quite a pain that we have to do it at all! I'd be happy to release an unshaded version, yes. We'd probably still want to guide people towards the shaded version normally, but for people who need to use an unshaded version it would make a nice option.

Ideally I'd hope it's something we can define at the parent POM level to avoid having to update every POM - is that what you'd have in mind?
Thanks!
Richard

[terribleherbst]

I've got a similar problem. When integrating the testcontainer dependency I get an jackson error java.util.ServiceConfigurationError: com.fasterxml.jackson.databind.Module: Provider org.testcontainers.shaded.com.fasterxml.jackson.module.jaxb.JaxbAnnotationModule not a subtype. This is caused by a customized Jackson configuration which autoconfigures modules at startup.
Is there a unshaded version of testcontainers available?

[rmchale]

I get this issue as well

java.util.ServiceConfigurationError: com.fasterxml.jackson.databind.Module: Provider org.testcontainers.shaded.com.fasterxml.jackson.module.jaxb.JaxbAnnotationModule not a subtype

[banderous]

BouncyCastle is a signed JAR which is being shaded by TestContainers, ie. repackaged into an unsigned JAR. The signature of this jar is verified (and fails to verify) when BouncyCastle types are used by java.security.

A particular problem is that a shaded BouncyCastleProvider is registered with java.security.Security by Netty, which if it happens first will prevent a signed BouncyCastleProvider being registered.

[bsideup]

Hi @banderous,

Thanks for explanation, it helps to understand the issue better 👍

FYI we have plans to replace Netty with OkHttp or some other alternative lightweight networking library once new docker-java is released, I'll post the update on this issue once it's done

[banderous]

Hi Sergei, Thanks for responding. In this case the issue isn't the use of Netty, it's repackaging BouncyCastle into an unsigned JAR; even if you replace Netty lots of other crytpo related functionality is likely to break in unpredictable ways with this approach. I suggest removing BouncyCastle from shading entirely, would you accept a PR if I put one together? Regards Alex

…

On Wed, Mar 28, 2018 at 9:59 AM, Sergei Egorov ***@***.***> wrote: Hi @banderous <https://github.com/banderous>, Thanks for explanation, it helps to understand the issue better 👍 FYI we have plans to replace Netty with OkHttp or some other alternative lightweight networking library once new docker-java is released, I'll post the update on this issue once it's done — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#338 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACZYJz6ccx17fiN_LuMhQD3wOH-v8h-pks5ti1EAgaJpZM4NUuFT> .

[bsideup]

@banderous
BouncyCastle is a transitive dependency of docker-java -> Netty, I don't think we use it anywhere else.

I would keep shaded because we don't want to leak it into our public API / dependencies, and the proper fix will be to remove it completely

[banderous]

I can see why you'd want to shade it but Netty itself may be broken in unpredictable ways by this approach, plus any code under test that makes use of BouncyCastle; it's part of the contract for BouncyCastle that it be in a signed jar. FYI this is likely to be a root cause of many other non-obvious and hard to reproduce bugs. For example, our tests sometimes worked and sometimes broke depending on the order they ran in, which was affecting whether a good or bad BouncyCastle was initialised first. We thought the root cause was a bug in our SSH client since that's where the error would sometimes come from; it took a lot of debugging to figure out the root cause. If you can remove the dependency on BouncyCastle that'd be great!

…

On Wed, Mar 28, 2018 at 11:18 AM, Sergei Egorov ***@***.***> wrote: @banderous <https://github.com/banderous> BouncyCastle is a transitive dependency of docker-java -> Netty, I don't think we use it anywhere else. I would keep shaded because we don't want to leak it into our public API / dependencies, and the proper fix will be to remove it completely — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#338 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACZYJ_IdG7wUTgoAsZa4-NwOjKn2GLKwks5ti2N0gaJpZM4NUuFT> .

[rnorth]

With removal of Netty in #1307, we're no longer shading BouncyCastle. It is still a transitive dependency via docker-java:

+--- com.github.docker-java:docker-java:3.1.0-rc-4
|    +--- com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:2.6.7
|    |    +--- com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:2.6.7
|    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.6.7
|    |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.6.7
|    |    |         +--- com.fasterxml.jackson.core:jackson-annotations:2.6.0
|    |    |         \--- com.fasterxml.jackson.core:jackson-core:2.6.7
|    |    +--- com.fasterxml.jackson.core:jackson-core:2.6.7
|    |    +--- com.fasterxml.jackson.core:jackson-databind:2.6.7 (*)
|    |    \--- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.6.7
|    |         +--- com.fasterxml.jackson.core:jackson-core:2.6.7
|    |         \--- com.fasterxml.jackson.core:jackson-databind:2.6.7 (*)
|    +--- commons-codec:commons-codec:1.11
|    +--- commons-lang:commons-lang:2.6
|    +--- commons-io:commons-io:2.6
|    +--- com.google.guava:guava:19.0
|    \--- org.bouncycastle:bcpkix-jdk15on:1.60
|         \--- org.bouncycastle:bcprov-jdk15on:1.60

As this issue was specifically about the shading, I'll close. If there are any issues just relating to the (unshaded) BC library being a dependency, let's open a new ticket.

[crusi]

With removal of Netty in #1307, we're no longer shading BouncyCastle. It is still a transitive dependency via docker-java:

+--- com.github.docker-java:docker-java:3.1.0-rc-4
|    +--- com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:2.6.7
|    |    +--- com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:2.6.7
|    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.6.7
|    |    |    \--- com.fasterxml.jackson.core:jackson-databind:2.6.7
|    |    |         +--- com.fasterxml.jackson.core:jackson-annotations:2.6.0
|    |    |         \--- com.fasterxml.jackson.core:jackson-core:2.6.7
|    |    +--- com.fasterxml.jackson.core:jackson-core:2.6.7
|    |    +--- com.fasterxml.jackson.core:jackson-databind:2.6.7 (*)
|    |    \--- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.6.7
|    |         +--- com.fasterxml.jackson.core:jackson-core:2.6.7
|    |         \--- com.fasterxml.jackson.core:jackson-databind:2.6.7 (*)
|    +--- commons-codec:commons-codec:1.11
|    +--- commons-lang:commons-lang:2.6
|    +--- commons-io:commons-io:2.6
|    +--- com.google.guava:guava:19.0
|    \--- org.bouncycastle:bcpkix-jdk15on:1.60
|         \--- org.bouncycastle:bcprov-jdk15on:1.60

As this issue was specifically about the shading, I'll close. If there are any issues just relating to the (unshaded) BC library being a dependency, let's open a new ticket.

there are still bouncycastle classes in the testcontainers.jar. And they still lead to errors like
testcontainers-1.12.5.jar has unsigned entries - org/testcontainers/lifecycle/Startable.class

As long as you have org.testcontainers.shaded.org.bouncycastle.jce.provider.BouncyCastleProvider in your jar, and your jar is not signed, the error will happen in other projects and the only workaround currently is to remove this provider before the test executes and add the correct one first, e.g. here