[improve][misc] Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded

[lhotari]

Fixes #22601
Fixes #21892
Fixes #19460

Motivation

Netty 4.1.111.Final contains important fixes:

• Optimize wrap buffer cumulation in SslHandler and don't mutate input buffers  netty/netty#14086
• Duplicate of slice should have the same capacity as the original slice so that it's not writable netty/netty#14093

These address Broker stability when TLS with Bookkeeper V2 protocol is used between Broker and Bookies.
This will also allow removing the extra buffer copies that have been in place in Pulsar broker and Pulsar client when TLS is enabled.

pulsar/pulsar-common/src/main/java/org/apache/pulsar/common/protocol/ByteBufPair.java

Lines 138 to 157 in 266243c

	public static class CopyingEncoder extends ChannelOutboundHandlerAdapter {
	@Override
	public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception {
	if (msg instanceof ByteBufPair) {
	ByteBufPair b = (ByteBufPair) msg;
	// Some handlers in the pipeline will modify the bytebufs passed in to them (i.e. SslHandler).
	// For these handlers, we need to pass a copy of the buffers as the source buffers may be cached
	// for multiple requests.
	try {
	ctx.write(b.getFirst().copy(), ctx.voidPromise());
	ctx.write(b.getSecond().copy(), promise);
	} finally {
	ReferenceCountUtil.safeRelease(b);
	}
	} else {
	ctx.write(msg, promise);
	}
	}
	}

This will be removed in a separate PR.

Modifications

• Upgrade Netty to 4.1.111.Final version

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[lhotari]

etcd tests break with Netty 4.1.111.Final. There's some compatibility issue.
I also tried upgrading jetcd to 0.8.2 and grpc to 1.64.0, but that didn't help.

[lhotari]

It seems that grpc-netty dependency will have to be excluded and replaced by grpc-netty-shaded since grpc is compatible only with specific Netty versions.

[lhotari]

Unfortunately jetcd-core isn't compatible with grpc-netty-shaded. There are conflicts with io.grpc.netty.GrpcSslContexts/io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts and io.netty.handler.ssl.SslContext/io.grpc.netty.shaded.io.netty.handler.ssl.SslContext classes. The only solution seems to be to do custom shading for jetcd-core and jetcd-test.

[lhotari]

I found a reasonable solution by shading jetcd-core in a way where vertx is relocated and included, but grpc is switched to use grpc-netty-shaded instead of grpc-netty. stream-storage-java-client supports switching to use grpc-netty-shaded without making changes to the module.
This resolves the issue with grpc-netty and Netty 4.1.111.Final.

[merlimat]

@lhotari There are quite a few merge conflict everywhere (except for 3.3). I'm not sure how difficult it would be to backport.

[lhotari]

@lhotari There are quite a few merge conflict everywhere (except for 3.3). I'm not sure how difficult it would be to backport.

I'll handle the backporting. I think it's doable.