Spring Kafka with GraalVM - org.apache.kafka.common.errors.SaslAuthenticationException: Failed to create SaslClient with mechanism PLAIN

[Richardmbs12]

In what version(s) of Spring for Apache Kafka are you seeing this issue?

3.0.1

Describe the bug
I am still new to Spring boot 3 aot. Best I can describe it is that I've used the spring boot 3 native profile defaults, but I get the following error only on graalvm, not on the JVM:

org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
--
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to create SaslClient with mechanism PLAIN
2023-01-12T14:54:57.583Z  WARN 9 --- [ad \| producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Error connecting to node dsbggena117v.standardbank.co.za:9092 (id: -2 rack: null)
java.io.IOException: Channel could not be created for socket java.nio.channels.SocketChannel[closed]
at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:348) ~[na:na]
at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:329) ~[na:na]
at org.apache.kafka.common.network.Selector.connect(Selector.java:256) ~[na:na]
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:992) ~[na:na]
at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:73) ~[na:na]
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1163) ~[na:na]
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1051) ~[na:na]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:558) ~[na:na]
at org.apache.kafka.clients.NetworkClientUtils.isReady(NetworkClientUtils.java:42) ~[na:na]
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:64) ~[na:na]
at org.apache.kafka.clients.producer.internals.Sender.awaitNodeReady(Sender.java:534) ~[na:na]
at org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:455) ~[na:na]
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:316) ~[na:na]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) ~[na:na]
at java.base@17.0.5/java.lang.Thread.run(Thread.java:833) ~[payment-validation:na]
at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775) ~[payment-validation:na]
at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203) ~[na:na]
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:240) ~[na:na]
at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:338) ~[na:na]
... 16 common frames omitted
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to create SaslClient with mechanism PLAIN

To Reproduce
Any project with spring boot 3.0.1, spring-kafka and the following configuration in the application.yml

spring:
  config:
    activate:
      on-profile: dev
  kafka:
    bootstrap-servers:
      - "XXX"
    jaas:
      loginModule: org.apache.kafka.common.security.plain.PlainLoginModule
      options:
        username: "XXX"
        password: "XXX"
      enabled: true
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      ssl:
        trust-store-type: PEM
        trust-store-location: file:/opt/cacrt/my-cachain.pem
    properties:
      sasl:
        mechanism: PLAIN
      security:
        protocol: SASL_SSL
    admin:
      ssl:
        trust-store-type: PEM
        trust-store-location: file:/opt/cacrt/my-cachain.pem

Expected behavior
I expect the required reflections and hints to be available and not receive this error that I am not sure how to resolve.

[artembilan]

Looks like it fails here:

                SaslClient retvalSaslClient = Sasl.createSaslClient(mechs, clientPrincipalName, servicePrincipal, host, configs, callbackHandler);
                if (retvalSaslClient == null) {
                    throw new SaslAuthenticationException("Failed to create SaslClient with mechanism " + mechanism);
                }

Which is really a call to javax.security.sasl package - Java by itself, nothing Spring or even Apache Kafka specific.
From its JavaDocs:

     *@return A possibly null {@code SaslClient} created using the parameters
     * supplied. If null, cannot find a {@code SaslClientFactory}
     * that will produce one.

The Java's ServiceLoader cannot load any SaslClientFactory impls in classpath?

I don't know, they claim here that there is some fix in GraalVM: oracle/graal#3664, but then we need to be sure that you use the latest GraalVM for your project.

Either way it is probably better to raise this concern with GraalVM team: not sure what Spring responsibility could be involved with this feature...

[artembilan]

Here is some old discussion as well: spring-attic/spring-native#1416.
Doesn't look in the end that there is a solution...

[artembilan]

This article says something promising: https://jeqo.github.io/posts/2022-03-18-kafka-clients-graalvm/#sasl-authentication

<buildArg>-H:AdditionalSecurityProviders=com.sun.security.sasl.Provider</buildArg>
<buildArg>--initialize-at-run-time=org.apache.kafka.common.security.authenticator.SaslClientAuthenticator</buildArg>

Just give it a try, please!

[Richardmbs12]

Thank you very much for the feedback it helped a lot to point me in the right direction and I did have some breakthroughs, I will provide final feedback soon when I get the below optimized.

Currently I have the following that fixed that issue:

Reflections:

{
   "name": "org.apache.kafka.common.security.ssl.DefaultSslEngineFactory",
   "allDeclaredConstructors": true,
   "allPublicConstructors": true,
   "allDeclaredMethods": true,
   "allPublicMethods": true,
   "allDeclaredFields": true,
   "allPublicFields": true
 },
 {
   "name": "javax.security.auth.Subject",
   "allDeclaredConstructors": true,
   "allPublicConstructors": true,
   "allDeclaredMethods": true,
   "allPublicMethods": true,
   "allDeclaredFields": true,
   "allPublicFields": true
 },
 {
   "name": "org.apache.kafka.common.network.Selector",
   "allDeclaredConstructors": true,
   "allPublicConstructors": true,
   "allDeclaredMethods": true,
   "allPublicMethods": true,
   "allDeclaredFields": true,
   "allPublicFields": true
 },
 {
   "name": "org.apache.kafka.common.network.SaslChannelBuilder",
   "allDeclaredConstructors": true,
   "allPublicConstructors": true,
   "allDeclaredMethods": true,
   "allPublicMethods": true,
   "allDeclaredFields": true,
   "allPublicFields": true
 }

Build arg added - -H:AdditionalSecurityProviders=com.sun.security.sasl.Provider

I still need to experiment with
--initialize-at-run-time=org.apache.kafka.common.security.authenticator.SaslClientAuthenticator
per @artembilan suggestion

Not sure if the runtime arg suggestion would be better or if reflections is better from a image optimization point of view?

Currently the above mentioned allows the project to startup with graalvm, however I find that the kafka SSL properties isn't getting loaded with graalvm

In the spring boot properties the producer's ssl trust store type and location is set, but on graalvm it does not want to pick up the ssl properties specifically, although it picks up the rest. (on JVM it works).

My yaml again:

spring:
  config:
    activate:
      on-profile: dev
  kafka:
    bootstrap-servers:
      - "XXX"
    jaas:
      loginModule: org.apache.kafka.common.security.plain.PlainLoginModule
      options:
        username: "XXX"
        password: "XXX"
      enabled: true
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      ssl:
        trust-store-type: PEM
        trust-store-location: file:/opt/cacrt/my-cachain.pem
    properties:
      sasl:
        mechanism: PLAIN
      security:
        protocol: SASL_SSL
    admin:
      ssl:
        trust-store-type: PEM
        trust-store-location: file:/opt/cacrt/my-cachain.pem

Only spring.producer.ssl doesnt pickup

Spring logs

2023-01-17T06:20:18.455Z  INFO 9 --- [pool-5-thread-1] o.a.k.clients.producer.ProducerConfig    : ProducerConfig values:
--
acks = -1
batch.size = 16384
bootstrap.servers = [XXX - hidden]
buffer.memory = 33554432
client.dns.lookup = use_all_dns_ips
client.id = producer-1
compression.type = none
connections.max.idle.ms = 540000
delivery.timeout.ms = 120000
enable.idempotence = true
interceptor.classes = []
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metadata.max.idle.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.adaptive.partitioning.enable = true
partitioner.availability.timeout.ms = 0
partitioner.class = null
partitioner.ignore.keys = false
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.connect.timeout.ms = null
sasl.login.read.timeout.ms = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.login.retry.backoff.max.ms = 10000
sasl.login.retry.backoff.ms = 100
sasl.mechanism = PLAIN
sasl.oauthbearer.clock.skew.seconds = 30
sasl.oauthbearer.expected.audience = null
sasl.oauthbearer.expected.issuer = null
sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
sasl.oauthbearer.jwks.endpoint.url = null
sasl.oauthbearer.scope.claim.name = scope
sasl.oauthbearer.sub.claim.name = sub
sasl.oauthbearer.token.endpoint.url = null
security.protocol = SASL_SSL
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class org.springframework.kafka.support.serializer.JsonSerializer
2023-01-17T06:20:18.464Z  INFO 9 --- [pool-5-thread-1] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Instantiated an idempotent producer.
2023-01-17T06:20:18.572Z  INFO 9 --- [pool-5-thread-1] o.a.k.c.s.authenticator.AbstractLogin    : Successfully logged in.
2023-01-17T06:20:18.575Z  WARN 9 --- [pool-5-thread-1] o.a.kafka.common.utils.AppInfoParser     : Error while loading kafka-version.properties: inStream parameter is null
2023-01-17T06:20:18.575Z  INFO 9 --- [pool-5-thread-1] o.a.kafka.common.utils.AppInfoParser     : Kafka version: unknown
2023-01-17T06:20:18.575Z  INFO 9 --- [pool-5-thread-1] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: unknown
2023-01-17T06:20:18.575Z  INFO 9 --- [pool-5-thread-1] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1673936418575
2023-01-17T06:20:18.726Z  INFO 9 --- [ad \| producer-1] o.apache.kafka.common.network.Selector   : [Producer clientId=producer-1] Failed authentication with dsbggena117v.standardbank.co.za/10.145.192.117 (channelId=-2) (SSL handshake failed)
2023-01-17T06:20:18.726Z  INFO 9 --- [ad \| producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Node -2 disconnected.
2023-01-17T06:20:18.726Z ERROR 9 --- [ad \| producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node -2 (XXX - hidden) failed authentication due to: SSL handshake failed
2023-01-17T06:20:18.726Z  WARN 9 --- [ad \| producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Bootstrap broker XXX - hidden (id: -2 rack: null) disconnected
2023-01-17T06:20:18.726Z ERROR 9 --- [pool-5-thread-1] o.s.k.support.LoggingProducerListener    : Exception thrown when sending a message with key='null' and payload=XXX - hidden
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Thus it doesnt load the SSL properties at all, it uses defaults, and this causes the SSL PKIX path issue.

So I am still trying to figure out why those properties wont build with graalvm and Spring Boot Autoconfiguration

This is the method that sets the properties
org.springframework.boot.autoconfigure.kafka.KafkaProperties.Ssl#buildProperties

Please let me know if you have any ideas. Thanks for the support

[Richardmbs12]

When I put reflections for all classes on for the whole apache.kafka package then everything works. Im struggling to find what the right reflection is needed to pickup the SSL config for the producer :(

[nandomelo]

In one of my environments, I kept getting the same error:

Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to create SaslClient with mechanism PLAIN

This error only occurred when the code was running on a Kubernetes server, while it worked perfectly fine locally.
My local JDK was more up to date than the one used to generate the image for the k8s, so I thought updating it could be a way to solve.

I changed the image used to build the native image from ghcr.io/graalvm/graalvm-ce:22.3.1 to ghcr.io/graalvm/graalvm-community:17-ol9. While the specific version is likely not a crucial factor, updating the GraalVM version seems to do the job