Remove warning logs of unknown configs for Kafka passwordless

• Feature request
• Cause analysis
• Expected behavior
• Design
• Needs to confirm

Feature request

As is reported in https://github.com/Azure/azure-sdk-for-java/issues/30800#issuecomment-1254620865, when using Event Hubs for Kafka passwordless-conneciton with SCS Kafka binder, there are a batch of warning logs saying, "The configuration 'xxx' was supplied but isn't a known config." We should consider preventing those warning logs being printed.

Currently there are 3 scenarios that can cause warning logs when using the password-less feature:

1. By default, developers use the least and required properties:

   # When using org.springframework.cloud:spring-cloud-starter-stream-kafka, developers set the only required broker config
   spring.cloud.stream.kafka.binder.brokers=<namespace>.servicebus.windows.net:9093

   or

   # When using org.springframework.kafka:spring-kafka in a Spring Boot application, developers set the only required bootstrap-server config
   spring.kafka.bootstrap-servers=<namespace>.servicebus.windows.net:9093

2. When developers manually configure any customized credential/profile properties via Spring Cloud Azure global configuration:

   # When using org.springframework.cloud:spring-cloud-starter-stream-kafka
   spring.cloud.stream.kafka.binder.brokers=<namespace>.servicebus.windows.net:9093
   spring.cloud.azure.credential.managed-identity-enabled=true

   or

   # When using org.springframework.kafka:spring-kafka in a Spring Boot application
   spring.kafka.bootstrap-servers=<namespace>.servicebus.windows.net:9093
   spring.cloud.azure.credential.managed-identity-enabled=true

3. When developers manually configure any customized credential/profile properties via Kafka configuration:

   # When using org.springframework.cloud:spring-cloud-starter-stream-kafka
   spring.cloud.stream.kafka.binder.brokers=<namespace>.servicebus.windows.net:9093
   spring.cloud.stream.kafka.binder.producer-properties.azure.credential.managed-identity-enabled=true

   or

   # When using org.springframework.kafka:spring-kafka in a Spring Boot application
   spring.kafka.bootstrap-servers=<namespace>.servicebus.windows.net:9093
   spring.kafka.properties.azure.credential.managed-identity-enabled=true

Cause analysis

The warning logs are caused because, in the above 3 cases, there are spring cloud azure credential/profile related properties in the application context, and we put all those credential/profile properties to the client portion of Kafka properties. Here the client portion of Kafka properties refers to:

1. for Spring Bboot Kafka properties with the prefix as spring.kafka., we put those Azure properties to KafkaProperties bean in the portion like spring.kafka.producer.properties.{key};

2. for SCS Kafka binder properties with the prefix as spring.cloud.stream.kafka.binder., we put those Azure properties to the KafkaBinderConfigurationProperties bean in the portion like spring.cloud.stream.kafka.binder.producer-properties.{key}.

those properties are treated as configuraiton targeted to Kafka clients and will be passed to clients, however since the properties are not Kafka-defined so when Kafka tries to parse them, it fails to recognize and then starts to print warning logs.

Expected behavior

We expect that when the feature gets done, developers could keep the existing way of using Kafka password-less connection including the same API, properties for spring boot and cloud stream, library dependency. But will not see any warning logs of "The configuration 'xxx' was supplied but isn't a known config.".

Design

In the above cases, there are Azure credential/profile properties placed to Kafka client properties. Those Azure properties are valuable since they provide the way to customizd the credentials used in password-less connection, and also mark the information about the target Event Hubs server, which will be picked by our callback handler when execute OAuth2 authentication. The thing is that we chose to place them in Kafka client configs, which breaks Kafka design to some extend. In that case, we need to consider to place those properties via other ways.

According to Kafka documentation, Kafka protocol supports SSL and SASL mechanism to go authentication and establish secured conneciton. In our case of password-less connection, we choose Kafka's SASL/OAUTHBEARER mechanism for the authentication procedure to connect Azure Event Hubs, which uses the OAuth2 framework. And, all of our provided credential/profile properties are used for the SASL/OAUTHBEARER authentication purpose. Referring to Kafka's security documentation, Kafka uses the Java Authentication and Authorization Service (JAAS) for SASL configuration, thus, a more proper position to place our Azure properties should be the JAAS configuration, which means we should place all the Azure properties from the client portion of Kafka properties to the JAAS portion, and to avoid causing warning logs, all Azure properties configured in client portion should be removed.

The JAAS configuration for Kafka is keyed as sasl.jaas.config, with the pattern of <LoginModule> <flag> <LoginModule options>;, where in our password-less support, the login module is org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule and flag is required. Then the login-module options can be used to place our Azure credential/profile properties, we can move Azure properties to Kafka JAAS configs and make it with the format like org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required azure.credential.managed-identity-enabled="true" key2="xxxxx";.

Note here is actually we that move Azure properties to Kafka JAAS configs and fill in Kafka JAAS configs, which is not to let developers to manually configure the credential/profile properties as the above format. Developers should still configure as the same way and not be aware of such changes.

For example, we should convert Azure credential/profile properties to JAAS configuration with the format like:

# When using org.springframework.cloud:spring-cloud-starter-stream-kafka
spring.cloud.stream.kafka.binder.consumer-properties.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required azure.credential.managed-identity-enabled=“true” key2="xxxxx";

Exit criteria

1. With all the above mentioned 3 scenarios, the application can connect successfully without warning logs.
2. When developers set other non-Azure-properties via LoginModule options with our expected pattern(JAAS login module and flag), then those options should not be overriden.
3. When developers set other Azure-properties via LoginModule options with our expected pattern(JAAS login module, flag and property key/values), then those options should only be overriden when there are the same properties configured by other higher priority configs.
4. When developers set properties from Spring Boot Kafka properties, we should make sure it can be passed to SCS Kafka without warnings.

Needs to confirm

1. What if developers configure JAAS configuration via static JAAS config file?
2. We have required that flag of JAAS configuration to be requried, can it be requisite?
3. When developers set JAAS configuration for KafkaJaasLoginModuleInitializer, should we take it into consideration?