Remove warning logs of unknown configs for Kafka passwordless

• Context
• Cause analysis
• Solution design
• Implementation design
  • Spring Kafka in Spring Boot applications
    • Merge Azure properties
    • Convert to JAAS configuration
    • Put to Kafka client properties
    • Remove Azure configuration
  • SCS Kafka applications
    • Merge Azure properties
    • Convert to JAAS configuration
    • Put to Kafka client properties
    • Remove Azure configuration
• Needs to confirm

Context

As is reported in https://github.com/Azure/azure-sdk-for-java/issues/30800#issuecomment-1254620865, when using Event Hubs for Kafka passwordless-conneciton, there are a batch of warning logs saying, "The configuration 'xxx' was supplied but isn't a known config". We should consider preventing those warning logs being printed.

Cause analysis

The warning logs are caused because, in the mentioned 3 cases from the feature spec, there are spring cloud azure credential/profile related properties in the application context, and we put all those credential/profile properties to the client portion of Kafka properties. Here the client portion of Kafka properties refers to:

1. for Spring Bboot Kafka properties with the prefix as spring.kafka., we put those Azure properties to KafkaProperties bean in the portion like spring.kafka.producer.properties.{key};

2. for SCS Kafka binder properties with the prefix as spring.cloud.stream.kafka.binder., we put those Azure properties to the KafkaBinderConfigurationProperties bean in the portion like spring.cloud.stream.kafka.binder.producer-properties.{key}.

Those properties are treated as configuraiton targeted to Kafka clients and will be passed to clients, however since the properties are not Kafka-defined so when Kafka tries to parse them, it fails to recognize and then starts to print warning logs.

Solution design

In the above cases, there are Azure credential/profile properties placed to Kafka client properties. Those Azure properties are valuable since they provide the way to customizd the credentials used in password-less connection, and also mark the information about the target Event Hubs server, which will be picked by our callback handler when execute OAuth2 authentication. The thing is that we chose to place them in Kafka client configs, which breaks Kafka design to some extend. In that case, we need to consider to place those properties via other ways.

According to Kafka documentation, Kafka protocol supports SSL and SASL mechanism to go authentication and establish secured conneciton. In our case of password-less connection, we choose Kafka's SASL/OAUTHBEARER mechanism for the authentication procedure to connect Azure Event Hubs, which uses the OAuth2 framework. And, all of our provided credential/profile properties are used for the SASL/OAUTHBEARER authentication purpose. Referring to Kafka's security documentation, Kafka uses the Java Authentication and Authorization Service (JAAS) for SASL configuration, thus, a more proper position to place our Azure properties should be the JAAS configuration, which means we should place all the Azure properties from the client portion of Kafka properties to the JAAS portion, and to avoid causing warning logs, all Azure properties configured in client portion should be removed.

The JAAS configuration for Kafka is keyed as sasl.jaas.config, with the pattern of <LoginModule> <flag> <LoginModule options>;, where in our password-less support, the login module is org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule and flag is required. Then the login-module options can be used to place our Azure credential/profile properties, we can move Azure properties to Kafka JAAS configs and make it with the format like org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required azure.credential.managed-identity-enabled="true" key2="xxxxx";.

Note here is actually we that move Azure properties to Kafka JAAS configs and fill in Kafka JAAS configs, which is not to let developers to manually configure the credential/profile properties as the above format. Developers should still configure as the same way and not be aware of such changes.

For example, we should convert Azure credential/profile properties to JAAS configuration with the format like:

# When using org.springframework.cloud:spring-cloud-starter-stream-kafka
spring.cloud.stream.kafka.binder.consumer-properties.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required azure.credential.managed-identity-enabled=“true” key2="xxxxx";
# When using org.springframework.kafka:spring-kafka in a Spring Boot application
spring.kafka.consumer.properties.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required azure.credential.managed-identity-enabled=“true” key2="xxxxx";

Note, when leveraging JAAS configs to set Azure properties, we should make sure:

1. When developers set non-Azure-properties via LoginModule options with our expected pattern(JAAS login module and flag), then those options should not be overridden.
2. When developers set Azure-properties via LoginModule options with our expected pattern(JAAS login module, flag and property key/values), then only when there are the same property configured by other alternatives, like Spring Cloud Azure/Spring Boot Kafka/SCS Kafka properties, can those LoginModule options be overridden with other high priority values.

Implementation design

The core principle about removing warning logs is to move the Azure properties from client portion of Kafka properties to the JAAS portion. Thus to do this, we should implement the following 4 steps:

1. For each Kafka client, merge all Azure properties from different sources with priorities.
2. Convert the merged properties to a set of string following JAAS pattern.
3. Configure each Kafka client's JAAS config with its JAAS string.
4. Remove all Azure configuration from Kafka properties.

For each step, since there are some inconsistency for Spring Boot Kafka properties and SCS Kafka binder properties, so we need to consider them seperatedly.

Spring Kafka in Spring Boot applications

Merge Azure properties

For the Spring Boot Kafka autoconfiguration, the properties can be divided into 4 parts:

1. The default properties which can be passed to all the Kafka clients with the lowest priority.
2. Producer client properties which can only apply to the KafkaProducer client.
3. Consumer client properties which can only apply to the KafkaConsumer client.
4. Admin client properties which can only apply to the KafkaAdmin client.

For each of the above 4 parts, the properties can be further divided into kafka-defined and customized ones, where the sasl.jaas.config is the former one, and any our Azure properties are the latter. The below picture shows the workflow of how the client properties are finally built from Spring Boot Kafka properties. Taking the producer as an example, the kafka-defined properties in default part will be passed upward to kafka-defined properties in the producer and get merged/overrided. Same for the customized configs.

When merging Azure credential/profile properties, we should take configuration from 4 sources into consideration with the priority from low to high: original JAAS configs, Spring Cloud Azure properties, Kafka default properties and client properties. To make the merging process easier, we can use a Map for the merged properties and then just iterate the sources and store to the Map. Thus we can use an Object of Jaas to abstract those configurqations which are actually the LoginModule-option member within it. And use an Object of JaasResolver to perform the merge process.

Convert to JAAS configuration

To convert the merged properties to a String of JAAS patter, we can just override the toString method of Jaas.

Put to Kafka client properties

Here we should make sure every Kafka client will be configured with its associated JAAS configuration. Therefore, we should put the configuration to the Kafka-defined client part of Kafka properties.

Remove Azure configuration

We should make sure both the client and default parts get removed. To make sure that the removed properties won't have side effects, we can remove after the properties have converted.

So the whole flow for Spring Boot Kafka application should be:

SCS Kafka applications

Merge Azure properties

For the SCS Kafka autoconfiguration, the properties can be divided into 4 parts:

1. The Spring Boot Kafka properties.
2. The default properties which can be passed to all the Kafka clients with the lowest priority.
3. Producer client properties which can only apply to the KafkaProducer client.
4. Consumer client properties which can only apply to the KafkaConsumer client.

For each of the above 4 parts, the properties can be further divided into kafka-defined and customized ones. However, the difference between SCS and Spring Boot Kafka is that, the customized configs in default properties won't be passed upwards to the producer or consumer. The below picture shows the workflow of how the client properties are finally built from SCS Kafka Binder properties.

When merging Azure credential/profile properties, we should take configuration from 4 sources into consideration with the priority from low to high: original JAAS configs, Spring Boot Kafka properties(merged with Spring Cloud Azure), Kafka default properties and client properties. However, when parsing the default portion, we should manually collect the Azure properties if any.

Convert to JAAS configuration

Same as boot.

Put to Kafka client properties

Here we should make sure every Kafka client will be configured with its associated JAAS configuration. Therefore, we should put the configuration to the Kafka-defined client part of Kafka properties. Since there is no admin part in binder properties, and Kafka binder handles both Spring Boot and binder default properties for admin. The we should put the JAAS configuration to default portion since it has the highest priority for admin. And the hanlding of admin should be the last to avoid it pollutes the property sources of producer and consumer.

Remove Azure configuration

Same as boot.

So the whole flow for SCS Kafka Binder application should be:

Needs to confirm

1. What if developers configure JAAS configuration via static JAAS config file?
2. We have required that flag of JAAS configuration to be requried, can it be requisite?
3. When developers set JAAS configuration for KafkaJaasLoginModuleInitializer, should we take it into consideration?