-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-26322][SS] Add spark.kafka.sasl.token.mechanism to ease delegation token configuration. #23274
Conversation
…tion token configuration.
@@ -642,9 +642,9 @@ This way the application can be configured via Spark parameters and may not need | |||
configuration (Spark can use Kafka's dynamic JAAS configuration feature). For further information | |||
about delegation tokens, see [Kafka delegation token docs](http://kafka.apache.org/documentation/#security_delegation_token). | |||
|
|||
The process is initiated by Spark's Kafka delegation token provider. When `spark.kafka.bootstrap.servers`, | |||
The process is initiated by Spark's Kafka delegation token provider. When `spark.kafka.bootstrap.servers` set, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've found this wording issue so fixed it here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In that case, it should be "is set".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed.
Spark considers the following log in options, in order of preference: | ||
- **JAAS login configuration** | ||
- **JAAS login configuration**, please see example below. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added this small pointer to make things more clear.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, @gaborgsomogyi . Which example is this pointing? Previous examples seems to be removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dongjoon-hyun down below with the same JAAS login configuration
name.
Test build #99917 has finished for PR 23274 at commit
|
The feature works on cluster but I'm not fully happy with the |
core/src/main/scala/org/apache/spark/internal/config/Kafka.scala
Outdated
Show resolved
Hide resolved
@@ -642,9 +642,9 @@ This way the application can be configured via Spark parameters and may not need | |||
configuration (Spark can use Kafka's dynamic JAAS configuration feature). For further information | |||
about delegation tokens, see [Kafka delegation token docs](http://kafka.apache.org/documentation/#security_delegation_token). | |||
|
|||
The process is initiated by Spark's Kafka delegation token provider. When `spark.kafka.bootstrap.servers`, | |||
The process is initiated by Spark's Kafka delegation token provider. When `spark.kafka.bootstrap.servers` set, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In that case, it should be "is set".
Test build #99963 has finished for PR 23274 at commit
|
Test build #99986 has finished for PR 23274 at commit
|
retest this, please |
Test build #99991 has finished for PR 23274 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It sounds a nice improvement to me, and LGTM on code changes.
Merging to master. |
…tion token configuration. ## What changes were proposed in this pull request? When Kafka delegation token obtained, SCRAM `sasl.mechanism` has to be configured for authentication. This can be configured on the related source/sink which is inconvenient from user perspective. Such granularity is not required and this configuration can be implemented with one central parameter. In this PR `spark.kafka.sasl.token.mechanism` added to configure this centrally (default: `SCRAM-SHA-512`). ## How was this patch tested? Existing unit tests + on cluster. Closes apache#23274 from gaborgsomogyi/SPARK-26322. Authored-by: Gabor Somogyi <gabor.g.somogyi@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
…tion token configuration. ## What changes were proposed in this pull request? When Kafka delegation token obtained, SCRAM `sasl.mechanism` has to be configured for authentication. This can be configured on the related source/sink which is inconvenient from user perspective. Such granularity is not required and this configuration can be implemented with one central parameter. In this PR `spark.kafka.sasl.token.mechanism` added to configure this centrally (default: `SCRAM-SHA-512`). ## How was this patch tested? Existing unit tests + on cluster. Closes apache#23274 from gaborgsomogyi/SPARK-26322. Authored-by: Gabor Somogyi <gabor.g.somogyi@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
What changes were proposed in this pull request?
When Kafka delegation token obtained, SCRAM
sasl.mechanism
has to be configured for authentication. This can be configured on the related source/sink which is inconvenient from user perspective. Such granularity is not required and this configuration can be implemented with one central parameter.In this PR
spark.kafka.sasl.token.mechanism
added to configure this centrally (default:SCRAM-SHA-512
).How was this patch tested?
Existing unit tests + on cluster.