-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-45753][CORE] Support spark.deploy.driverIdPattern
#43615
Conversation
Could you review this Spark |
core/src/main/scala/org/apache/spark/internal/config/Deploy.scala
Outdated
Show resolved
Hide resolved
Patterns like below will use space for padding, "driver-%s-%4d".format("20231111111111", 1)
driver-20231111111111- 1 Can we have a test too to see if other components, such as UI, driver work properly? |
Thank you for review.
|
", e.g., `driver-20231031224459-0019`. Please be careful to generate unique IDs") | ||
.version("4.0.0") | ||
.stringConf | ||
.checkValue(!_.format(new Date(), 0).exists(_.isWhitespace), "Whitespace is not allowed.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We execute String.format
here in advance to prevent the failure in Master
class.
assert(master.invokePrivate(_newDriverId(submitDate)) === "my-driver-00001") | ||
} | ||
|
||
test("SPARK-45753: Prevent invalid driver id patterns") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is a test case for that.
Could you review this once more, @yaooqinn ? I believe this is the best effort we can. |
LGTM, thanks for the updates |
Thank you so much always! |
Thank you! Merged to master for Apache Spark 4.0.0. |
### What changes were proposed in this pull request? This PR aims to support `spark.deploy.driverIdPattern` for Apache Spark 4.0.0. ### Why are the changes needed? This allows the users to be able to control driver ID pattern. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs with the newly added test case. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#43615 from dongjoon-hyun/SPARK-45753. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
What changes were proposed in this pull request?
This PR aims to support
spark.deploy.driverIdPattern
for Apache Spark 4.0.0.Why are the changes needed?
This allows the users to be able to control driver ID pattern.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the CIs with the newly added test case.
Was this patch authored or co-authored using generative AI tooling?
No.