Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Support --archives option in SparkApplication CRD and spark-submit #2245

Closed
kaka-zb opened this issue Oct 14, 2024 · 2 comments · Fixed by #2256
Closed

[FEATURE] Support --archives option in SparkApplication CRD and spark-submit #2245

kaka-zb opened this issue Oct 14, 2024 · 2 comments · Fixed by #2256
Labels
enhancement New feature or request

Comments

@kaka-zb
Copy link
Contributor

kaka-zb commented Oct 14, 2024

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

PySpark is used In many AI data cleaning scenarios, then in most of those scenarios, the --archives option or spark.archives configuration will be used to deal with Python package management. Therefore, we should consider adding a archives field in deps field of SparkApplication CRD to support this frequently used spark param.

Describe the solution you would like

  • update the crd defination by adding the archives field
  • add --archives params when executing spark-submit command in operator

Describe alternatives you have considered

Additional context

If needed, I can submit a PR

@kaka-zb kaka-zb added the enhancement New feature or request label Oct 14, 2024
@kaka-zb kaka-zb changed the title [FEATURE] Brief description of the feature [FEATURE] Support --archives option in SparkApplication CRD and spark-submit Oct 14, 2024
@ChenYi015
Copy link
Contributor

@kaka-zb Sounds reasonable, it would be great if you can implement this feature.

@kaka-zb
Copy link
Contributor Author

kaka-zb commented Oct 14, 2024

@ChenYi015 OK,I'm glad to implement this feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants