Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider using Maven based spark job submission #211

Closed
xiaoyongzhu opened this issue May 3, 2022 · 0 comments · Fixed by #334
Closed

Consider using Maven based spark job submission #211

xiaoyongzhu opened this issue May 3, 2022 · 0 comments · Fixed by #334

Comments

@xiaoyongzhu
Copy link
Member

Since Feathr is on Maven now (https://search.maven.org/artifact/com.linkedin.feathr/feathr_2.12), we should consider using Maven as the source to submit Spark jobs,rather than using the public wasb path, since it's a bit hard to maintain and doesn't distribute well (like don't have mirrors etc., and will be slow).

Databricks supports maven based library: https://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/libraries#--library

Synapse can be achieved by using this config: spark.jars.packages. See (https://spark.apache.org/docs/3.2.1/configuration.html) for more details.

windoze added a commit that referenced this issue Jun 9, 2022
windoze added a commit that referenced this issue Jun 9, 2022
@xiaoyongzhu xiaoyongzhu linked a pull request Jun 9, 2022 that will close this issue
windoze added a commit that referenced this issue Jun 13, 2022
* #211 Enable using maven package

* #211 Enable using maven package

* Print Maven package artifact id when using it

* Add e2e test for Maven job submission
hangfei pushed a commit that referenced this issue Jun 13, 2022
* #211 Enable using maven package

* #211 Enable using maven package

* Print Maven package artifact id when using it

* Add e2e test for Maven job submission
hangfei pushed a commit that referenced this issue Jun 13, 2022
* #211 Enable using maven package

* #211 Enable using maven package

* Print Maven package artifact id when using it

* Add e2e test for Maven job submission
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant