-
Notifications
You must be signed in to change notification settings - Fork 260
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider using Maven based spark job submission #211
Comments
windoze
added a commit
that referenced
this issue
Jun 9, 2022
windoze
added a commit
that referenced
this issue
Jun 9, 2022
windoze
added a commit
that referenced
this issue
Jun 13, 2022
hangfei
pushed a commit
that referenced
this issue
Jun 13, 2022
hangfei
pushed a commit
that referenced
this issue
Jun 13, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Since Feathr is on Maven now (https://search.maven.org/artifact/com.linkedin.feathr/feathr_2.12), we should consider using Maven as the source to submit Spark jobs,rather than using the public wasb path, since it's a bit hard to maintain and doesn't distribute well (like don't have mirrors etc., and will be slow).
Databricks supports maven based library: https://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/libraries#--library
Synapse can be achieved by using this config:
spark.jars.packages
. See (https://spark.apache.org/docs/3.2.1/configuration.html) for more details.The text was updated successfully, but these errors were encountered: