Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dependency downloader #4

Merged
merged 2 commits into from
Feb 16, 2016

Conversation

chipsenkbeil
Copy link
Contributor

Adds the following options to %AddDeps:

  • --repository <URL> to include another repository when downloading
  • --trace to print out download information
  • --verbose to print out miscellaneous information
%AddDeps org.apache.spark spark-streaming-mqtt_2.10 1.5.2 --trace --transitive --repository https://extra-repo.com

These options will show up in the help menu. You can provide --repository more than once to include more than one additional repository.

Will ignore any malformed URL. Will try repositories provided by --repository first before default repositories.

@chipsenkbeil chipsenkbeil reopened this Feb 8, 2016
@lbustelo
Copy link
Contributor

taking a look at this now

@lbustelo
Copy link
Contributor

Tried a couple of dependencies and different repositories and all worked fine.

I saw that dependencies are getting installed under /tmp/.ivy2/https/repo1.maven.org/maven2/io/netty/netty/3.6.3.Final/netty-3.6.3.Final.jar. Is it fair to say that we still do not have dependency isolation with AddDeps between kernel instances?

@lbustelo
Copy link
Contributor

)*/
val dependencyDownloader = new CoursierDependencyDownloader
dependencyDownloader.setDownloadDirectory(
new File(config.getString("ivy_local"))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we change this to something else, we could avoid clashes between kernel instances. Or, if we set the default ivy_local to a temporary directory, that would work as well.

@chipsenkbeil
Copy link
Contributor Author

@chipsenkbeil
Copy link
Contributor Author

FYI, I read that using Coursier for resolution/downloading over standard sbt has a large increase in performance. Might be worth adding the plugin for us to see if it helps with Toree dependency resolution/downloading, especially when running automated build/test.

@lbustelo
Copy link
Contributor

Lets handle the location of the deps in a separate PR. I opened https://issues.apache.org/jira/browse/TOREE-261. We should just go ahead and get that work done asap since this is fresh.

@asfgit asfgit merged commit 116d8bc into apache:master Feb 16, 2016
@chipsenkbeil chipsenkbeil deleted the UpdateDependencyDownloader branch February 16, 2016 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants