Local Git dependency using relative file path

[fluffysquirrels]

@ghost proposed in issue #2252 :

Currently Rust 1.32.0 doesn't work with local Git relative paths, like this:

[dependencies]
some-lib = { git='file:../../some-lib/', tag='...' }

[fluffysquirrels]

To anyone trying to do this, as a workaround, you could checkout the right version of the dependency as a Git sub-module at the chosen path, then use a relative path in the Cargo config.

[Denommus]

This solution doesn't work when you use workspaces and the submodule in question is also a workspace, though.

[cbs228]

While submodules work, they have one big disadvantage: if a crate X has a submodule dependency on Y SHA-1 DEADB33F, then every crate which subsequently uses X is also forced to use Y version DEADB33F. For complex projects, this is a non-starter. If you do this too much, then your project also becomes full of recursive submodules. A cargo-based solution is desirable.

What should relative git URLs be relative to? I see two obvious candidates:

1. Relative URLs could be interpreted relative to the working copy. I am not sure if this is useful.

2. URLs could also be interpreted relative to one or more of the git remotes.

(2) is how relative URLs in git-submodule work, and I think this is the way to go. Git submodules use the following rule to interpret relative URLs:

The default remote is the remote of the remote-tracking branch of the current branch. If no such remote-tracking branch exists or the HEAD is detached, "origin" is assumed to be the default remote.

Support for relative URLs in {git=…} would improve dependency specification in unpublished crates. Related unpublished crates could depend on each other while also being flexible about their URL. Forking several crates is simplified. As long as the repos are in one place, they will continue to work. This may also improve patches for forks of published crates.

Is this a good idea? Are PRs sought for this feature?

[cbs228]

After a quick reading of the source code, I find the following barriers to implementation:

• URL Parsing: URLs in libgit2/git2 are not url::Url. Not all URLs for git remotes can be parsed as WHATWG URLs. At present, url does not support scp-style URLs. (Related: Support SSH Git URLs for authenticated connections to private repositories #1851.) scp URLs will need to be converted "by hand."

• SourceId needs to be created from a type that is IntoUrl. This requirement means that the source must be identified by absolute URL.

• Determine the git remote which corresponds to the current branch. (This is much easier than the other items, by comparison.)

• Add a field for the remote repository URL in the Config. The URL will need to be auto-discovered whenever the Config is instantiated.

Projects which use this feature would no longer build when exported from git (i.e., via git-archive or the like). They are fundamentally un-publishable.

In addition, the TOML requires that the git repository address be a URL. This means that the address will require an URL scheme : file:, ssh:, etc. This prevents cargo from working like git-submodule, which does not use schemes in relative paths (the scheme is inferred).

There are probably other issues here, but these are the ones I could find.

[cbs228]

Alternatively, maybe trying to re-invent git submodules in cargo is fighting too much inertia. Cargo really wants crates to be identified by URL. Rather than trying to infer a base URL from the environment, we could create an URL scheme for explicit indirection.

We could allow the user to create repository groups within .cargo/config.

[repogroups]

awesome = "ssh://git@somewhere.org/git"

and use like this within Cargo.toml

[dependencies]

baz = { git = "repogroup://awesome/project/baz.git" }

which would expand to "ssh://git@somewhere.org/git/project/baz.git".

I see this functionality as a lightweight alternative to custom registries. One can use this to make repository trees that can be relocated anywhere, and the "anywhere" is left to the discretion of the developer.

@fluffysquirrels: Any opinions?

[jplatte]

the TOML requires that the git repository address be a URL

@cbs228 What do you mean by this?

[cbs228]

@jplatte For unpublished crates, like git dependencies, Cargo strongly couples the notion of identity with URL. To identify a crate—that is, its name and version—it is also necessary to specify where absolutely the crate can be obtained. URLs must be absolute, and everyone must be able to retrieve the same URL.

This starts to become a problem when you have forks and/or mirrors.

It is difficult to [patch] your way out of this situation. There are a number of issues like #5478 or #6921 that make it more difficult to patch { git } sources.

The obvious workarounds are Git submodules and/or subtrees. There is also a way to use Git itself to replace URLs, which I described here.

I am looking to improve the user experience for no-shared-infrastructure distributed development. What's the best way to have an ecosystem of crates without a package server like crates.io? I'm not sure there's a good answer for this yet, but if there is one, I'd appreciate some enlightenment.

[lukaz-vaultree-com]

Nice to have!

[blueforesticarus]

Nice to have!

Absolutely necessary.

[blueforesticarus]

Also, what @cbs228 seems to be discussing seems a lot more complicated the basic version of this feature needs to be.

In general, anywhere a git repo can be used, it should be possible to use a local file.
It's just another place to clone from.

[epage]

Not caught up on all of this yet but SCP-like URLs were supported for git submodules in #12359.

[Alvenix]

It would be nice to have this. With this I would have been able to easily use an old version from an internal crate.

crate_name = { path = "../crate_name" }
crate_name_v1 = { package = "crate_name", git = "file://..", tag = "v1"}