Add cargo pkgid for cargo-script

[Rustin170506]

Problem

ref #12207 (comment)

For now, if you try to use cargo pkgid against to a cargo script file. It will generate an error:

cargo pkgid sql2.rs
error: could not find `Cargo.toml` in `/Users/rustin/code/rs-scripts` or any parent directory

Proposed Solution

We need to support cargo pkgid for cargo-script.

For instance, for itself, it would look like:

$pwd
/Users/rustin/code/rs-scripts

$ls
sql.rs  sql1.rs sql2.rs sql3.rs sql4.rs sql5.rs sql6.rs sql7.rs sql8.rs sql9.rs

$cargo pkgid sql2.rs
path+file:///Users/rustin/code/rs-scripts/sql2.rs#embedded

For its dependencies, it would look like:

cargo pkgid sql2.rs -p clap

https://github.com/rust-lang/crates.io-index#clap:4.5.21

Notes

Not sure specifying the rust file as an argument is a good idea. Because cargo pkgid itself already takes the SPEC as the argument. So it might be a very confusing syntax.

[weihanglo]

I've tried thinking it hard but not sure about the use case of getting pkgid from a cargo script. Mind sharing more on your thought?

[epage]

This is less about the specific command and more about all of the places PackageIdSpec is exposed to the user and making sure we define something that we want to stabilize rather than accidently leaking the current implementation and being stuck.

[weihanglo]

Regarding that, I feel like this should not block the stabilization. Unless there is a clear use case we want to support, I'd suggest we make sure pkgid not leaking when stabilization.
Adding an explicit unsupported statement is also helpful, since we can mark it as a bug if we any unexpected stuff is exposed.

[Rustin170506]

Regarding that, I feel like this should not block the stabilization. Unless there is a clear use case we want to support, I'd suggest we make sure pkgid not leaking when stabilization.

I think a reasonable use case is that we don't include the cargo.lock file under the script file dir. So there is no good way to know which version your dependency is using. So can we just get it with this command and use that specification?

[epage]

@weihanglo package ids come up in

• cargo pkgid as an output
  • We could make this command unstable for cargo-script, see also --package
• cargo metadata as an output (fix(metadata): Stabilize id format as PackageIDSpec  #12914)
  • Taking the stabilized output and making it unstable with certain stable conditions seems like its prone to problems
• cargo <cmd> --message-format json as an output (fix(json-msg): use pkgid spec in in JSON messages #13311)
  • Same as cargo metadata
• cargo <cmd> --package as an input
  • While we could make this flag unstable for cargo-script, this comes up in programmatic use cases to reduce the chance of ambiguity due to either virtual manifests or dependencies with the same name as a workspace member (usually from a dependency depending on the registry version of your package)

I don't see it being practical to skip this step to stabilizing this feature.

[epage]

$ cargo pkgid sql2.rs
path+file:///Users/rustin/code/rs-scripts/sql2.rs#embedded

My hope is we can get away without a # or ? for this. It puts it completely on the Source to then disambiguate whether sql2.rs is a file or a directory.

$ cargo pkgid sql2.rs -p clap

https://github.com/rust-lang/crates.io-index#clap:4.5.21

One problem with this is it is inferring --manifest-path off of spec. That is a very sketchy thing to be doing. If we remove that, this becomes

$ cargo pkgid --manifest-path sql2.rs sql2.rs -p clap

https://github.com/rust-lang/crates.io-index#clap:4.5.21

At which point, the value of using a cargo-script file in a <spec> is greatly diminished because a cargo-script's workspace can only have one member.

Having a short-hand for PackageidSpec is also not required for the first release of cargo script.

I would suggest we defer out a short-hand until someone requests it of us with a use case.

[weihanglo]

I don't see it being practical to skip this step to stabilizing this feature.

By practical, did you mean from a technical perspective it's hard to make them out of the stabilization?
I don't object to treating them stabilization blockers, as I am fine with them being unstable for a few more months. It was just these usages don't look like worth waiting for six more months. The current feature set (without pkgid/fmt/clippy) of -Zscript has covered 80% of its use cases. I have no data though.

While we could make this flag unstable for cargo-script, this comes up in programmatic use cases to reduce the chance of ambiguity due to either virtual manifests or dependencies with the same name as a workspace member (usually from a dependency depending on the registry version of your package)

Could you share a practical use case of it? Anyway, I feel like a rename of the script can fix 90% of issues like this.

[epage]

Could you share a practical use case of it? Anyway, I feel like a rename of the script can fix 90% of issues like this.

To clarify, I'm not saying that using --package is needed for cargo-scripts but that some third-party tools are likely to always pass in --package because of confusion within the workspace. If we then take one of these tools and run them on a cargo-script, they will fail and either the user has to live with that or the maintainer has to detect a cargo-script is being used and special case it.

I don't know if I can name a specific concrete example of one that does this today but that I've personally run into enough --package problems by-hand and programmatically, that I view it as nearly essential for programmatic use.

[weihanglo]

Having a short-hand for PackageidSpec is also not required for the first release ofcargo script.

Agree we shouldn't bother short-hand at this moment.

One thing may be related to pkgid is executing a remote script. Something like cargo https://github.com/rust-lang/cargo/path/to/script, where the URL may be a representation of pkgid. Personally I would defer any kind of remote script execution, or even just ban this feature. Whoever want to execute a remote script should use curl + cargo combination, such as

curl --proto '=https' --tlsv1.2 -sSf https://example.com/cargo-script.rs | cargo

And if we ever want to support this combo, what does its pkgid look like?

[epage]

By practical, did you mean from a technical perspective it's hard to make them out of the stabilization?

Yes.

The current feature set (without pkgid/fmt/clippy) of -Zscript has covered 80% of its use cases. I have no data though.

The stated goal in the RFC is for first-class support. If we wanted something that half-way worked, we could have kept to cargo-script.

I personally am regularly annoyed with the lack of cargo fmt support in each of my throw-away scripts I write. I share these in bug reports and don't want to manually clean up the code for them to be presentable to the user. I've also heard of lots of reports of people who want to replace their long-lived bash and python scripts with Cargo scripts which entails wanting standard tooling compared to a throw-away script.

Granted, how much we have stable in the first release is open for discussion. That depends on what level of quality people will expect from it being stable. However, as I said before, I don't see a way for us to have the package id spec format for cargo-scripts to be unstable.

[epage]

One thing may be related to pkgid is executing a remote script. Something like cargo https://github.com/rust-lang/cargo/path/to/script, where the URL may be a representation of pkgid. Personally I would defer any kind of remote script execution, or even just ban this feature. Whoever want to execute a remote script should use curl + cargo combination, such as

curl --proto '=https' --tlsv1.2 -sSf https://example.com/cargo-script.rs | cargo

And if we ever want to support this combo, what does its pkgid look like?

We don't support cargo run --manifest-path <url>, so I don't see this as any different. That is a whole orthogonal feature that would need to be developed.

[weihanglo]

We don't support cargo run --manifest-path , so I don't see this as any different. That is a whole orthogonal feature that would need to be developed.

That is true. We may need a complete new source kind for it, not just for -Zscript. Thanks for pointing it out!

The stated goal in the RFC is for first-class support. If we wanted something that half-way worked, we could have kept to cargo-script.
…

It is indeed important to hear your voice as a user! You just convinced me that their supports is fairly essential. cargo-fmt/cargo-clippy support is out of scope of this issue for sure though.

I still think it is not too hard to just ban cargo metadata and --message-format json when cargo script is involved at the CLI level, as the script is always the root (primary) package. But anyway, I should focus on adding the support first. If we can't make it then we discuss different stabilization plans.

[weihanglo]

$ cargo pkgid sql2.rs
path+file:///Users/rustin/code/rs-scripts/sql2.rs#embedded

My hope is we can get away without a # or ? for this. It puts it completely on the Source to then disambiguate whether sql2.rs is a file or a directory.

If we don't put any # or ?, we then need to add a new protocol I guess. Otherwise it is not sufficient to disambiguate a foo script at a workspace root of a package foo@0.0.0.

$ cargo +nightly -Zscript metadata --manifest-path foo --format-version 1 | jq -r .resolve.root
warning: `package.edition` is unspecified, defaulting to `2021`
path+file:///Users/whlo/dev/dirty/foo#0.0.0
$ cargo +nightly metadata --manifest-path Cargo.toml --format-version 1 | jq -r .resolve.root
path+file:///Users/whlo/dev/dirty/foo#0.0.0

Note that ? is not allowed in a URI of file protocol.

[epage]

I still think it is not too hard to just ban cargo metadata and --message-format json when cargo script is involved at the CLI level, as the script is always the root (primary) package. But anyway, I should focus on adding the support first. If we can't make it then we discuss different stabilization plans.

A caller may want access to those features without wanting the package id field. A lot of custom commands (first or third-party) are built on top of cargo metadata. What problem they would have is that conditionally removing the package id field would break them. Technically, its a feature being used in a new way, so we could say its on them, but the two sides to this are decoupled enough that I think thats an unfair approach to take with tool maintainers.

[epage]

If we don't put any # or ?, we then need to add a new protocol I guess. Otherwise it is not sufficient to disambiguate a foo script at a workspace root of a package foo@0.0.0.

$ cargo +nightly -Zscript metadata --manifest-path foo --format-version 1 | jq -r .resolve.root
warning: `package.edition` is unspecified, defaulting to `2021`
path+file:///Users/whlo/dev/dirty/foo#0.0.0
$ cargo +nightly metadata --manifest-path Cargo.toml --format-version 1 | jq -r .resolve.root
path+file:///Users/whlo/dev/dirty/foo#0.0.0

I think some context is needed for this. Was that inside of /Users/whlo/dev/dirty/foo with foo and Cargo.toml files? If so, then that is the motivating case for this issue, rather than a note in the tracking issue for us to look at in stabilizing this.

Changing the protocol isn't the only solution. As I proposed in #14831 (comment), the output would be

$ cargo +nightly -Zscript metadata --manifest-path foo --format-version 1 | jq -r .resolve.root
warning: `package.edition` is unspecified, defaulting to `2021`
path+file:///Users/whlo/dev/dirty/foo/foo#0.0.0
$ cargo +nightly metadata --manifest-path Cargo.toml --format-version 1 | jq -r .resolve.root
path+file:///Users/whlo/dev/dirty/foo#0.0.0

Note that ? is not allowed in a URI of file protocol.

Oof, that limits our options.

[weihanglo]

Was that inside of /Users/whlo/dev/dirty/foo with foo and Cargo.toml files?

Yes. And I didn't realize your proposal is that. Thanks for the clarification. That seems probably the most simple way to extend both the spec and the implementation.

[Rustin170506]

$ cargo pkgid sql2.rs
path+file:///Users/rustin/code/rs-scripts/sql2.rs#embedded

My hope is we can get away without a # or ? for this. It puts it completely on the Source to then disambiguate whether sql2.rs is a file or a directory.

Did you mean by:

$ cargo pkgid sql2.rs
path+file:///Users/rustin/code/rs-scripts/sql2.rs

[weihanglo]

@Rustin170506
I believe so. The second code block in #14831 (comment) shows an example.

[epage]

@Rustin170506 if you are referring to #14831 (comment), I was quoting an example from you and replying to it, not intending to say that was what I was suggesting. Sorry for the confusion!

[epage]

Sounds like we generally agree on a direction:

For now cargo scripts are only relevant for Path sources. Cargo scripts should not work as dependencies and we should make sure we have a test for it. The test should likely be using artifact deps since we only support bin cargo scripts and we warn people away from depending on a bit without artifact deps.

Currently, path source ids look like

path+file:///home/epage/src/personal/cargo
path+file:///home/epage/src/personal/cargo#0.86.0

Where Cargo.toml is appended to the path.

The proposal is to support

path+file:///home/epage/src/personal/cargo/foo.rs
path+file:///home/epage/src/personal/cargo/foo.rs#0.86.0

As we support scripts without an extension, that does make the following ambiguous

path+file:///home/epage/src/personal/cargo
path+file:///home/epage/src/personal/cargo#0.86.0

I'm presuming that most of Cargo won't care and when we do care (in the source, for use cases like deps, cargo install --path), we can check whether it resolves to a file or directory to know whether to append Cargo.toml

When we support registry sources, I expect it will be transformed into a regular package and nothing special will be needed. Git sources are interesting because you can't specify a location within a git repo and we have no way to know what all files may be cargo scripts. Walking of workspace members may help when we support those.

As this is unstable, we can leave the full team decision to stabilization. We'll want to note the design in the tracking issue so it can be reviewed when we stabilize.