Allow rust-project.json to include arbitrary shell commands for runnables #16840

Wilfred · 2024-03-15T00:00:02Z

This is a follow-up on #16135, resolving the feedback raised :)

Allow rust-project.json to include shell runnables, of the form:

{
  "build_info": {
    "label": "//project/foo:my-crate",
    "target_kind": "bin",
    "shell_runnables": [
      {
        "kind": "run",
        "program": "buck2",
        "args": ["run", "//project/foo:my-crate"]
      },
      {
        "kind": "test_one",
        "program": "test_runner",
        "args": ["--name=$$TEST_NAME$$"]
      }
    ]
  }
}

If these runnable configs are present for the current crate in rust-project.json, offer them as runnables in VS Code.

This PR required some boring changes to APIs that previously only handled cargo situations. I've split out these changes as commits labelled 'refactor', so it's easy to see the interesting changes.

bors · 2024-03-15T13:30:08Z

☔ The latest upstream changes (presumably #16845) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2024-03-18T09:18:43Z

☔ The latest upstream changes (presumably #16868) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2024-03-21T11:50:30Z

☔ The latest upstream changes (presumably #16895) made this pull request unmergeable. Please resolve the merge conflicts.

davidbarsky · 2024-03-19T01:47:27Z

crates/project-model/src/project_json.rs

@@ -87,6 +87,21 @@ pub struct Crate {
    pub(crate) exclude: Vec<AbsPathBuf>,
    pub(crate) is_proc_macro: bool,
    pub(crate) repository: Option<String>,
+    pub build_info: Option<BuildInfo>,


i meant to change this in the original PR, but build_info's runnables should be under ProjectJson, not the Crate. label/target_kind should stay under Crate, however.

(that assumes a Bazel/Buck-style labeling system, but even I think for non-Cargo build systems, that's a good enough bar to set.)

@davidbarsky If I'm reading this right, you want the runnables to be stored at the root of the project.json. That will require:

a substitution mechanism like $label so that the "runnable templates" can be turned into real commands for a specific crate

probably RA using the build_info.kind field to figure out which runnables to offer for a given target

I also think we also want custom runnables in build_info for specific crates anyway. Right? Because the build system might simply know better. Substitution is nice because it saves a few MB of JSON. But ultimately you want the ability to add more to specific crates.

Because I believe you're going to want runnables per-crate anyway, and they are completely capable of handling anything substitution can, I think we can merge this PR pretty much as-is. I think the substitution mechanism is more of an optimisation than anything else.

I am happy to put together a PR that allows flycheck to use these.

@cormacrelf

a substitution mechanism like $label so that the "runnable templates" can be turned into real commands for a specific crate

Yup: the substitution mechanism is thankfully not too complex: I think it boils down to just a str::replace.

(If you squint, I'd argue that the substitution already exists in rust-analyzer today via Cargo, but the substitution knowledge is pretty implicit...)

probably RA using the build_info.kind field to figure out which runnables to offer for a given target

Will need to repage some context in, but I'm pretty sure the answer is also "yup".

I also think we also want custom runnables in build_info for specific crates anyway. Right? Because the build system might simply know better. Substitution is nice because it saves a few MB of JSON. But ultimately you want the ability to add more to specific crates.

I don't know if I agree: while I'm assuming that while there is a smart build system, I don't think there isn't much surface area for more specialized commands for individual crates. There's also the fact that rust-analyzer is pretty darn fast even compared to buck2, which is the fastest build system I'm aware of—single digit milliseconds versus ~300 milliseconds. There's also the matter of the duplication being somewhat concerning for the Fuschia folks, as their rust-project.json is already pretty large and I'm not really inlined to make things more complicated for them, however slightly.

Because I believe you're going to want runnables per-crate anyway, and they are completely capable of handling anything substitution can, I think we can merge this PR pretty much as-is. I think the substitution mechanism is more of an optimisation than anything else.

You are right it's an optimization, but it's a pretty cheap one. Wilfred and I need to get this over the finish line, but it's genuinely not too much work to implement it.

OK. A good metric for the design is whether rust-project.json can encode all the cargo-related logic. If you set build_info.label to a cargo crate name, and a root level fly check template of cargo check -p $label --message-format=json, then we almost have that mission in the bag. The main missing piece is flycheck for a crate vs for the whole workspace, which is configurable through check.workspace in RA. Having two more runnable kinds, flycheck and flycheckWorkspace, should do it.

OK. A good metric for the design is whether rust-project.json can encode all the cargo-related logic. If you set build_info.label to a cargo crate name, and a root level fly check template of cargo check -p $label --message-format=json, then we almost have that mission in the bag.

Yeah, that's generally the stance I've pursuing.

The main missing piece is flycheck for a crate vs for the whole workspace, which is configurable through check.workspace in RA. Having two more runnable kinds, flycheck and flycheckWorkspace, should do it.

hmm, probably. i don't know if i'd want to set a flycheckWorkspace for rust-project, but I guess it can be a buck2 build 'rdeps(:target, 1)'. at least, i don't know if i'd put it on by default.

(A blessing and a curse of Buck is that there's no clearly delineated "workspace" concept like there is in Cargo.)

I wouldn't set it either.

nemethf · 2024-03-27T16:37:41Z

docs/dev/lsp-extensions.md

+
+```typescript
+{
+    kind: string;


How should the LSP client use the kind field? Does it describe what the runnable does? Then how does it differ then from label field of the parent Runnable?

Also should the "shell" kind have a expectTest field just like the "cargo" kind?

BTW, how should the LSP client interpret the workspaceRoot, expectTest, and overrideCargo fields?

Additionally, I think it would make sense to copy the example from the Pull Request into the lsp-extension.md file, or just reference a user-level documentation describing this feature.

Thanks

Also, shouldn't this PR explain the new build_info field of rust-project.json in manual.adoc?

When expectTest is set the client should set the UPDATE_EXPECT env var to 1 when invoking the runnable. ideally there should be a field for arbitrary env var overrides in the payload, then this field could be replaced by the server instructing the client to just set the env var with that general mechanism as we'll need this for other things as well.

overrideCargo I think should just replace the cargo command that is being used by the client.

bors · 2024-03-29T02:21:10Z

☔ The latest upstream changes (presumably #16971) made this pull request unmergeable. Please resolve the merge conflicts.

Veykril · 2024-04-18T12:13:13Z

This is a follow-up on #16135, resolving the feedback raised :)

So this means #16135 can be closed in favor of this one?

davidbarsky · 2024-04-18T12:41:27Z

This is a follow-up on #16135, resolving the feedback raised :)

So this means #16135 can be closed in favor of this one?

Yup! Just closed. Wilfred or I will push to this branch as needed.

Veykril

Generally looks good to me (aside from what has been raised already by others)

bors · 2024-06-06T10:35:37Z

✌️ @davidbarsky, you can now approve this pull request!

If @Veykril told you to "r=me" after making some further change, please make that change, then do @bors r=@Veykril

davidbarsky · 2024-06-06T13:43:46Z

@cormacrelf here's the commit in rust-project: facebook/buck2@7169723.

@petr-tik yes, I think that bazelbuild/rules_rust would be able to take advantage of this feature once this lands.

bors · 2024-06-07T11:02:04Z

☔ The latest upstream changes (presumably #17058) made this pull request unmergeable. Please resolve the merge conflicts.

Wilfred · 2024-06-10T23:20:38Z

crates/project-model/src/project_json.rs

-/// Additional metadata about a crate, used to configure runnables.
+/// Additional, build-specific data about a crate.
+///
+/// This is primarily used to


Unfinished.

Wilfred · 2024-06-10T23:20:55Z

crates/project-model/src/project_json.rs

-    /// a check build or a test run.
+    /// Configuration for commands.
+    ///
+    /// Examples include a check build or a test run.


I'd probably explicitly say this is CLI invocations.

Wilfred · 2024-06-10T23:21:35Z

crates/project-model/src/project_json.rs

-    /// to offer a 'run' button for library crates.
+    /// The kind of target.
+    ///
+    /// Examples (non-exhaustively) include [TargetKind::Lib], [TargetKind::Lib],


Lib twice here.

Wilfred · 2024-06-10T23:22:09Z

crates/project-model/src/project_json.rs

+/// A template for runnables.
+///
+/// This to create runnables for running/debugging binaries and tests.
+/// In the future, it can be used to setup Flycheck.


I'd focus on what it can do today, rather than future work.

Wilfred · 2024-06-10T23:22:34Z

crates/project-model/src/project_json.rs

+///          "{test_id}",
+///          "--print-passing-details"
+///     ],
+///     "cwd": "/Users/dbarsky/fbsource",


How about something more generically named here? Buck conventions aren't super useful.

whoops, i thought i removed this! I'll make this something like /home/user/repo-root/.

davidbarsky · 2024-06-10T23:29:03Z

crates/project-model/src/project_json.rs

+///          "{test_id}",
+///          "--print-passing-details"
+///     ],
+///     "cwd": "/Users/dbarsky/fbsource",


whoops, i thought i removed this! I'll make this something like /home/user/repo-root/.

lnicola · 2024-06-11T14:02:16Z

crates/project-model/src/project_json.rs

+    /// The kind of target.
+    ///
+    /// Examples (non-exhaustively) include [TargetKind::Bin], [TargetKind::Lib],
+    /// and [TargetKind::Test]. This information is used to determine what sort


what sort of

lnicola · 2024-06-11T14:04:15Z

crates/project-model/src/project_json.rs

+    /// the label might be something like `//ide/rust/rust-analyzer:rust-analyzer`.
+    ///
+    /// Do not attempt to parse the contents of this string; it is a build system-specific
+    /// identifier similar to [Crate::display_name].


Nit: Crate::display_name will only show up as monospace in the docs if you wrap it in backquotes.

thanks! fixed.

lnicola · 2024-06-11T14:25:47Z

crates/project-model/src/project_json.rs

+    pub label: String,
+    /// Path corresponding to the build system-specific file defining the crate.
+    ///
+    /// It is roughly analogous to [ManifestPath], but none of the indexing/loading


Suggested change

/// It is roughly analogous to [ManifestPath], but none of the indexing/loading

/// It is roughly analogous to [`ManifestPath`], but none of the indexing/loading

lnicola · 2024-06-11T14:26:10Z

crates/project-model/src/project_json.rs

+/// # Example
+///
+/// Below is an example of a test runnable. `{label}` and `{test_id}`
+/// are explained in [Runnable::args]'s documentation.


Suggested change

/// are explained in [Runnable::args]'s documentation.

/// are explained in the [`Runnable::args`] documentation.

lnicola · 2024-06-11T14:26:31Z

crates/project-model/src/project_json.rs

+    ///
+    /// For example, this might be `cargo`, `buck`, or `bazel`.
+    pub program: String,
+    /// The arguments passed to [Runnable::program].


Since you've changed one..

Suggested change

/// The arguments passed to [Runnable::program].

/// The arguments passed to [`Runnable::program`].

Veykril · 2024-06-11T15:15:13Z

crates/project-model/src/project_json.rs

+    /// It is roughly analogous to [`ManifestPath`], but none of the indexing/loading
+    /// operations can be directly applied/used with this file.


I am not sure I can tell what indexing/loading operations are in this context

it's in reference to ProjectManifest::from_manifest_file. you can't run cargo metadata on a rust-project.json or Starlark-ified TARGETS file.

(Arguably, it should, but that's more of a discussion for #17246. I'll clarify this comment.)

davidbarsky · 2024-06-11T16:12:02Z

@bors r=@Veykril

bors · 2024-06-11T16:12:05Z

📌 Commit 71a78a9 has been approved by Veykril

It is now in the queue for this repository.

bors · 2024-06-11T16:13:14Z

⌛ Testing commit 71a78a9 with merge 4af21ff...

bors · 2024-06-11T16:25:51Z

☀️ Test successful - checks-actions
Approved by: Veykril
Pushing 4af21ff to master...

gvozdvmozgu · 2024-06-12T06:42:34Z

It seems that after this PR, the tests have stopped running.

 *  Executing task: cargo cargo test --package sqruff-lib --lib -- rules::layout::LT13::tests::test_fail_leading_whitespace_comment --exact --show-output 

error: no such command: `cargo`

        View all installed commands with `cargo --list`
        Find a package to install `cargo` with `cargo search cargo-cargo`

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 15, 2024

Wilfred mentioned this pull request Mar 15, 2024

Refactor extension to support arbitrary shell command runnables #16839

Merged

Wilfred force-pushed the shell_runnable branch 5 times, most recently from 65e9be3 to c7bdd91 Compare March 15, 2024 23:14

Wilfred marked this pull request as ready for review March 15, 2024 23:14

Wilfred force-pushed the shell_runnable branch from c7bdd91 to f88aff3 Compare March 19, 2024 19:14

davidbarsky requested changes Mar 21, 2024

View reviewed changes

nemethf mentioned this pull request Mar 27, 2024

Announcements: Client Changes #4604

Open

nemethf reviewed Mar 27, 2024

View reviewed changes

Wilfred force-pushed the shell_runnable branch 2 times, most recently from 125b58b to b9662e6 Compare March 28, 2024 00:04

Veykril approved these changes Apr 18, 2024

View reviewed changes

Veykril added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 18, 2024

davidbarsky mentioned this pull request Apr 23, 2024

WIP: Add support for detecting tests in source files, and implement it for Rust zed-industries/zed#10873

Closed

4 tasks

Veykril mentioned this pull request May 23, 2024

Fix inconsistent cwd of run and debug command in client #17275

Merged

davidbarsky force-pushed the shell_runnable branch 5 times, most recently from 0d0ef0a to 660fcbf Compare June 5, 2024 14:22

davidbarsky force-pushed the shell_runnable branch 2 times, most recently from c062791 to 0dc68a9 Compare June 10, 2024 17:52

Wilfred commented Jun 10, 2024

View reviewed changes

davidbarsky approved these changes Jun 10, 2024

View reviewed changes

davidbarsky force-pushed the shell_runnable branch 2 times, most recently from 877c39d to 48a2435 Compare June 10, 2024 23:32

lnicola reviewed Jun 11, 2024

View reviewed changes

davidbarsky force-pushed the shell_runnable branch from 48a2435 to e1c1bb1 Compare June 11, 2024 14:10

lnicola reviewed Jun 11, 2024

View reviewed changes

davidbarsky force-pushed the shell_runnable branch from e1c1bb1 to a64c2b0 Compare June 11, 2024 14:57

Veykril reviewed Jun 11, 2024

View reviewed changes

davidbarsky force-pushed the shell_runnable branch from a64c2b0 to 2b47745 Compare June 11, 2024 15:44

feature: add build system info; runnables to rust-project.json

71a78a9

davidbarsky force-pushed the shell_runnable branch from 2b47745 to 71a78a9 Compare June 11, 2024 15:55

bors merged commit 4af21ff into rust-lang:master Jun 11, 2024
11 checks passed

Veykril mentioned this pull request Jun 12, 2024

cargo runnables no longer work #17402

Closed

philips-software-forest-releaser bot mentioned this pull request Jun 18, 2024

chore(deps,rust): update rust-lang.rust-analyzer philips-software/amp-devcontainer#456

Merged

philips-software-forest-releaser bot mentioned this pull request Jun 25, 2024

chore(deps,rust): update rust-lang.rust-analyzer philips-software/amp-devcontainer#469

Merged

Wilfred deleted the shell_runnable branch June 25, 2024 22:02

	/// It is roughly analogous to [ManifestPath], but none of the indexing/loading
	/// It is roughly analogous to [`ManifestPath`], but none of the indexing/loading

	/// are explained in [Runnable::args]'s documentation.
	/// are explained in the [`Runnable::args`] documentation.

	/// The arguments passed to [Runnable::program].
	/// The arguments passed to [`Runnable::program`].

		/// It is roughly analogous to [`ManifestPath`], but none of the indexing/loading
		/// operations can be directly applied/used with this file.

Allow rust-project.json to include arbitrary shell commands for runnables #16840

Allow rust-project.json to include arbitrary shell commands for runnables #16840

Conversation

Wilfred commented Mar 15, 2024 • edited Loading

bors commented Mar 15, 2024

bors commented Mar 18, 2024

bors commented Mar 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bors commented Mar 29, 2024

Veykril commented Apr 18, 2024

davidbarsky commented Apr 18, 2024

Veykril left a comment

Choose a reason for hiding this comment

bors commented Jun 6, 2024

davidbarsky commented Jun 6, 2024

bors commented Jun 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidbarsky Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidbarsky Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

davidbarsky commented Jun 11, 2024

bors commented Jun 11, 2024

bors commented Jun 11, 2024

bors commented Jun 11, 2024

gvozdvmozgu commented Jun 12, 2024

Wilfred commented Mar 15, 2024 •

edited

Loading

davidbarsky Jun 11, 2024 •

edited

Loading

davidbarsky Jun 11, 2024 •

edited

Loading