-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Option to ignore missing workspace members when building #14566
Comments
Every member is relevant and contributes to the workspace's Similar feature requests have been made:
and #6179 is the major one tracking it. That being said, this docker use case is slightly different from others. Could you expand a bit about your workflow? Also wonder where the ignore file came from. |
Most of those conditional ones are for dealing with targets, cfgs, etc. This is related to docker caching from what I understoof of the text. Docker fingerpints the inputs to a layer and rebuilds if they change. If you have a layer that is meant for only a subset of your dependency tree, then you only want that fingerprinted. This is more similar to #2644 but in the workspace, rather than dealing with dependencies. https://crates.io/crates/cargo-chef is the primary place for docker layer caching experiments. I'd recommend seeing if that meets or your needs or if it could be made to meet your needs. |
Let's say we have a virtual workspace with just two binaries, members = [
"apps/server1",
"apps/server2",
] and the following file structure: project/
├── apps/
│ ├── server1/
│ │ ├── src/
│ │ │ └── main.rs
│ │ └── Cargo.toml
│ └── server2/
│ ├── src/
│ │ └── main.rs
│ └── Cargo.toml
├── config/
│ └── docker/
│ ├── dev/
│ └── prod/
│ ├── Server1.Dockerfile
│ ├── Server1.Dockerfile.dockerignore
│ ├── Server2.Dockerfile
│ └── Server2.Dockerfile.dockerignore
├── Cargo.toml
├── Cargo.lock
└── docker-compose.yaml Docker is used to build and run these binaries, the first workflow is for production and the second is for local development (both suffer from the same issue) To build for production the following is used: FROM rust:1.80.1-bullseye AS build
RUN --mount=type=bind,source=apps/server1,target=apps/server1,rw \
--mount=type=bind,source=Cargo.toml,target=Cargo.toml \
--mount=type=bind,source=Cargo.lock,target=Cargo.lock \
--mount=type=cache,target=target/ \
--mount=type=cache,target=/usr/local/cargo/registry/ \
<<EOF
set -e
cargo build -p server1 --locked --release <--- fails: "missing server2 member"
EOF
... File:
Attempting to build this with For the local development workflow, it's similar but with server1:
build:
context: .
dockerfile: config/docker/dev/Server1.Dockerfile
working_dir: /code
volumes:
- ./apps/server1:/code/apps/server1
- ./Cargo.toml:/code/Cargo.toml
- ./Cargo.lock:/code/Cargo.lock
- server1_cargo_target:/code/target/
- server1_cargo:/usr/local/cargo
ports:
- "8080:8080"
command: ["cargo", "watch", "-x", "run", "-p", "server1"] <--- Fails: missing server2 member Running
Ah caching isn't the concern for this, (we've long given up on getting caching to work correctly in CI 😅 at least for now) This came up because our current setup does not make use of cargo workspaces and instead treats each binary separately with their own Are there issues that we're not considering that would come out of building a specific binary in a workspace without the presence of other binaries? Our thinking was that this wouldn't cause issues since each binary is unrelated to the other.
Since caching isn't the issue for this we're a bit hesitant on taking on a dependency to solve this. As a workaround we're currently thinking of having a |
So this would also be a problem for caching iiuc, even if that isn't your problem.
Maybe I missed it but could you expand on why you don't mount the entire source? |
Hmm yes, most likely
This is a recommended docker best practice as far as I know, we have a monorepo which contains many unrelated things, android app, ios app, non-rust libraries, many rust binaries, etc. Our common workflow is to simply exclude everything and only include the project being built. From: https://docs.docker.com/build/concepts/context/#dockerignore-files
|
why u do not use thinks like? [workspace]
resolver = "2"
members = [
"apps/*",
"libs/*",
] If no folder exists, then there is no member, |
We have a monorepo which contains many unrelated things, android app, ios app, non-rust libraries, many rust binaries, etc.
This works when you have a simple flat structure and "rust only" apps/libs. It is much simpler and easier to list all members directly which is how |
I wonder if we should loosen this restriction. #4593 is the issue for this and #11405 is wider discussion on this feature.
If we had #4593, would this then be resolved? Would anything in #11405 help?
But if you use globs, then that isn't a problem anyways. |
If #4593 was resolved with the addition of #6009 to exclude cargo directories when needed then yes using the glob approach would work. But for us to consider that approach a "clean" solution, we'd then want a way to disable the default behavior of i.e, if we had
and ran
each time something is added, and so you'd now have to remember to go to the root This is what we're used to in other worlds like yarn workspaces (javascript) where a workspace such as: {
"private": true,
"workspaces": [
"apps/web_admin",
"doesnt_exist"
]
} can still be built even if a member isn't present. May I ask what is the rationale behind the restriction of refusing to build if a listed member is not present? i.e, isn't the outcome the expected one in either case? Is there something we're not considering? |
yes, this -> We have a monorepo which contains many unrelated things, android app, ios app, non-rust libraries, many rust binaries, etc. On the one hand, you wish to have a clean solution, Cargo is for rust and it's structures, and not there to handle other unrelated things outside of the scope of rust. When u mix different programming langugage tools/package managers/projects/etc it is all time a challenge, IDE: RustRover + IntelliJ Rust workspaces with rust libs/bins + Flutter (android/ios apps) + vcpgk + docker + github workflow can work smoothly together, Then u got a monorep with rust + dart + kotlin + c/c++ and perhaps some usefull python scripts, As more different tools or languages you have in use, as higher the difficulty factor grows up. My best solution for this is to reorganise your monorepo and perhaps to adjust many configs,
with RustRover the problem does not occur, which ide do you use? or are you using vim/neovim? |
I'm a bit unsure how else to describe this issue outside of what I've already described, this issue isn't about mixing different languages, the presence of non-rust code is just 1 additional reason which goes into the main issue being that one cannot build a rust package if any member is missing from the workspace, even if the missing member is not needed at all to build that package. To go back to the initial example where even if the project only contained rust with the following stripped down structure as described earlier:
the issue is still present, you cannot mount To workaround this you must then use glob patterns for your members, which then breaks as soon as you have a folder that doesn't contain a
VSCode I appreciate your time looking into this with me to come up with a solution, though, am I understanding correctly that your recommended solution to this is to forego groupings (nested structures), remove any folder without a
Although, as mentioned, this issue isn't about mixing different languages, I'd just like to note that this workflow didn't pose any challenge for yarn workspaces, a missing member doesn't cause the whole build to fail. Which brings me back to my question, @weihanglo or @epage would you be able to shed some light into the rationale behind the restriction of refusing to build if a listed member is not present even if the missing member is not needed for the build? Would this feature request of allowing the build to continue, as long as all required members are present, cause issues? |
This is a bug, please open an issue for this. We have a test for a simple glob but it seems other globs aren't working. Before cargo/tests/testsuite/cargo_new/add_members_to_workspace_with_members_glob/in/Cargo.toml Lines 1 to 3 in 01e1ab5
After cargo/tests/testsuite/cargo_new/add_members_to_workspace_with_members_glob/out/Cargo.toml Lines 1 to 3 in 01e1ab5
|
This comment has been minimized.
This comment has been minimized.
The answer to this was given earlier
Note that you said you were running with $ cargo build -p bin1 --locked --release The |
So I tried it again and you're right, it doesn't add the full path member when running
Before creating
the weird thing is that I deleted all files to try it again to be sure but this time it didn't do it, adding I tried it again a few more times but in the end I only got it to happen 1 more time after many tries. I'm still not sure why it only happened twice (well 3 times if counting the original one last week) but I'm concluding that it's a non issue since I doubt anyone would run into this realistically. |
Yeah we're on the same page for that, removing the Alright, then if I understand correctly the root issue with our desired setup is wanting to build our binaries in a container by only mounting the source of the binary (+ the source of its "internal" dependencies a.k.a libs). Which means that in the end even the glob pattern solution "apps/*" would still not work since even if we just had: project/
├── apps/
│ ├── server1/
│ └── server2/
├── Cargo.toml
└── Cargo.lock and used
even though now we would be able to run |
Did you update rust in that time frame? We might have had bugs around that that were fixed. |
Yes, thats correct. |
No updates in-between, happened twice in about 50 or so tries so it can be ignored imo.
Okay then we'll be sticking with the current setup of not using a workspace for the time being, thank you for your time! |
Problem
Given the following virtual workspace:
Building
bin1
withcargo build -p bin1
should be allowed even ifbin2
is missing from the file system.Use case: In docker when building a binary, only the relevant files are sent to the build context. For example, consider the following ignore file for building
bin1
The binary is then built with:
cargo build -p bin1 --locked --release
This fails with
error: failed to load manifest for workspace member /code/app/bin2
because the other binaries which are members of the workspace aren't present, but they are irrelevant to this build so should not be needed.
Proposed Solution
An option to prevent a build from failing because of a missing member.
This could be done either:
through a flag:
cargo build -p bin1 --ignore-missing-members
(with a better flag name)or directly in the manifest?
Notes
No response
The text was updated successfully, but these errors were encountered: