Check Cargo.lock in version control for libraries

[tiziano88]

Describe the problem you are trying to solve

The official FAQ and various other places strongly suggest (almost mandate) that libraries do not check in Cargo.lock. This serves so that if the library is used as a dependency from another crate, the parent crate should get to decide what particular versions of dependencies to use for itself and all its transitive dependencies, respecting versioning restrictions indicated in the relevant Cargo.toml files.

Describe the solution you'd like

• strongly suggest versioning Cargo.lock in version control even for libraries; without this, it is practically impossible to achieve reproducibility of builds, tests and CI pipelines: the same exact version (commit checksum) of a library may compile / pass at some point in time on someone's machine, but spuriously break / fail later on or on some other machine, just because some dependency published a new version on crates.io
• add an option to cargo to skip packaging the local Cargo.lock when publishing a library to crates.io (by default it will exclude it for libraries and include it for binaries)
  • perhaps it is sufficient to manually exclude Cargo.lock from a package until this is implemented
• suggest relying on https://dependabot.com/rust/ or manually bumping dependencies in order to detect incompatibilities with new version of dependent crates in an automated way that is tracked under version control and maintains the hermeticity and reproducibility of builds, even for libraries

Notes

[Eh2406]

Just to be clere a libraries Cargo.lock has no effect on resolution for a project that depends on that library. That is true if the library does not commit its Cargo.lock, and it is true if it commits it but does not include it when publishing, and it is true even if it ends up in the final published artifact. If you commit your Cargo.lock, so your CI is not broken by some new version on crates.io, then the first you will hear about the brokege is from your users.

Whether it is better to have flakey but real test in CI or to have reliable but mocked tests in CI, really depends on the priorities of the library. It is not so different from every other time a project considers using mock objects. Do you call that other microservice, then you will now fast if they broke their api but tests will be flakey, or do you mock it so you have fast reliable test of what that microservice used to respond.

[ehuss]

I think it would be fine to add some more details to the documentation. There's a tradeoff, if Cargo.lock is included in source control, you'll likely be testing out-of-date dependencies. That's good for reproducibility as you say, but is bad because you won't quickly catch problems with new dependencies. If you have the CI budget, you can of course test both with Cargo.lock, and with the latest versions (by calling cargo update && cargo test).

[tiziano88]

Catching problems with new dependencies should not happen when someone is trying to test an unrelated change. Which often results in a new dependency breaking CI, which prevents even other, unrelated changes from going through.

dependabot (or something similar) seems the ideal solution to me, since it guarantees deterministic and hermetic CI, while at the same time allowing testing against new dependencies, in their own PR, so that if there is any issue after the fact it is also easy to narrow it down to the specific commit that introduced the new dependency version.

cargo update && cargo test has again reproducibility issues, since the result of cargo update depends on the exact point in time at which it is run.

[epage]

Another reason to revisit this decision is interest in MSRV testing. Until we have a minimal versions resolver feature stablized, the lockfile is the easiest way to emulate it when verifying MSRV in libraries.

[demurgos]

I sent a PR to update the book and recommend always using a lock file: #12275

[epage]

Wanted to add on some more data points around MSRV.

We've seen some people misuse version requirements (rather than using a lock file) to make working with MSRV easier, including:

• Cargo is unable to resolve dependency versions #6584 (comment)
• https://www.reddit.com/r/rust/comments/p8clcx/how_to_fix_cargo_dependency_issue/

Apparently, the version requirement approach is common enough that its a top google search (haven't been able to find it myself)

Some quick Googling suggested that I could explicitly depend on half in my own Cargo.toml, and therefore choose a compatible version there;

See https://www.reddit.com/r/rust/comments/14khqdt/is_there_something_different_about_cargos/

And I've noticed that for winnow that 16 out of 26 dependents exist for MSRV version req hacks. That number would be higher except I've specifically reached out to some of the dependents and tried to steer them to doing things a different way.

[epage]

Motivations

• Until MSRV-dependent dependency version resolution #9930, there is not a viable way of having a fixed-length-lookback MSRV (X number of releases)
  • With a lockfile, even if the latest version of a dependency is bumps its MSRV, you can still verify that a version exists that matches your version requirement
  • You can set upper bounds on version requirements but that has issues (Check Cargo.lock in version control for libraries #8728 (comment))
  • You can only depend on packages with wider MSRVs but that requires cooperation or being very picky about dependencies. This can easily devolve into very long arguments (e.g. MSRV policy for libc crate libs-team#72,. MSRV policy is changing beginning 2023-07-01 time-rs/time#535)
• Green master (local and CI)
  • Master failing for local development or a PR's CI from an unrelated change is a bad contributor experience
    • The contributor assumes its their fault and looks into it, wasting their time. Hopefully they recognize its not their fault. If they don't, then they waste even more time figuring out how their change broke it
    • Maintainers need to drop everything and fix it to not make the experience worse for contributors
    • This is delayed by (detection_time + report_time + maintainer_availability + fix_time + review_time + ci_time_pr + ci_time_bors)
  • Different lockfiles between different contributors is easy to overlook when dealing with "it works on my machine"
  • Example: Before cargo had its own lockfile, snapshot tests would fail when human readable output from a dependency changed, causing CI to go red
  • Example: Before cargo had its own lockfile, CI would turn red if a new dependency was released that deprecated parts of it API (Reduce warning strictness #10742 was also considered for resolving it)
  • Example: if -Zdirect-minimal-versions (Dependencies resolution with --minimal-versions #5657) is merged and people verify it, then a dependency updating a shared dependency can cause a breakage
  • Example: Not being broken by dependencies that yank versions (see also internals)
• git bisect
  • Example: I had to bisect a problem with cargo but found no "root cause" commit. Most likely, the cause was in some dependency but I couldn't control for that
• Documents a "golden stack" for debugging problems
• If Cargo.lock represents minimal versions of dependencies (see also Dependencies resolution with --minimal-versions #5657), it can help developers relying on too-new APIs that will break when they get to CI
  • Ideally there would be a rustc or clippy lint to report API items with a #[stable] version that is newer than the minimal version that matches the dependency
• If you start out with a lib and later turn it into a workspace with a bin, you are likely to overlook adding a lockfile

Downsides with having a policy of checking-in lockfiles

• [lib]s CI jobs act as a distributed ecosystem-focused "crater" run. This puts peer pressure on the ecosystem for fixing bugs and yanking unintended breakages, helping maintain quality and adherence to semver.
  • Note that this is limited to [lib]-only repos that are following the guideline (e.g. clap checks in its lockfile) and only for releases that are semver compatible
• Testing the latest dependencies reflects what dependents will be doing
  • This is also true for workspaces with [[bin]] crates ("cargo install" apparently ignores "Cargo.lock" as opposed to "cargo build" #7169) but isn't covered in the policy
  • This is only sampling dependencies at specific times and does not reflect if a dependent is locked to a version that wasn't not covered in CI or newer than the last CI run. Even worse is bugs can be dependent on the specific versions of dependencies and dependents.
• To verify latest dependencies requires extra resources
• Libraries might be running malware in their CI
  • At least for Github, Dependabot seems to run for security vulnerabilities even if its not otherwise enabled
• Confusion over differences in lockfiles when dependents file bugs or contribute changes

Related questions:

• What should Cargo.lock represent as opposed to Cargo.lock.<topic> if used?
  • Possibilities
    • Optimal dev environment (latest for features, fixes)
    • New dependent (latest)
    • MSRV
  • Larger lockfile upgrades make bisecting dependency problems harder
  • The newer the lockfile versions are (closer to latest), the more dependency versions that can be captured when bisecting an end-users issue
  • Juggling Cargo.lock.<topic> files can be a pain especially if there are specific policy constraints on what they may contain (MSRV compatible, minimal version, latest versions)
    • Extra work is needed to verify these in CI
    • A contributor likely won't know something is wrong until CI, requiring an extra round of iteration which is frustrating
    • The contributor would then need to cycle between the different lock files and make sure they get verified in the way that specific lockfile needs or delegate that to CI for extra turnaround time

Potential routes

• MSRV-dependent dependency version resolution #9930 and Cargo time machine (generate lock files based on old registry state) #5221
  • Even with MSRV-dependent dependency version resolution #9930, a lockfile will still be needed for dependencies that don't specify an MSRV
• Recommend maintaining a Cargo.lock.msrv
  • A CI job copies it into place and does verification with --locked (to ensure it isn't stale)
  • High friction for adding or changing dependencies
• Change the Cargo.lock policy to check it in, updating cargo new to not exclude it in the .gitignore file
  • Recommend either staying up-to-date with dependencies (e.g. Dependabot or RenovateBot) or verify the latest dependencies always work with a job that runs cargo update before doing its verification (whether just cargo check or cargo test)
    • Automated dependency update allows people to choose their level of frequency, whether they want to be bleeding edge or play it safe and isolates the updates (and potential breakages) to PRs doing the update
    • The verification job should likely be a scheduled job to avoid making CI go red for contributors but Github will auto-disable the full Workflow for all Triggers if it is scheduled and there has been a lull in activity (2 months?), hurting projects that are passively maintained
  • Also recommend Lint to detect semantic versioning failures #374 once its available to reduce breaking of dependents
• (exclusively for MSRV) Abandon an MSRV policy
• (exclusively for MSRV) Change MSRV policy to be max-of-all-dependencies-at-current-moment
• (exclusively for MSRV) Limit dependencies to only those that have the same MSRV policy or wider

See also a zulip thread from earlier this year

[epage]

#12275 highlights the approach taken from other ecosystems

• Yarn for Node.js

  Which files should be gitignored?

  [...]

  • yarn.lock should always be stored within your repository (even if you develop a library).

• Poetry for Python

  As a library developer

  A simple way to avoid such a scenario [testing with latest transitive dependencies] is to omit the poetry.lock file. However, by doing so, you sacrifice reproducibility and performance to a certain extent. Without a lockfile, it can be difficult to find the reason for failing tests, because in addition to obvious code changes an unnoticed library update might be the culprit. [...]

  If you do not want to give up the reproducibility and performance benefits, consider a regular refresh of poetry.lock to stay up-to-date and reduce the risk of sudden breakage for users.

• Bundler for Ruby

  Q: Should I commit my Gemfile.lock when writing a gem?

  A: Yes, you should commit it. The presence of a Gemfile.lock in a gem’s repository ensures that a fresh checkout of the repository uses the exact same set of dependencies every time. We believe this makes repositories more friendly towards new and existing contributors. Ideally, anyone should be able to clone the repo, run bundle install, and have passing tests. If you don’t check in your Gemfile.lock, new contributors can get different versions of your dependencies, and run into failing tests that they don’t know how to fix.

[epage]

@zackw and I met during office hours to discuss their concerns over lockfiles.

This is going to be a rough recollection of the discussion that I'm hoping people find helpful.

There were bad experiences with lockfiles in other ecosystems

• "I downloaded a thing; it doesn't work. I deleted the lockfile; now it works"
  • For example, the compilers having broken compatibility and new dependency versions are needed to work with the current compiler versions
• Lockfiles affecting dependents in node
  • Someone locked a dependency on something that only used TLS 1.1 (which was no longer being supported by services)
• How do we build trust with people who have used similar looking techniques, even if our approach is different?

From this came up a concern: Would cargo eventually follow node and have lockfiles from dependencies affect dependents?

• I don't think this will happen. The role of a lockfile is for your current project and version requirements are for your dependents
• Unsure if any of this is codified. If not, I might codify it as part of https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Version.20Requirements.20documentation

Testing

• Testing a lockfile only tests a specific version
  • Minimum version requirements might be wrong
  • Bugs might have happened outside of the locked version that won' be caught
• Unfortunately, CI resources are limited (and we can't time travel to test future versions of dependencies), so the question is which combination of dependencies should be tested
• At this point, this seems orthogonal to lockfiles (checking in tests one set, not checking-in just tests a different set though it does evolve over time)
• If we make a change in our Cargo.lock recommendation, we'd likely also recommend in our CI section for people to have a CI job that first runs cargo update, so people test Cargo.lock + latest
  • If Dependencies resolution with --minimal-versions #5657 was stable, then that too

"Dependanbot"-like bots

• These cause extra work and most of it wouldn't be needed if lockfiles didn't exist (some of us would still want automation of upgrading across breaking releases)
• For me, I've found RenovateBot took away most of the churn
• I also see it beneficial to have PRs for updating the lockfile because I then get CI runs to verify the upgrade worked rather than unpredictable CI failures on unrelated PRs

"I add a dependency on foo and its failing. I go to contribute a fix to foo and I can't reproduce it"

• Yes, for first-time contributors not having a lockfile checked-in produces fewer chances of confusion
• However, if its your second time interacting with foo, you can have the opposite experience where your local Cargo.lock from last time can't reproduce a failure in CI for your PR
• Ends up being a trade off of which experience you focus on (along with all of the other trade offs)

Is checking-in a lockfile for MSRV just a hack?

• Mostly
• I started checking in my lockfiles because I found there was no way to maintain my MSRV policy without doing so
• Ideally MSRV-dependent dependency version resolution #9930 would make this need go-away but that is dependent on whether MSRV is set in dependencies
  • I'll also conservatively say its a year out from stable versions of Rust. Longer if we have problems getting good error messages

How is it with the experience with old, unmaintained Rust projects?

• For dependents, the lockfile doesn't matter
• For "CI running latest helps verify new dependencies", the point is moot because CI isn't being run
  • You could have scheduled jobs but at least for Github, they deactivate the entire workflow if it has a cron trigger
• For forking the project, I've just updated dependencies as part of the cleanup and it hasn't been a big deal

[safinaskar]

This fiasco is possibly related: rust-lang/rust#113152 (comment) . Nearly all my binary projects created nearly May 2023 (assuming they have Cargo.lock) don't build anymore. Removing Cargo.lock fixes them

[parasyte]

@safinaskar That breakage is unrelated to Cargo.lock; it is caused by a change to the nightly compiler. You can either build with the latest stable compiler (as mentioned in a response to your comment on that ticket) or, more appropriately perhaps, pin your nightly version with a toolchain file to avoid those kinds of breaking changes in the future.

[Nemo157]

(or try to apply pressure on crate authors to not silently opt their users into unstable breakage simply because they use a nightly compiler)

[epage]

I agree with @Nemo157 that that was a bug in proc-macro2 to use nightly features without a users consent (usually a unstable or nightly feature flag).

[ctz]

suggest relying on https://dependabot.com/rust/

Note that dependabot uses the existence of Cargo.lock as a heuristic that a crate is an application. The behaviour seems not configurable, and by default means a library-with-a-lockfile will churn dependency versions forwards in the Cargo.toml.

ref: https://github.com/dependabot/dependabot-core/blob/81ca7cb1d8d016583834c4a632d8b78daec934a5/cargo/lib/dependabot/cargo/update_checker.rb#L111-L115

[Nemo157]

This might be a good incentive to open an issue about having that configurable in dependabot, it's a big reason I don't use it on most of my repos.

[epage]

The primary reason I switched to RenovateBot was to make dependency updating faster (dropped my monthly updates from taking a couple days to part of a day) but the greater control was a close second. For example, here is the config for clap