Turn the new lock file format on by default

[alexcrichton]

This commit enables the support added in #7070 by default. This means
that gradually over time all Cargo.lock files will be migrated to the
new format. Cargo shipped with Rust 1.38.0 supports this new lock file
format, so any project using Rust 1.38.0 or above will converge quickly
onto the new lock file format and continue working.

The main benefit of the new format is to be more friendly to git merge
conflicts. Information is deduplicated throughout the lock file to avoid
verbose depedencies lists and the checksum data is all listed inline
with [[package]]. This has been deployed with rust-lang/rust for some
time now and it subjectively at least seems to have greatly reduced the
amount of bouncing that happens for touching Cargo.lock.

[rust-highfive]

r? @ehuss

(rust_highfive has picked a reviewer for you, use r? to override)

[ehuss]

So I imagine this will be problematic for projects that have extra long backwards compatibility. Is it correct that this is a forced upgrade for everyone?

What about having it so that it only applies to new projects for a while? That is, it only uses the new format if Cargo.lock is missing. That might allow more people to start using it now without disrupting projects with longer compatibility concerns.

[alexcrichton]

Yeah that's true that this would be an upgrade for everyone, but it notably only affects projects which span many toolchain versions and are also actively developed on all those versions. While that seems pretty rare to me, I think you've got a good idea to start out by just phasing in with new lock files, so I'll see if I can whip that up.

[alexcrichton]

Ok updated!

[ehuss]

@alexcrichton the macos test failure looks legitimate. I haven't been able to fully dig into it, but it looks like it is rebuilding when it shouldn't. I can maybe try to dig more tonight.

[ehuss]

@alexcrichton Here's a repro (any platform):

1. cargo new foo
2. Add a build.rs with fn main() {}
3. cargo build
4. cargo build again. The second one will rebuild when it should be fresh.

The issue is that the Cargo.lock is getting rewritten with an extra blank line. I believe this is because cargo new creates a v2 lock, then cargo build reverts it to v1 (because there are no dependencies) which adds the newline. The build script is necessary to make Cargo notice the timestamp change on Cargo.lock.

I'm a bit surprised this didn't trip up any other tests.

[alexcrichton]

Ok I've added a new test and pushed up what I believe should be a fix for this, but it's a pretty targeted fix and I don't have a ton of confidence in it. Next time we change formats I think we should do so via #7144 since that'll make this much easier to deal with.

[ehuss]

Seems good enough to me.

@bors r+

[bors]

📌 Commit c37a46c has been approved by ehuss

[bors]

⌛ Testing commit c37a46c with merge dba478b...

[bors]

☀️ Test successful - checks-azure
Approved by: ehuss
Pushing dba478b to master...

[mathstuf]

Is there some way to continue using the old format? I have CI testing -Z minimal-versions, but I test that configuration with my MSRV, which is older than 1.38, so some -Z flag to get the older format would be appreciated.

Can migrate to an issue if that is better.

[mathstuf]

The doccomment for this still says V1 is the default.

[ehuss]

#7637

[ehuss]

Is there some way to continue using the old format?

If you have the old format, it will continue to be used. If you deleted the lock file, you can use an older version of Cargo to recreate it.

[mathstuf]

No, I can't, because I generate the file with nightly to get -Z minimal-versions support. But, I want to test with my MSRV. Now the Cargo.lock is unusable for this setup.

[mathstuf]

I guess to clarify, this is a library, so there's no Cargo.lock committed.

[mathstuf]

And if a Cargo.lock is committed, I found it hard to justify testing -Z minimal-versions anyways; anyone using it isn't getting the version creep that -Z minimal-versions will help to avoid in the first place.

[ehuss]

Unfortunately we are recommending that people discontinue using minimal-versions, as we don't believe the design is going to work out.

[gwillen]

My group just hit this, and it seems like the error message that older cargo produces when reading a newer lockfile is particularly unhelpful -- I don't know that anything can be done about this now, but it doesn't seem great in terms of the debugging-time cost for people that encounter it. (Backwards-incompatible changes are easier to deal with if they at least produce obvious failures pointing to the need to upgrade.)

I can't imagine this is really fixable at this point, since the behavior of old cargo can't be changed, and the format can't be adjusted without making a second breaking change, but is there some way this could be improved for the future? I see that newer cargo understands the file to be implicitly versioned, and determines this by its contents. But older cargo of course doesn't understand this and just crashes. For future updates, would an explicit version field help avoid this, so that older cargo can declare that it does not understand the newer format?

(An approach I've seen before, which I think would work well here, would be a "minimum cargo version to parse this file" at the top of the file -- not the version that generated the file, but the first version that supports the format being used. Then older versions could not only error out helpfully, they could even point to what version is required to upgrade to.)

[ehuss]

@gwillen Discussion of a version in Cargo.lock is in #7144.
I think RFC 2495 would implement something along the lines of a minimum version declaration for Cargo.toml which would be an implicit minimum for Cargo.lock, though that would be opt-in, and is not yet implemented.

[kentfredric]

@mathstuf I have duct taped together some utterly terrible code, that will mangle a new style Cargo.lock contents back into an old style lock file in "Cargo.lock".

Its presented as-is in the hope its useful to somebody.

It may be a bit messy to read given how I've had to implement parts of a statemachine and various parser grammar in-line, so ... good luck extending it.

Regex ninja skills may be required, but all the regex is relatively simple, but in ways that mean anything weird I didn't see in my own lock files may not be dealt with at all.

The logic is doable in rust, but I wanted to spend less than an hour making something work.

Enjoy? Feel free to hate the perl.

( I had initially embarked on hand-transcribing a lockfile in a text editor before I decided it would be faster and more reliable to script it poorly )

#!perl
use strict;
use warnings;

# This is very crude.
# 
# 1. Read ./Cargo.lock trying to extract crate name/version/checksum/source details from 'new' format
# 2. Read Cargo.lock a second time, writing it to Cargo.lock.new streamed, but
#    a. strip checksum = from [[ package ]] entries on the way through
#    b. rewrite dependency = [ sections using data collected on the first pass
# 3. Then, generate a [metadata] section from the data obtained in the first pass and emit to the new file.
#
# Its up to you to copy the new file back into place.

my $states = parse_meta("Cargo.lock");
rewrite_file("Cargo.lock","Cargo.lock.new", $states);
1;

sub fix_dep {
    my ( $depstring, $states ) = @_;
    if ( $depstring =~ /^([^ ]+)$/ ) {

        # single map
        my $package = $1;
        if ( not exists $states->{$package} ) {
            die "$package not found";
        }
        my ( $first, @garbage ) = values %{ $states->{$package} };
        if (@garbage) {
            die "Too many versions for $package";
        }
        return
            $package . " "
          . $first->{version} . " ("
          . $first->{source} . ")";
    }
    if ( $depstring =~ /^([^ ]+)[ ]+([^ ]+)$/ ) {
        my ( $package, $version ) = ( $1, $2 );
        if ( not exists $states->{$package} ) {
            die "$package not found";
        }
        if ( not exists $states->{$package}->{$version} ) {
            die "$package v$version not found";
        }
        return
            $package . " "
          . $version . " ("
          . $states->{$package}->{$version}->{source} . ")";
    }
    die "Unhandled depstring $depstring";
}

sub parse_meta {
    my ($filename) = @_;
    open my $fh, "<", $filename or die "Can't read $filename";

    my $states = {};

  lines: while ( my $line = <$fh> ) {
        next unless $line =~ /^\[\[package\]\]/;
        my $record;

      fields: while ( my $line = <$fh> ) {
            last fields if $line =~ /^\s*$/;
            if ( $line =~ /^([^ ]+) = "([^"]+)"/ ) {
                my ( $field, $value ) = ( $1, $2 );
                $record->{$field} = $value;
            }
        }
        if ( not exists $record->{name} ) {
            die "Didn't parse record name";
        }
        if ( not exists $record->{version} ) {
            die "Didn't parse record version for $record->{name}";
        }
        if ( not exists $record->{source} ) {
            warn
"Didn't parse record source for $record->{name} $record->{version}";
        }
        if ( not exists $record->{checksum} ) {
            warn
"Didn't parse record checksum for $record->{name} $record->{version}";
        }
        $states->{ $record->{name} }->{ $record->{version} } = $record;
    }
    return $states;
}

sub rewrite_file {
    my ( $src, $dest, $states ) = @_;
    open my $fh,  "<", $src  or die "Can't read $src";
    open my $wfh, ">", $dest or die "Cant write $dest";

  lines: while ( my $line = <$fh> ) {
        if ( $line !~ /^\[\[package\]\]/ ) {
            $wfh->print($line);
            next lines;
        }
        $wfh->print($line);
      fields: while ( my $line = <$fh> ) {
            if ( $line =~ /^\s*$/ ) {
                $wfh->print($line);
                last fields;
            }

            if ( $line =~ /^([^ ]+) = "/ ) {
                if ( "$1" ne "checksum" ) {
                    $wfh->print($line);
                    next fields;
                }
            }
            if ( $line =~ /^dependencies = \[/ ) {
                $wfh->print($line);
                dep: while ( my $dep = <$fh> ) {
                    if ( $dep =~ /^\]/ ) {
                        $wfh->print($dep);
                        next fields;
                    }
                    if ( $dep =~ /^(\s+")([^"]+)(".+\z)/ms ) {
                        my ( $lpad, $content, $rpad ) = ( $1, $2, $3 );
                        $wfh->print(
                            $lpad . fix_dep( $content, $states ) . $rpad );
                        next dep;
                    }
                    die "Unhandled $dep";
                }
            }
        }
    }
    # write a metadata section
    $wfh->print("\n[metadata]\n");
    for my $crate ( sort keys %{$states} ) {
      for my $version ( sort keys %{$states->{$crate}} ) {
        next unless exists $states->{$crate}->{$version}->{checksum};
        $wfh->printf(qq["checksum %s %s (%s)" = "%s"\n],
          $crate,
          $version,
          $states->{$crate}->{$version}->{source},
          $states->{$crate}->{$version}->{checksum});
      }
    }
}

1;

[est31]

Btw, there is also the cargo-lock crate which can translate between formats: https://github.com/rustsec/cargo-lock