Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement the Cargo half of pipelined compilation #6864

Closed
wants to merge 2 commits into from
Closed

Implement the Cargo half of pipelined compilation #6864

wants to merge 2 commits into from

Commits on Apr 25, 2019

  1. Start preparing Cargo for pipelined rustc compilation 

    This commit starts to lay the groundwork for rust-lang#6660 where Cargo will
invoke rustc in a "pipelined" fashion. The goal here is to execute one
command to produce both an `*.rmeta` file as well as an `*.rlib` file
for candidate compilations. In that case if another rlib depends on that
compilation, then it can start as soon as the `*.rmeta` is ready and not
have to wait for the `*.rlib` compilation.

The major refactoring in this commit is to add a new form of
`CompileMode`: `BuildRmeta`. This mode is introduced to represent that a
dependency edge only depends on the metadata of a compilation rather
than the the entire linked artifact. After this is introduced the next
major change is to actually hook this up into the dependency graph.

The approach taken by this commit is to have a postprocessing pass over
the dependency graph. After we build a map of all dependencies between
units a "pipelining" pass runs and actually introduces the `BuildRmeta`
mode. This also makes it trivial to disable/enable pipelining which
we'll probably want to do for a preview period at least! The
`pipeline_compilations` function is intended to be extensively
documented with the graph that it creates as well as how it works in
terms of adding `BuildRmeta` nodes into the dependency graph.

This commit is not all that will be required for pieplining
compilations. It does, however, get the entire test suite passing with
this refactoring. The way this works is by ensuring that a pipelined
unit, one split from `Build` into both `Build` and `BuildRmeta`, to be a
unit that doesn't actually do any work. That way the `BuildRmeta`
actually does all the work currently and we should have a working Cargo
like we did before. Subsequent commits will work in updating the
`JobQueue` to account for pipelining...

Note that this commit itself doesn't really contain any tests because
there's no functional change to Cargo, only internal refactorings. This
does have a large impact on the test suite because the `--emit` flag has
now changed by default, so lots of test assertions needed updating.
    @alexcrichton
    alexcrichton committed Apr 25, 2019
    Configuration menu
    Copy the full SHA
    1caae51 View commit details
    Browse the repository at this point in the history

  2. Implement JobQueue support for pipelining 

    This commit adds support to Cargo's internal `JobQueue` to execute
pipelined rlib compilations actually in a pipelined fashion. This
internally invovled some refactoring to ensure we juggled the right jobs
into the right places, ensuring that as soon as an rmeta file is produce
we can start subsequent compilations but still synchronize with the
finished result of a `Build` unit.

Internally this continues to still not actually pipeline anything in the
sense that rustc doesn't support pipelining, but it's in theory all the
groundwork necessary to read a signal from rustc that a metadata file is
ready to go and then plumb that into Cargo's job scheduler.
    @alexcrichton
    alexcrichton committed Apr 25, 2019
    Configuration menu
    Copy the full SHA
    2114c5d View commit details
    Browse the repository at this point in the history