-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Long running compile commands can cause incorrect dirty status computation #1162
Comments
I guess it's fine for any build system (ninja, make, etc) to make an assumption "inputs are not changing while command is running", so build system should write down to cache mtimes of inputs before it started executing a command, and than build system should gracefully pickup new changes in second run. At least this is what I expect from any build system on the market. |
Two thoughts:
|
On Mon, Jun 6, 2016 at 4:23 PM, Evan Martin notifications@github.com
-- Dirk |
Sorry, I mixed two ideas there: there's 2a (mix file mtimes into the hash of "all inputs"), which should be about as fast as the current code and will solve this bug. And then I mentioned content hashing as idea 2b that this would enable, but it's something to consider another time. (It definitely has speed challenges but they are surmountable.) |
@martine: What about a simpler solution (for now at least), where the log file keeps track of the mtime of the most recent input for each output at the time the build was started? This would require a log version bump, but I've got this working locally. I can clean it up and submit the pull request if you are interested in seeing it. Basically, in |
Or instead of storing the most recent input's time, we could just store the time that the edge started building and use that as the output's mtime on subsequent runs. The most recent input's time doesn't quite work with generated inputs. |
Rather than changing ninja, wouldn't it be sufficient to wrap your build commands in a script which would record mtime (or md5) of the input files before running the tool, verify these are still the same after the tool has written the output, and touching the changed inputs to make these newer than the output (or deleting the output and failing the command if you consider this an error)? This won't be quite trivial with the implicit dependencies (I mean header files which are only discovered in process of building the target, using 'dep files'), but none of your proposals seems to address that either. |
Just to update this issue, I did resolve the bug a few months back with my pull request referenced above. This could technically be solved with a wrapper script, but is unnecessarily complex and error-prone. The build tool should philosophically be able to handle cases such as this. My fix resolves all the issues I listed, plus the concern @maximuska brought up. On subsequent builds, it will always use the time the output began building as its "mtime" so any implicit dependency discovery or input file differences will be compared against the time the output began building instead of comparing against its actual filesystem mtime. It makes ninja's dirty output checking actually correct by removing the race condition that occurs by changing an input between the output starting its build and the output being written to disk. If there are any qualms about my pull request, we can start a discussion there. |
This was a bit of a head scratcher for us and took us awhile to nail down. It seems that ninja is looking purely at the mtimes of inputs and outputs to determine if an edge is dirty. This doesn't seem to be quite sufficient, because a long running compile command might have an input that is changed during the compilation, which would still result in an output file with a newer mtime than the most recent input file, even though the output file is actually dirty since it was started with an older version of one of its inputs.
This is difficult to reproduce, but I did the following:
Create a gcc wrapper that splits compile commands into two: a preprocess step and a compile step.
Add a sleep between the preprocess and compile to simulate a long running compile command.
Create a simple build.ninja file that just compiles a single test file into an executable.
Run ninja using the wrapper. While it's sleeping, change the source file (eg, return a different value).
After it finishes linking, running the executable will return the original value (as expected), but subsequent ninja runs will not detect that the output is dirty and will report nothing to do, even though it is stale.
This is an issue for us here as we have a lot of large translation units and tend to have a rapid modify/build/test cycle.
The following code is where it checks the mtimes of the most recent input against the output:
ninja/src/graph.cc
Lines 161 to 184 in b231274
Though I'm not sure what the best (fastest) way to actually solve this would be, I'm willing to take a crack at it if we come up with a decent idea for how to do so.
The text was updated successfully, but these errors were encountered: