-
Notifications
You must be signed in to change notification settings - Fork 12.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Format only modified files #105702
Format only modified files #105702
Conversation
(rustbot has picked a reviewer for you, use r? to override) |
r? @jyn514 |
🤔 what if I change something in |
x.py fmt takes ~1.6 seconds (2.4 seconds with an empty disk cache) for me. Is it much slower for others? I'm not sure optimizing a few seconds is worth doing -- most of the time, the expectation is that folks are enabling rustfmt on save in their editor, so running it isn't even necessary (this is the case for me), but even without that it should be quite fast to run it. In any case, as a separate item, we need to make sure that we're running it against all files on changes to the rustfmt binary (and other such wide-spread changes, like @matthiaskrgr notes). |
Also, I think it is common to to fix the format differences introduced by the merge/rebase, will rustfmt still recognize files that were changed but committed as modified? |
You can pass any path and the early culling is not applied.
It takes me over 30 secs, but with this modification it now only takes a few seconds when no file changed.
See #105688 for the motivation
I still need to implement this part
This seeks for files modified between the |
For the record |
6 seconds sounds about right; for me at least it's too slow to run format-on-save. If we want to limit the impact, we can make it opt-in and run it only in the pre-push hook, but I would like to eventually turn it on globally or make it opt-out. |
What do you mean by this? I don't run x.py fmt to format on save, I run rustfmt directly. x.py fmt is too slow on save, but that's fine; you're not supposed to use it for that. (Or should pass it the current file).
I think making it opt-in is also good. Regardless, please do make sure that CI is not using this optimization, because I don't trust it to not cause us problems down the line. |
|
😒 , I work in WSL, it seems really slow, maybe need to buy a new device for |
@chenyukang Are those benches made with this branch, or |
it's |
That's not quite enough - changes in |
I think currently you can do |
It doesn't correctly detect files right now, will work on that. @rustbot author |
If
If you're using wsl2 then make sure you have the rust project checked out in a directory mounted as the native linux filesystem (e.g. ext4) as part of the disk image. That should be much faster than the 9p mounts that are used to access the windows filesystem. You also should check if you have any generated directories other than |
@the8472 on a Windows 10 laptop with antivirus scanning excluded for the folder and no untracked files other than |
"no untracked files" can be misleading because |
I wonder if switching from using That said, at least for me 80% of x.py fmt (which takes ~3 seconds) is in rustfmt itself, so it's not clear that significant improvements from which files we visit will be had. I wonder what the split there is like on non-Linux(?) platforms, maybe we spend much more time in directory traversal. Presumably rustfmt itself is about as fast anywhere, regardless of platform. |
I used
👍 seems useful and simpler than the current approach
How did you measure this? Happy to report what it's like on Windows if you know how. |
Wait hold on, this doesn't make a lot of sense to me. Presumably rustfmt runs in about the same time on files that need formatting and those that don't, so visiting less files in the first place should still be a big improvement even if rustfmt's runtime is the limiting factor. |
Sorry, I meant "how we visit files", not "which". |
I measured by trying to produce a flamegraph (with perf on Linux). Not sure what is available on windows like that. |
@the8472 Here is a proof that hyperfine's user/system time is incorrect in this case: When directly invoking python, the user/system time is the same tiny amount regardless of the work done. When indirecting through powershell (which is what x does on windows), the user/system time is bigger than direct python, but the same regardless of the work done. When subtracting the user/system time from the powershell times, we get roughly the same times as their direct counterparts.
|
Which hyperfine version are you using? The user time accounting was changed in 1.15, released in september. |
I do use hyperfine v1.15.0 |
I think the discussion about benchmarking is important to have but not necessarily relevant to this PR. Formatting only modified files will benefit everyone, not just @albertlarsan68, and I think the impact of a bug is limited enough it's ok to land given that we don't have specific code we think is buggy. @albertlarsan68 if you add that line in the changelog I mentioned in #105702 (review), I think we can land this :) |
Looks great, thanks! @bors r+ rollup (not tested in CI) |
…iaskrgr Rollup of 8 pull requests Successful merges: - rust-lang#104402 (Move `ReentrantMutex` to `std::sync`) - rust-lang#104493 (available_parallelism: Gracefully handle zero value cfs_period_us) - rust-lang#105359 (Make sentinel value configurable in `library/std/src/sys_common/thread_local_key.rs`) - rust-lang#105497 (Clarify `catch_unwind` docs about panic hooks) - rust-lang#105570 (Properly calculate best failure in macro matching) - rust-lang#105702 (Format only modified files) - rust-lang#105998 (adjust message on non-unwinding panic) - rust-lang#106161 (Iterator::find: link to Iterator::position in docs for discoverability) Failed merges: r? `@ghost` `@rustbot` modify labels: rollup
Dont use `--merge-base` during bootstrap formatting subcommand I use a development image with Ubuntu 20.04 LTS, which has git 2.25. Recently, `./x.py test tidy --bless` regressed in rust-lang#105702 because it uses the `--merge-base` option on `diff-index`, which was only introduced in git 2.30 (git/git@0f5a1d4). Luckily, it can be replicated via two calls to `git merge-base` + `git diff-index`, so let's just use that.
Dont use `--merge-base` during bootstrap formatting subcommand I use a development image with Ubuntu 20.04 LTS, which has git 2.25. Recently, `./x.py test tidy --bless` regressed in rust-lang#105702 because it uses the `--merge-base` option on `diff-index`, which was only introduced in git 2.30 (git/git@0f5a1d4). Luckily, it can be replicated via two calls to `git merge-base` + `git diff-index`, so let's just use that.
Dont use `--merge-base` during bootstrap formatting subcommand I use a development image with Ubuntu 20.04 LTS, which has git 2.25. Recently, `./x.py test tidy --bless` regressed in rust-lang#105702 because it uses the `--merge-base` option on `diff-index`, which was only introduced in git 2.30 (git/git@0f5a1d4). Luckily, it can be replicated via two calls to `git merge-base` + `git diff-index`, so let's just use that.
Previously, `x fmt` would only format modified files, while `x fmt .` and `x fmt --check` would still look at all files. After this change, `x fmt --check` only looks at modified files locally. I feel pretty confident in this change - other than rust-lang#106261, no one has reported bugs in `get_modified_rs_files` since it was added in rust-lang#105702.
…rsan68 x fmt: Only check modified files locally Previously, `x fmt` would only format modified files, while `x fmt .` and `x fmt --check` would still look at all files. After this change, `x fmt --check` only looks at modified files locally. I feel pretty confident in this change - other than rust-lang#106261, no one has reported bugs in `get_modified_rs_files` since it was added in rust-lang#105702. Combined with the changes in rust-lang#108772, this brings the time for me to run `x t tidy` with a hot FS cache down from 5 to 2 seconds (and moves the majority of the time spent back to `tidy check`, which means it can be sped up more in the future).
As discussed on #105688, this makes x fmt only format modified files.
Fixes #105688