Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows Defender causing slowdown on contents of .cargo\bin folder #5028

Open
krolli opened this issue Feb 11, 2018 · 12 comments
Open

Windows Defender causing slowdown on contents of .cargo\bin folder #5028

krolli opened this issue Feb 11, 2018 · 12 comments
Labels
A-rustup Area: rustup interaction O-windows OS: Windows Performance Gotta go fast! S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted.

Comments

@krolli
Copy link

krolli commented Feb 11, 2018

Hi,
a recently I noticed that running rustfmt is much slower than it used to be. Digging into it, I found out the slowdown is caused by Windows Defender (specifically MsMpEng.exe, Antimalware Service Executable), but is not limited to rustfmt. In fact, rustfmt in toolchain runs fast on its own, it's rustfmt binary in .cargo\bin directory that is causing large CPU usage spikes in scanning process. Setting Windows Defender to ignore rust-related directories or changing PATH environment variable to search directly in toolchain directory instead of taking the roundabout way through .cargo/bin dispatch binaries fixes the problem.

Since this affects all binaries, it also drags build performance down. I did some measurements on my test project (relatively small) with/without exception in Windows Defender and with direct path to toolchain and indirect path via .cargo:

full rebuild:
no exception, indirect path : 143s
exception, indirect path : 92s
no exception, direct path: 84s
exception, direct path: 83s

incremental build (single changed file):
no exception, indirect path: 4.6s (11.6s from button press)
exception, direct path: 2.8s (3.8s from button press)

Incremental build times are combination of cargo output value (shorter) and ETW trace measurement from pressing build button until link.exe is finished. Incremental build also includes running rustfmt (since the file gets saved).

edit: I tried signing rust binaries in both directories (toolchain and .cargo) with trusted certificate, but it didn't seem to help. However, it was my first time doing this, so I might have made a mistake somewhere.

@ehuss
Copy link
Contributor

ehuss commented Feb 11, 2018

I have noticed this, too. The rustup wrapper itself adds a significant amount of time to execution.

Exe With Defender Without Defender
rustc -V 0.7s 0.09s
~/.rustup/toolchains/stable-x86_64-pc-windows-msvc/bin/rustc.exe -V 0.1s 0.08s
cl.exe 0.08s 0.056s
MinGW gcc --version 0.2s 0.11s

This may not be a cargo-specific issue. It would be nice if the rustup wrapper didn't add a nearly 8x slowdown.

I typically just turn it off while doing development. I don't know if there is a realistic workaround.

@krolli
Copy link
Author

krolli commented Feb 12, 2018

I thought this was a cargo thing, given ".cargo" folder name.

@alexcrichton
Copy link
Member

Oh dear this seems bad! Is there a known way to fix this though? I just realized that I ended up disabling Windows Defender for the .cargo/.rustup directories presumably long ago...

@krolli
Copy link
Author

krolli commented Feb 13, 2018

I just heard from some people this happens even for nvidia gl dll, which is definitely signed.

I don't know of any way to convince Windows Defender to trust those binaries (short of contacting Microsoft). Workaround I am using is I just added path to toolchain directly into PATH variable, ahead of .cargo\bin, but I'm not sure why those dispatching binaries are there, so maybe this is not acceptable in general. Switching toolchains in rustup would then be just changing PATH variable. Cleaner way to do that might be to introduce another variable (eg. RUST_TOOLCHAIN) and just reference that from PATH using.

But, like I said, I'm not sure what is being solved by those dispatch binaries, so this might not be the right way.

edit: Also, I noticed that all of the .cargo\bin binaries are exactly the same. Maybe pulling most of their contents into dll that would be shared by all running processes would prevent it from being scanned multiple times. It would still be painful on single run though.

@steffengy
Copy link
Contributor

@krolli
Is this related to the referenced rustup issue?
Try replacing your rustc.exe, rustup.exe or whatever slow rust binary you're using with https://win.rustup.rs/x86_64.

@krolli
Copy link
Author

krolli commented Feb 13, 2018

@steffengy
It is indeed faster. While adding exception to Defender and directly invoking rust tools still shaves off some time, it is not nearly as pronounced as before. I did a quick trace and it appears that scanning still occurs. It seems to take up about quarter of the original time, which is somewhat better improvement than I'd expect if the slowdown was linear in size of scanned binaries. Still, I think speedup is largely due to their smaller size.

@steffengy
Copy link
Contributor

@krolli

Still, I think speedup is largely due to their smaller size.

It actually isn't.
i686-msvc is still slow, even though it's only 5MB.
x86-64-gnu (13 MB) is fast, even though the file size is even bigger.
So it's likely related to some architectural differences (and special treatment from windows defender).

Anyways those fixes landed in rustup.

@krolli
Copy link
Author

krolli commented Feb 14, 2018

@steffengy
I haven't tried i686-msvc, but I though I had x86-64-gnu installed before when it was really slow. I'll play around with it later to find out more.

@krolli
Copy link
Author

krolli commented Feb 14, 2018

@steffengy
You are right. Both gnu and msvc 32-bit binaries are taking way longer to scan. Looking at the results from profiler, it looks quite odd. Scanning itself is spending a bunch of extra time in things that are not there in fast case (eg. Lua VM and regex matching). But, there are things taking up majority of time in slow case with what seems to be erased callstack information...

@stale
Copy link

stale bot commented Sep 16, 2018

As there hasn't been any activity here in over 6 months I've marked this as stale and if no further activity happens for 7 days I will close it.

I'm a bot so this may be in error! If this issue should remain open, could someone (the author, a team member, or any interested party) please comment to that effect?

The team would be especially grateful if such a comment included details such as:

  • Is this still relevant?
  • If so, what is blocking it?
  • Is it known what could be done to help move this forward?

Thank you for contributing!

(The cargo team is currently evaluating the use of Stale bot, and using #6035 as the tracking issue to gather feedback.)

If you're reading this comment from the distant future, fear not if this was closed automatically. If you believe it's still an issue please leave a comment and a team member can reopen this issue. Opening a new issue is also acceptable!

@stale stale bot added the stale label Sep 16, 2018
@retep998
Copy link
Member

Is this still relevant?

Yes.

If so, what is blocking it?

The lack of a developer willing to put in the time and effort to get this fixed.

Is it known what could be done to help move this forward?

Someone could use a tool like procmon to compare how slow various syscalls are with and without Defender. A side by side comparison should make it clear which syscalls are impacted most by Defender and then the code can be adjusted to avoid those syscalls.

@weihanglo
Copy link
Member

FWIW, #11917 might speed up this a bit with rust-lang/rustup#3178 enabled.

@rustbot label +S-needs-design

@rustbot rustbot added the S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted. label Jun 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-rustup Area: rustup interaction O-windows OS: Windows Performance Gotta go fast! S-needs-design Status: Needs someone to work further on the design for the feature or fix. NOT YET accepted.
Projects
None yet
Development

No branches or pull requests

7 participants