Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need a big optimization for Windows #990

Closed
OshotOkill opened this issue Oct 13, 2016 · 81 comments
Closed

Need a big optimization for Windows #990

OshotOkill opened this issue Oct 13, 2016 · 81 comments

Comments

@OshotOkill
Copy link

OshotOkill commented Oct 13, 2016

Do you want to request a feature or report a bug?

Feature

What is the current behavior?

I've tested a lot about the installation speed between MacOS and Windows. According to the results it seems that yarn has far less optimizations for Windows. e.g. Here are the comparisons of installing react-native :


Test Machines:

  • ThinkPad X1 Carbon 4, 1TB PCI-E SSD, 16GB Memory
  • MacBook Air 2014, 256GB SSD, 4GB Memory

No cache & same network environment


MacOS

npm@3.10.9: 1m 31s

2016-10-13 17 52 24

yarn@0.15.1: 39s

2016-10-13 17 54 53


Windows

npm@3.10.9: 2m 24s

2

yarn@0.15.1: 2m 19s

1


So, it seems yarn has no advantage over npm on Windows. Did anyone face with this appearance?

Please mention your node.js, yarn and operating system version.
nodejs: 6.8.0
yarn: 0.15.1
OS: Windows 10 14393.321 & MacOS 10.12

@atkawa7
Copy link

atkawa7 commented Oct 13, 2016

+1

@Daniel15
Copy link
Member

Daniel15 commented Oct 13, 2016

Hi @OshotOkill! Thanks for trying Yarn. Are you using Cygwin or WSL ("Bash on Ubuntu on Windows")? Both are known to have pretty bad disk IO performance.

Also, React Native has a huge number of files so copying them into node_modules is pretty slow, and disk IO for lots of little files on Windows is generally slower than Mac OS (which itself is slower than Linux ext4). We do have a task to experiment with hardlinks (#499) which should improve perf in this scenario.

No cache & same network environment

The main improvement with Yarn is when you have a warm cache (ie. after you've installed a package at least once), but with React Native, the huge number of files will also be a cause of some of the slowness.

@OshotOkill
Copy link
Author

@Daniel15 Nope, I'm not using Cygwin/MinGW/MSYS2 or WSL (the latter fails due to a knotty bug).

According to your description I can assume the problem is caused by the file system (NTFS) right ? Even if a warm cache exists, the copying process still runs much slower than MacOS.

Hope the dev teams can come up with a solution ASAP. Thanks.

@fooey
Copy link

fooey commented Oct 14, 2016

I'm seeing the same.

Do install, wipe node_modules, install

MacBookPro takes 17 seconds, my Windows machine takes 122 seconds.

@cpojer
Copy link
Contributor

cpojer commented Oct 14, 2016

Somebody pointed out this might be related to anti-virus software scanning node_modules and the global yarn cache continuously. Can you try disabling it for those folders?

@OshotOkill
Copy link
Author

OshotOkill commented Oct 14, 2016

@cpojer I guess they are right . I don't have any anti-virus software on my machine except the pre-installed Windows Defender, so I banned the scanning of global cache folder & my project folder and did some test:

Default: 128.08s

2


No scanning of cache folder: 104.43s

3


No scanning of project folder: 78.28s

5


No scanning of cache folder & project folder: 53.48s

4


Though it's slower than Mac for 10+s, it has a significant boost.

This should be informed from the official docs I think.

@Daniel15
Copy link
Member

Daniel15 commented Oct 14, 2016

Somebody pointed out this might be related to anti-virus software scanning node_modules and the global yarn cache continuously.

Good catch! I totally forgot about this as I already have c:\src whitelisted on my computers.

@OshotOkill - Would you like to submit a pull request adding a note about antivirus apps to the website, in the Windows installation instructions? Here's the file you'd need to edit: https://github.com/yarnpkg/website/blob/master/en/docs/_installations/windows.md (you can edit it directly on Github). It'd be appreciated 😄

@fooey
Copy link

fooey commented Oct 14, 2016

I wasn't as meticulous as @OshotOkill, but I added exceptions for my source and my node install folder, and then specifically exempted the yarn, npm and node binaries and now my fresh install time on Windows is down to 50 seconds from 122 seconds.

@OshotOkill
Copy link
Author

OshotOkill commented Oct 15, 2016

@Daniel15 PR is ready. Apologize for my poor English.

@OshotOkill
Copy link
Author

PR has been merged. Close this issue.

@voliva
Copy link

voliva commented Nov 22, 2016

This is still painfully slow on windows, even deactivating anti-virus and Windows defender. I don't think it's just an environment issue (like this anti-virus solution) but it looks like yarn tries to copy all the files, 1-by-1 even if you install some unrelated dependency.

Why not just delete/copy the files that need to change? If I had webpack installed and is not modified when I installed rimraf, It shouldn't have to be copied again from the cache to the local node_modules folder.

@amcsi
Copy link

amcsi commented Nov 26, 2016

I have created a StackOverflow article about this too: http://stackoverflow.com/questions/40566222/yarn-5x-slower-on-windows

By the way in my (dual-booted) Ubuntu benchmarks I was using the same NTFS drive as the one Windows normally runs on; and it's still fast there.

@bestander
Copy link
Member

Adding node.exe to Windows Defender exclusions gave me a huge performance boost http://126kr.com/article/1884rsed7l

@amcsi
Copy link

amcsi commented Dec 22, 2016

I'll definitely try this out!

@amcsi
Copy link

amcsi commented Dec 29, 2016

It did seem to improve the speed a bit 212 -> 170 seconds
So it seems to help, but it could still be improved, because it's still more than 3x slower than in Linux

@bestander
Copy link
Member

bestander commented Dec 29, 2016

Another issue I have noticed - Indexing service on Windows tries to index every file in node_modules.
I don't really need it at all, so I disabled it http://www.softwareok.com/?seite=faq-Windows-10&faq=53 and gained another performance boost.

@amcsi
Copy link

amcsi commented Dec 29, 2016

My windows isn't set to index the path in question, so that still doesn't solve the issue.

@Altiano
Copy link

Altiano commented Jan 29, 2017

So to sum up there are 4 ways to improve performance:

  • Whitelist project folder from AV
  • Whiteilst the Yarn cache directory ((%LocalAppData%\Yarn)) from AV
  • Adding node.exe to Windows Defender exclusions
  • Disabling Indexing service on Windows on node_modules folder

@amcsi
Copy link

amcsi commented Jan 29, 2017

@Altiano yes, but it's still not enough to get performance even close to Mac/Linux

@michaellashinsky
Copy link

michaellashinsky commented Jan 30, 2017

Seems kinda sketchy that you'd have to disable AV or indexing on directories to make yarn as fast or faster than npm. After all, you don't have to do this for npm. I decided to give yarn a shot because it stated it was fast and the offline installs made coding without a network connection a plausible thing. Is there no way to optimize the linking?

@OshotOkill
Copy link
Author

OshotOkill commented Feb 4, 2017

According to some issues that relate here and the comments above, I'd like to reopen this issue in order to gather some other solutions.

Personally I suggest to list the hardware configurations about your test machine and upload some related pics. There could be many other irrelevant elements that make a big difference between platforms rather than Yarn itself, i.e. the benchmark performance of the SSD on a MacBook is usually much better than a Windows machine.

@OshotOkill OshotOkill reopened this Feb 4, 2017
@bestander
Copy link
Member

I think you have badly hoisted node_modules, this bug is going to be fixed in #2676

@ghost
Copy link

ghost commented Feb 15, 2017

With @bestander's introduction of hardlinking in #2620 (Which works fine in Windows 7 without administrator privileges), my overall installation times, and node_modules/ size, dropped.

Without hardlinking:

Done in 167.76s.

real    2m49.633s
user    0m0.229s
sys     0m1.368s

du -sh node_modules/
216M    node_modules/

With hardlinking:

Done in 58.07s.

real    0m59.967s
user    0m0.183s
sys     0m1.369s

du -sh node_modules/
189M    node_modules/

@bestander
Copy link
Member

bestander commented Feb 15, 2017 via email

@kuncevic
Copy link

kuncevic commented Mar 27, 2017

I am on Win7 /w Yarn v0.21.3

[3/4] Linking dependencies...
Done in 947.71s. 

Had to wait this amount of time adding any new package with yarn add ...
Defender off
Indexing disabled

Have some other AV running so just following these steps as above mentioned by @Altiano

Whitelist project folder from AV
Whiteilst the Yarn cache directory ((%LocalAppData%\Yarn)) from AV

Will update on this one

@bestander
Copy link
Member

@kuncevic, what would be a clean install time for the project?
What is the size of node_modules folder in files?
How does it compare to npm install?

@kumarharsh
Copy link

kumarharsh commented Mar 27, 2017

@bestander this is the same problem with me. Any yarn add or yarn remove takes equal time - about , even after @kittens' hoisting fixes.

Everytime this happens:

  1. First, the Fetching packages runs (takes about 30s):
    image
  2. Then, linking dependencies pauses for about 1 or 2 minutes.
    image
  3. Then, it resumes with next step, and installs(?) 63k files again.
    image

As I said, this happens every single time I run yarn add or yarn remove. It does not matter if the dependency I'm installing depends on any other dependency, A simple npm install for installing a new dependency or upgrading an existing one takes a fraction of this time. Things did improve by 2x with @kittens' hoisting fixes, but still the time taken is too much.

@bestander if you want a reproducible case, please clone this repo: https://github.com/kumarharsh/yarn-bug, and run yarn install, and then yarn add react-helmet.

@bestander
Copy link
Member

Yarn is preserving determinism every time it runs add/remove, so it needs to check if any dependency got hoisted to the root of node_modules when dependencies change.
That is why it runs full linking phase.

Fetching dependencies - download, you can't optimise it.
First linking phase (1561 operations) - it creates all the folders for all the dependencies.
Second linking phase (63K operations) - it copies the files from cache to node_modues.

Yarn optimises file copy operations by checking if the files are the same before doing the copy.
We might want to profile this area better and see if we can decrease number of unnecessary IO.
Maybe on Windows copying would be faster then checking?

What about npm, how fast it does clean install?

@kumarharsh
Copy link

A clean install for npm (npm install) takes 552301.1944ms.
Installing an additional dependency (npm install weird) takes 57023.7593ms. (Most of this time is wasted in paperjs trying to install canvas as a dep - but this time would be common for both npm & yarn)

A clean install for yarn (yarn install) takes 612698.4915ms.
Installing an additional dependency (yarn add weird) takes 495633.0307ms.

npm version 3.10.9
yarn version 0.21.3

@vbfox
Copy link
Contributor

vbfox commented Mar 28, 2017

@bestander @kumarharsh Yarn doesn't optimises file copy operations on windows due to a libuv/nodejs bug (See #2958 for a potential fix in yarn code) that isn't present on node 7.1+ so you can get your second command (yarn add) to be a lot faster just by upgrading node.

Using windows file copy operation is a little bit faster than using node API to copy files too (See #2960 for a potential PR) and would optimize yarn install a little bit but I don't know if it would egalize with npm (didn't test)

@kuncevic
Copy link

kuncevic commented Mar 30, 2017

Just updated to 7.8.0

nvm install 7.8.0
npm install npm -g (came with 4.4.4)
nvm use 7.8.0
`git clone https://github.com/angular/material2`
cd material2
yarn install - Done in 210.22s.
rimraf node_modules
yarn install - Done in 180.66s.
rimraf node_modules
yarn install - Done in 181.11s.

However by doing yarn add rimraf got it done in 20.52s. but why yarn install after removing node_modules taking so long?

p.s.

rimraf node_modules
npm install - Done in 332.4s
rimraf node_modules
npm install - Done in 402s
rimraf node_modules
npm install - Done in 489.6s

@vbfox
Copy link
Contributor

vbfox commented Mar 31, 2017

@kuncevic Nice to see that upgrading node works for yarn add :)

Regarding empty node_modules a good thing to do is to measure how much is due to yarn and how much is due to FileSystem, Hard drive & Anti virus.
What I did to test that was to copy the full node_module (As generated by yarn, not npm) of material2 somewhere in yarn cache :

for /f "delims=" %i in ('yarn cache dir') do set yarncachedir=%i
xcopy /E /Y /I /Q node_modules %yarncachedir%\x-temp

And then for each test I cleaned node_modules & ran either yarn install, npm install or an xcopy from the previously created folder :

rd node_modules /s /q
powershell -Command "Measure-Command { xcopy /E /Y /I /Q %yarncachedir%\x-temp node_modules}"

And took the total seconds.

Results

Here are the results on 3 PCs

  • 🏠 Home PC: Samsung 950 Pro NVMe, ESET Nod32
  • 🏢 Work PC: Samsung 850 EVO SATA, TrendMicro OfficeScan that I can't disable
  • 🍎 MacBook pro: 2015 version, on macos, no anti virus
yarn 🏠 npm 🏠 xcopy 🏠 yarn 🏢 npm 🏢 xcopy 🏢 yarn 🍎 npm 🍎
AV Disabled 34s 90s 23s - - - 32s 92s
AV Exclude cache & code 38s 104s 29s - - - - -
Av Exclude cache only 43s - 31s - - - - -
Av full 48s 122s 32s 100s 274s 236s - -

Each time AV was enabled it was toping the CPU chart during yarn install or xcopy (On my home PC 30% cpu total was taken at the max but on my work PC it fill one core for xcopy & all my cores for yarn)
xcopy is slower on my work PC than yarn, I suspect because it doesn't copy files in parallel while yarn does (That shouldn't matter for IO bound operations but AV are making it a CPU bound operation & xcopy wasn't written to fight so much stupidity 😄 )

In conclusion

  • yarn is faster than npm and can even be faster than xcopy when AV make file copy CPU bound
  • Windows on a good SSD isn't really slower than a MacBook Pro 2015 (That already has a good SSD) even if it's hard to compare as not exactly the same packages install, & not all post-install scripts do the same thing
  • Some changes could be done in yarn to sidestep that (symlinking files ?) but essentially coping lot of small files is slow
  • On windows AV can make it slower, mine add 30% when enabled in both source & dest folders 😞
  • Corporate AVs can be a magnitude slower than home AVs & kill performance enough for any copy operation to be painful (when it make the naturally IO bound operation CPU bound) 😡

@silverqx
Copy link

silverqx commented Apr 15, 2017

Adding npm, yarn cache folders and node.exe to defender's exclusion list would be enough, of course all this can't be in indexed folders. Now yarn add / rm takes 7 secs

@bestander
Copy link
Member

Thanks everyone, a significant optimization for Windows landed in 0.24 #3234 (comment)

@cowwoc
Copy link

cowwoc commented Jul 6, 2017

@vbfox Can you please add version numbers for npm and yarn in your benchmark?

@nebulr
Copy link

nebulr commented Oct 5, 2017

this is still a piece of shit for MacOSX

@thecjharries
Copy link

I'm still experiencing some crazy install times. yarn add seems to install and link everything (all items in package.json, ~30k dependencies) all the time.

Linux versions:

$ yarn -v
1.3.2
$ node -v
v8.9.3

Windows versions:

> conemu-cyg-64.exe --version
ConEmu cygwin/msys connector version 1.2.2
> wslbridge.exe --version
wslbridge 0.2.3
> Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion' | Select-Object ProductName, CurrentMajorVersionNumber, CurrentMinorVersionNumber, ReleaseId, CurrentBuild, CurrentBuildNumber, BuildLabEx


ProductName               : Windows 10 Pro Insider Preview
CurrentMajorVersionNumber : 10
CurrentMinorVersionNumber : 0
ReleaseId                 : 1709
CurrentBuild              : 17025
CurrentBuildNumber        : 17025
BuildLabEx                : 17025.1000.amd64fre.rs_prerelease.171020-1626

I've got two (and a half) questions:

  1. What's the accepted solution to the issue in this thread? Was it use close instead of finish event for file copy completion #3234 or tweaking Windows Defender?

    • If the solution was tweaking Windows Defender, is there a complete writeup of what to do somewhere?
  2. Is my issue actually related to this thread, or should I create a new one?

@bestander
Copy link
Member

Thanks for raising a new issue, I'll respond there

@mkamranhamid
Copy link

It's been almost an hour now and I'm waiting for this command to finish the process. I've followed the above points mentioned by @Altiano but nothing works

@mkamranhamid
Copy link

do we have any alternative for this? like can i use npm i -g . will it act in the same way or I'll have to make some changes because this code uses yarn workspace

@mkamranhamid
Copy link

So finally after struggling for 2-3 hours, I had to use npm i -g . instead of yarn global add file:.. npm worked like a charm

@robbykrlos
Copy link

robbykrlos commented Feb 5, 2021

Not sure how this is helping, but it is. Initially one change triggered a compilation that took ~1 minute. After running the steps below, it took 3-10 seconds. Not sure if it's still related to Windows Defender, or how fast the file access is improved after the complete delete and re-install (maybe it's like a defragmentation that helps on reading speeds?)

Install RimRaf:
npm install rimraf -g
And in the project folder delete the node_modules folder with:
rimraf node_modules
Install again all node_modules:

yarn install
yarn serve

Build and Serve are now working normally 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests