Windows runners are consistently extremely slow compared to Linux and macOS #7320
Comments
|
I suspect it might be Windows Defender in action. |
|
Here are many other people similarly seeing that |
See commit myitcvscratch/slow-windows-actions@8b71de3 which resulted in run https://github.com/myitcvscratch/slow-windows-actions/actions/runs/4480846656/jobs/7876717612. It has basically no effect. I also tried making this change in the CUE project in https://review.gerrithub.io/c/cue-lang/cue/+/551317. That resulted in https://github.com/cue-lang/cue-trybot/actions/runs/4476128503 which again showed no effect. So assuming my testing is valid, turning off Windows Defender does not appear to have any effect in our case. |
Thanks, @mvdan. Just to emphasise however that the use of The reason I flag this is that ultimately we will consider this issue "fixed" when the |
|
I've also tried to disable Windows Defender, no significant difference so far https://github.com/ilia-shipitsin/slow-windows-actions/actions/runs/4483502669/jobs/7882868552 |
|
@myitcv , as for I mean, is degradation |
We only have the numbers from GitHub actions workflow runs. |
|
we need to narrow it, whether standalone Windows server behaves the same slow or not |
|
Duplicate of #5166 |
|
@mikhailkoliada - isn't #5166 demonstrating a slowdown between 2019 and 2022? The numbers we are seeing show consistent slowness on both 2019 and 2022. Therefore I'm not clear this is a duplicate. |
|
side note, I did some investigation on "checkout slowness", it looks like there's some delay between git and automation task. git itself takes 2-3 seconds (I put commands into cmd and wrapped with
I can beleive that agent communication could add 15-20 sec, but it does not look like a root cause for |
|
Hey! let me chat with the team and see what is going on and see if we need a separate wrap up ticket for Windows Perf (or I will get someone to re-open this one!) |
|
@mvdan @myitcv we think this may be the same as the checkout issue. Given that 'hunch' we will tackle that tracking our progress here: actions/checkout#1186 We will keep this issue open until we can validate if it is a dupe and go from there (or start working on a new root cause for this after I guess 😱 let's hope not!) |
|
@nebuk89 - thanks for looking into this and the detailed update. Much appreciated. |
This job is still timing out sometimes. There are some open bugs about slow Windows runners, maybe relevant: actions/runner-images#7320
This job is still timing out sometimes. There are some open bugs about slow Windows runners, maybe relevant: actions/runner-images#7320
This job is still timing out sometimes. There are some open bugs about slow Windows runners, maybe relevant: actions/runner-images#7320
Trying a suggestion mentioned at:
actions/runner-images#7320 (comment)
Signed-off-by: Paul Jolly <paul@myitcv.io>
Change-Id: I37200a5c6bc936f2de3bb8b96034a2892404e48a
Dispatch-Trailer: {"type":"trybot","CL":1174197,"patchset":1,"ref":"refs/changes/97/1174197/1","targetBranch":"master"}
Any chance we can get Windows XP runners? |
For completeness, this had no impact in our situation. |
|
Switching to |
|
Also having issues with Windows runners, they are extremely slow compared to their MacOS and Ubuntu counterparts:
(Can't link as the actions are in a private repo) |
|
Our tests are now running at 50+mins compared to Linux < 10mins, moving to windows-2019 didn't fix anything, neither did disabling windows defender. We are moving away from GitHub managed runners for these, moving to in-house self-hosted runners. |
|
For what it's worth, over at Astral we switched to using a Dev Drive with ReFS to great benefit astral-sh/uv#3522 (the GitHub Windows Runners are still the bane of my existence though) |
|
End of July 2024 - any progress on this oldish Windows anomaly? My partner and I test our work on Ubuntu, MacOS, and Windows (significantly slower) to make sure we can run on all 3 environments. We are lucky that our runs are short at the moment (< 8 minutes) but expect project growth and therefore longer run times in the future. |
|
Our benchmarks are also incredibly slow on Windows. The build in our case is a Rust package, windows spends 10x more time to download and compile crates compared to MacOS and Linux.
|
Discovered while updating `rabbitmq/rabbitmq-tutorials` to version `7.0.0-rc.8` of this library. * Add `trackConfirmations` argument to `ConfirmSelectAsync` to allow disabling internal confirm tracking. * Increase CI timeouts since GHA Windows runners are slow (actions/runner-images#7320)
|
Yeah this is aweful. Besides jobs taking way longer, I also always get failures from network timeouts when on windows only. It only ever happens on windows runners... never on macos or linux runners. |
|
Not a Windoze lover but our project workfile runs on 3 OSes; Windows is the slowewwwwwwwwwwweest leg in our jobs. cc: @platypusguy |
|
This is getting ridiculous, to the point where "optimizations" actually turn out to be "pessimizations", because of this. Together with https://github.com/orgs/community/discussions/42335, not to mention actions/runner#1182 (to potentially reduce copy-pasting somewhat, as a workaround for the workaround for the slowness...), it really makes Windows+GHA difficult to like... |
|
|
|
I recently had to use a Windows runner and came here to say the experience was awful due to how slow it is. On the same size runner, it can take more than 8x the time vs using a Mac or Linux runner. I had to use Windows to simulate a specific environment. |
and others
Description
In the CUE project we are seeing incredibly slow run times on Windows 2022 runners.
For a recent example see https://github.com/cue-lang/cue-trybot/actions/runs/4477594562/.
Roughly speaking, here are the numbers we are seeing averaged out across 20-30 builds per day.
actions/checkoutgo test(cache hit)ubuntu-22.04macOS-11windows-2022Notice that
actions/checkoutis consistently slower on Windows. Yes, there is a network access element to this, but it is minimal.The
go testcomparison is the cleanest comparison. As indicated in the column heading, these are the timings for a full cache hit. i.e. no network access required, no rebuilds required, no tests actually running (because they hit the test cache). So thego testcommand is purely a function of disk access and CPU. For this command, Windows is consistently 15 times slower than Linux. macOS is, pleasingly, comparable with Linux.Platforms affected
Runner images affected
Image version and build link
For
windows-2019:For
windows-2022:Is it regression?
Unclear
Expected behavior
Windows runners to be comparable in terms of speed to Linux and macOS for
actions/checkoutandgo teststeps.Actual behavior
Windows runners consistently taking 15 times as long as Linux and macOS for CPU and disk-intensive commands.
Repro steps
The CUE repo itself is quite involved. So as a proxy for something that is relatively CPU and disk intensive we have created a slimmed down repo using
actions/checkout.https://github.com/myitcvscratch/slow-windows-actions
See the most recent run for results:
https://github.com/myitcvscratch/slow-windows-actions/actions/runs/4477601338
Looking at averages of this setup across a number of runs we see similar figures for
actions/checkoutto those seen in our CUE setup:actions/checkoutubuntu-20.04ubuntu-22.04macos-11macos-12windows-2019windows-2022So whilst this doesn't include the
go teststep (because getting a warm cache is a tricky step to reproduce) the use ofactions/checkoutis a sufficiently good proxy to show the problem.The text was updated successfully, but these errors were encountered: