-
Notifications
You must be signed in to change notification settings - Fork 392
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WITH DOCKER is not executed in parallel in latest v0.6.28 (but it was in version v0.6.25) #2377
Comments
Hi @jrodrigv - what's your |
Hi @vladaionescu - In both cases I was using:
|
I suspect it has something to do with either
Can you try the following:
It's a way to narrow down to what the cause of this might be. |
Hi @vladaionescu . I have tested the Thanks |
I was hoping you would be able to try the second suggestion too and see if that influences things too. The |
Hi @vladaionescu , I have tried the 2nd suggestion as well using just the In this scenario, all the Not sure which could be the root cause but looking at the logs it looks like each test-executor is doing a final image transfer of the docker compose service. And then the next test-executor is pulling the previous transferred image from the embedded registry? Creating a kind of artificial dependency between the test-executor instances.
However, going back to |
I have tested this issue on 0.7 I can reproduce it. I hace created a Earthfile here that can be used to test it. https://github.com/jrodrigv/EarthlySamples/blob/test/Earthfile |
Seeing the same issue here! Version Also re-parallelizes for me if I use |
…rent args (#3406) Fixes #2377 The previous `VisitedCollection` implementation attempts to wait for ongoing targets with the same name to complete before comparing the inputs of the target. This has the advantage of being more precise with which ARGs actually influence the outcome of the targets, ignoring overriding ARGs that end up being unused within those targets. But it also has the disadvantage that all targets with the same name (but different overriding args) execute sequentially. This new implementation simplifies this greatly, by computing the target input hash upfront based on all overriding args, without knowing if there are any args that will end up being unused. The result is that we are able to run all targets with the same name but different args in parallel. But it also has the disadvantage that in some cases we would create duplicated LLBs for some targets when certain overriding args are different but they are unused. Buildkit will generally de-duplicate the LLB in these cases, although it's possible that there might be edge cases if the LLB construction is not consistent. --------- Co-authored-by: nacho <idelvall@brutusin.org> Co-authored-by: Vlad A. Ionescu <vladaionescu@users.noreply.github.com>
Thanks Vlad! I was waiting specifically for this one to jump from 0.6 to
0.7, we execute all tests projects in parallel :)
El lun, 23 oct 2023 10:11 p. m., Vlad A. Ionescu ***@***.***>
escribió:
… Closed #2377 <#2377> as
completed via #3406 <#3406>.
—
Reply to this email directly, view it on GitHub
<#2377 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB7IR62ZAOAFXEEG3V3Q24TYA3FPDAVCNFSM6AAAAAAR2FIA5KVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJQG42DKNZXGY4DMOI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Great - please note that this is behind a feature flag. To enable it, you need to use |
Hi @vladaionescu , I have been doing some testing with the flag using v0.7.21 using the example I provided and It does not seem to work in parallel but still sequentially. I have recorded a video as evidence. earthly-parallel-withdocker.mp4 |
Hi @jrodrigv - Many thanks for the video - it showcases the issue clearly. It seems that I fixed a slightly different situation than your setup. They're closely related though. It's possible that we either don't yet support the FOR loop, or the series of COPY commands as a way to parallelize. I'll take a look into this. The old |
Hi @vladaionescu today I was doing some testing with v0.8.11 and I found something interesting that I'd like to share with you. It seems the problem is caused when using FOR + WITH DOCKER when the targets are being executed sequentially rather that in parallel, however N explicit BUILD commands are executed in parallel successfully.
|
Hi,
I have some targets meant for running tests projects in parallel that are using WITH DOCKER to run a Rabbimq service before running the tests. Using v0.6.25 I can see how the test-executor instances are running fully in parallel loading the rabbitmq tar files in parallel etc. However when using v0.6.28 this is not the case anymore and the test-executor instances are running sequentially instead. Is there a way to prevent this from happening?
The text was updated successfully, but these errors were encountered: