Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add process lock to avoid parallel cross issues. #963

Closed
wants to merge 1 commit into from

Conversation

Alexhuszagh
Copy link
Contributor

@Alexhuszagh Alexhuszagh commented Jul 18, 2022

Docker can have deadlocks or similar issues when invoked multiple times
in rapid succession, so this implements a small and correct process lock
with an identical API to named_lock. This allows us to prevent
invoking Docker multiple times.

The implementation uses flock on Linux/UNIX-like systems, which on
Linux is guaranteed to be cleaned up when the process exits. On Windows, it uses CreateMutexW, which also guarantees to be cleaned up when the process exits.

We use this rather than fd-lock because fd-lock has more dependencies, and it also uses LockFile on Windows which is not guaranteed to be cleaned up in a timely manner. The API is also designed for file locks, not process locks, which is only a good abstraction on UNIX-like systems.

We use this rather than named_lock since it has numerous, unnecessary dependencies. This allows us to have no external dependencies besides libc and the Windows API, for an extremely lightweight locking library.

Fixes #496.

@Alexhuszagh
Copy link
Contributor Author

Alexhuszagh commented Jul 18, 2022

This seems to handle SIGKILL just fine, and should work on any UNIX-like or Windows system. The lock on SIGKILL comes from cargo and therefore can be fixed with cargo clean. It also works on Windows perfectly fine, with the only lock issue actually being with cargo as well on Windows. It seems that flock respects SIGKILL (it doesn't cleanup, but it ensures that the process can be locked again), and CreateMutexW works just fine on Windows.

This is tested to work with 15 simultaneous processes on Windows and Linux. The Windows is slow, but that's to be expected. And the bottleneck is almost certainly Docker. Also, it seems flock itself is guaranteed to release on any form of process exit on Linux.

For Windows, we can also see CreateMutex has the same guarantees:

Use the CloseHandle function to close the handle. The system closes the handle automatically when the process terminates. The mutex object is destroyed when its last handle has been closed.

@Alexhuszagh Alexhuszagh marked this pull request as ready for review July 18, 2022 19:08
@Alexhuszagh Alexhuszagh requested a review from a team as a code owner July 18, 2022 19:08
@Emilgardis
Copy link
Member

this adds a fair bit of dependencies, is there another (simpler) crate for this?

@Alexhuszagh
Copy link
Contributor Author

Alexhuszagh commented Jul 19, 2022

this adds a fair bit of dependencies, is there another (simpler) crate for this?

I'll see, we probably could use parking_lot directly. Or we could just document this relatively prominently. We could also just use this on Unix-only, where there's significantly less dependencies, since I believe due to how Windows works, this is unlikely to be an issue.

Or, we could probably just use libc directly and spin this out into a simple crate (since it has general purpose): using flock and CreateMutexW alone should be fairly doable, and we don't need to worry about any other dependencies.

@Alexhuszagh
Copy link
Contributor Author

Alexhuszagh commented Jul 19, 2022

I've changed it to use CreateMutexW and flock directly. However, it is worth testing the code just to ensure that the logic is sound. It's pretty straightforward though.

@Alexhuszagh Alexhuszagh marked this pull request as draft July 19, 2022 05:36
@Alexhuszagh Alexhuszagh changed the title Add lock to to prevent deadlock with docker when invoked multiple times. Add process lock to avoid parallel cross issues. Jul 19, 2022
Docker can have deadlocks or similar issues when invoked multiple times
in rapid succession, so this implements a small and correct process lock
with an identical API to `named_lock`. This allows us to prevent
invoking Docker multiple times.

The implementation uses `flock` on Linux/UNIX-like systems, which on
Linux is guaranteed to be cleaned up when the process exits. On Windows, it uses `CreateMutexW`, which also guarantees to be cleaned up when the process exits.

We use this rather than `fd-lock` because `fd-lock` has more dependencies, and it also uses `LockFile` on Windows which is not guaranteed to be cleaned up in a timely manner. The API is also designed for file locks, not process locks, which is only a good abstraction on UNIX-like systems.

We use this rather than `named_lock` since it has numerous, unnecessary dependencies. This allows us to have no external dependencies besides libc and the Windows API, for an extremely lightweight locking library.
@Alexhuszagh Alexhuszagh marked this pull request as ready for review July 19, 2022 06:14
@Alexhuszagh Alexhuszagh marked this pull request as draft July 19, 2022 06:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Error when building in parallel: sh: 1: cargo: not found
2 participants