Handle parallel fetches correctly #14
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
arajasek
approved these changes
Jun 14, 2021
| }() | ||
| if lockfail { | ||
| // we've managed to get the lock, but we need to re-check file contents - maybe it's fetched now | ||
| ft.maybeFetchAsync(ctx, name, info) |
There was a problem hiding this comment.
is this recursive call really better than just calling checkFile here and falling through to doFetch as needed?
There was a problem hiding this comment.
It's not really recursive as maybeFetchAsync runs this in a goroutine.
We're just calling maybeFetchAsync because:
- We don't want to be hashing files with those locks (want to do that in parallel, locks only for writing)
- Given the above, we'd want to drop those locks before calling checkFile
- After calling checkFile we'd need to acquire locks again
- So at that point we can just call maybeFetchAsync
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously running 2 parallel paramfetch instances would result in corrupted parameter files, this PR adds an fs lock which should guard against that.
If we fail to acquire the lock, we'll wait for some time, then we'll re-check the file, and retry acquiring the lock if the file still isn't fetched