Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add synchronize call between MOFF and Save #4

Merged
merged 1 commit into from
Apr 1, 2020
Merged

Add synchronize call between MOFF and Save #4

merged 1 commit into from
Apr 1, 2020

Conversation

adampbeardsley
Copy link

@adampbeardsley adampbeardsley commented Apr 1, 2020 •
edited
Loading

I'll just copy-paste Jayce's explanation because I don't fully understand it.

The underlying problem was that all of the CUDA operations are done asynchronously. Normally Bifrost forces a sync when the data cross a memory space boundary. However, when using a CUDA-based ring you end up with data being written that isn't actually there until the memcpy completes. The reserve on the ring doesn't know about this so when that reserve is released the next block starts to read. That read can start before the copy finishes.

@jaycedowell
Copy link

Yep.

@adampbeardsley adampbeardsley merged commit 8d7d622 into master Apr 1, 2020
@jaycedowell
Copy link

If anyone wants a bit more info: bifrost#138

@adampbeardsley adampbeardsley deleted the fix_sync branch April 2, 2020 16:07
@adampbeardsley adampbeardsley mentioned this pull request Sep 16, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jaycedowell jaycedowell jaycedowell approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@adampbeardsley @jaycedowell