-
Notifications
You must be signed in to change notification settings - Fork 328
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Ray] Ray execution state #3002
Conversation
Will this ray actor influence the lineage reconstruction in #2972:
|
Currently, some operands use a remote object to sync states. If the state actor is reconstructed, the simplest way to recover the compute is,
We should make the operands stateless to avoid above complex recovery. This PR is to maximize compatibility with existing Mars execution logic, the fault recovery is not included in this PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@chaokunyang could you please review this PR? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. I left two comments where i am a little confused.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
What do these changes do?
Mars execution context provides remote object to sync states between multiple running operands. Ray execution backends use a task state actor for this feature.
With this PR, ray backend supports incremental index.
HeadOptimizedDataSource
.The drawback is that,
Related issue number
#2893
Check code requirements