Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: make actors reschedule indefinitely with backoff #1475

Closed
wants to merge 1 commit into from
Closed

feat: make actors reschedule indefinitely with backoff #1475

wants to merge 1 commit into from

Conversation

MasterPtato
Copy link
Contributor

@MasterPtato MasterPtato commented Nov 26, 2024 •
edited
Loading

Fixes RVT-4196
Fixes RVT-4179

Implemented but untested

Changes

@cloudflare-workers-and-pages Cloudflare Workers and Pages
Copy link

cloudflare-workers-and-pages bot commented Nov 26, 2024 •
edited
Loading

Deploying rivet with  Cloudflare Pages  Cloudflare Pages

Latest commit: 2084acf
Status: ✅  Deploy successful!
Preview URL: https://567c4649.rivet.pages.dev
Branch Preview URL: https://11-26-feat-make-actors-resch.rivet.pages.dev

View logs

@graphite-app Graphite App
Copy link
Contributor

graphite-app bot commented Nov 26, 2024

Your org requires the Graphite merge queue for merging into main

Add the label “merge-queue” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge.

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

@MasterPtato Graphite App
Copy link
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@linear Linear
Copy link

linear bot commented Nov 26, 2024

RVT-4196 Add backoff for actor rescheduling

RVT-4179 Fix durability when going from 1 -> 0 -> 1 available pb clients

@MasterPtato MasterPtato marked this pull request as ready for review November 26, 2024 02:32
NathanFlurry
NathanFlurry requested changes Nov 27, 2024
View reviewed changes
Copy link
Member

@NathanFlurry NathanFlurry left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question on slack regarding retry if actor fails to start vs just retrying on failed alloc

edit: handle retries for all reschedule events except for draining

@MasterPtato MasterPtato force-pushed the 11-26-feat_make_actors_reschedule_indefinitely_with_backoff branch from edb783b to a11a624 Compare November 27, 2024 19:24
@MasterPtato Graphite App
Copy link
Contributor Author

@NathanFlurry updated to reset retry count only after ready

@NathanFlurry NathanFlurry force-pushed the 11-26-feat_make_actors_reschedule_indefinitely_with_backoff branch from a11a624 to 982c1e7 Compare December 2, 2024 05:25
NathanFlurry
NathanFlurry requested changes Dec 2, 2024
View reviewed changes
packages/services/ds/src/workflows/server/pegboard/mod.rs Outdated
}
};
// Sleep for backoff
ctx.sleep_until(next).await?;
Copy link
Member

@NathanFlurry NathanFlurry Dec 2, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Afaik this still doesn't handle actors that crash immediately, those should have a backoff for restarting too. Shouldn't the backoff be executed at the start of reschedule_actor? Like if !first_reschedule { /* do backoff */ }

Context: https://rivet-gg.slack.com/archives/D02R3KVNKHN/p1732672061669669?thread_ts=1732671085.517149&cid=D02R3KVNKHN

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actors always retry immediately the first time, then backoff kicks in: https://rivet-gg.slack.com/archives/D02R3KVNKHN/p1732672128678809?thread_ts=1732671085.517149&cid=D02R3KVNKHN

@MasterPtato MasterPtato force-pushed the 11-26-feat_make_actors_reschedule_indefinitely_with_backoff branch 4 times, most recently from a25cdcb to 67ce43f Compare December 5, 2024 20:09
@MasterPtato MasterPtato force-pushed the 11-26-feat_make_actors_reschedule_indefinitely_with_backoff branch from 67ce43f to 2084acf Compare December 5, 2024 20:12
@graphite-app Graphite App
Copy link
Contributor

graphite-app bot commented Dec 9, 2024 •
edited
Loading

Merge activity

  • Dec 9, 1:37 PM EST: A user added this pull request to the Graphite merge queue.
  • Dec 9, 1:38 PM EST: CI is running for this PR on a draft PR: #1552
  • Dec 9, 1:42 PM EST: A user merged this pull request with the Graphite merge queue via draft PR: #1552.

NathanFlurry pushed a commit that referenced this pull request Dec 9, 2024
@MasterPtato
feat: make actors reschedule indefinitely with backoff (#1475)
04ec2bd 
<!-- Please make sure there is an issue that this PR is correlated to. -->
Fixes RVT-4196
Fixes RVT-4179

**Implemented but untested**
## Changes

<!-- If there are frontend changes, please include screenshots. -->
@graphite-app graphite-app bot closed this Dec 9, 2024
@graphite-app graphite-app bot deleted the 11-26-feat_make_actors_reschedule_indefinitely_with_backoff branch December 9, 2024 18:42
@github-actions github-actions bot mentioned this pull request Dec 9, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@NathanFlurry NathanFlurry NathanFlurry approved these changes

Assignees

@NathanFlurry NathanFlurry

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@MasterPtato @NathanFlurry