Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

streamingest: add custom ctx to support cutting over #60536

Closed
wants to merge 2 commits into from
Closed

streamingest: add custom ctx to support cutting over #60536

wants to merge 2 commits into from

Conversation

adityamaru
Copy link
Contributor

@adityamaru adityamaru commented Feb 12, 2021 •
edited
Loading

This change adds logic to leverage context cancelation as a means of
trigerring the RevertRange logic that is run when the job is requested
to cutover.

This change also adds a poller to check if the job has been marked for
completion.

Release note: None

@adityamaru adityamaru requested review from pbardea and a team February 12, 2021 17:31
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@adityamaru
Copy link
Contributor Author

adityamaru commented Feb 12, 2021 •
edited
Loading

@pbardea the second commit has the relevant changes. The first commit is from #59588. This commit also adds very rudimentary ctx.Done() error handling support to the stream ingestion processor. Those will be finalized by your work in #60160.

@adityamaru
Copy link
Contributor Author

Closing in favour of #60763.

@adityamaru adityamaru closed this Feb 19, 2021
@adityamaru adityamaru reopened this Feb 22, 2021
@adityamaru adityamaru changed the title [WIP] streamingest: add custom ctx to support cutting over streamingest: add custom ctx to support cutting over Feb 23, 2021
pbardea and others added 2 commits February 23, 2021 00:03
@pbardea @adityamaru
streamingest: add ctx cancelation to support cutting over
90ec8f3 
This change adds logic to leverage context cancelation as a means of
trigerring the RevertRange logic that is run when the job is requested
to cutover.

This change also adds a poller to check if the job has been marked for
completion.

Release note: None
@adityamaru
streamingest: add OnFailOrCancel ClearRange logic to stream ingestion
ac660ae 
When the job fails, we must ClearRange the target keyspace being
ingested into.

Release note: None
@adityamaru
Copy link
Contributor Author

TestStreamIngestionJobWithRandomClient is currently skipped cause it takes ~30seconds but I unskipped it while testing the custom ctx cancelation. The majority of the time is spent waiting for the job to transition to the Canceled state so wondering if y'all have any ideas to restructure this?

@adityamaru
Copy link
Contributor Author

Closing in favour of #61001, Reviewable doesn't work on this PR.

@adityamaru adityamaru closed this Feb 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pbardea pbardea Awaiting requested review from pbardea

@miretskiy miretskiy Awaiting requested review from miretskiy

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@adityamaru @cockroach-teamcity @pbardea