Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove CausalOrdering from task-run-recorder #15244

Merged
merged 3 commits into from
Sep 5, 2024

Conversation

jakekaplan
Copy link
Contributor

@jakekaplan jakekaplan commented Sep 5, 2024

The task-run-recorder consumer takes in task run events, converts them into task runs and task run states, and inserts them into the database. Previously we ensured that these inserts are forced to be done in order of the events occurring through the CausalOrdering.

In practice, the only benefit is that if someone were to observe the actual task run state inserts, a COMPLETED row would never exist before a RUNNING row. Additionally as long as we are using a queue that is always ordered (currently the only option is an in memory queue), events will never come out of order. Additionally the final state of the task run will always be reported correctly.

This PR removes the ordering as there is no real benefit and it is likely adding unnecessary strain to the server.

Copy link
Collaborator

@chrisguidry chrisguidry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That test makes this easy to trust!

record_task_run_event_mock.assert_awaited_with(event)
@pytest.mark.parametrize(
"event_order",
list(permutations(["PENDING", "RUNNING", "COMPLETED"])),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥

@pytest.mark.parametrize(
"event_order",
list(permutations(["PENDING", "RUNNING", "COMPLETED"])),
ids=lambda x: "->".join(x),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

Copy link

codspeed-hq bot commented Sep 5, 2024

CodSpeed Performance Report

Merging #15244 will not alter performance

Comparing remove-causal-ordering-from-task-run-recorder (f52cf51) with main (76c4323)

Summary

✅ 3 untouched benchmarks

Copy link
Member

@cicdw cicdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

niiiiice thank you!

@cicdw
Copy link
Member

cicdw commented Sep 5, 2024

Related to ongoing investigations in #15153

@cicdw cicdw added enhancement An improvement of an existing feature performance Related to an optimization or performance improvement labels Sep 5, 2024
@cicdw cicdw merged commit 2b994af into main Sep 5, 2024
30 checks passed
@cicdw cicdw deleted the remove-causal-ordering-from-task-run-recorder branch September 5, 2024 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature performance Related to an optimization or performance improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants