Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introducing MAPQ: Multi-tenant, Auto-partitioned, Persistent Queue #6132

Merged
merged 9 commits into from
Jun 24, 2024
Merged

Introducing MAPQ: Multi-tenant, Auto-partitioned, Persistent Queue #6132

merged 9 commits into from
Jun 24, 2024

Conversation

taylanisikdemir
Copy link
Member

@taylanisikdemir taylanisikdemir commented Jun 13, 2024 •
edited
Loading

What changed?
See the README

How did you test it?
Unit tests are added to cover basic initialization, policy based tree construction.

@codecov Codecov
Copy link

codecov bot commented Jun 13, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 81.19403% with 63 lines in your changes missing coverage. Please review.

Project coverage is 72.36%. Comparing base (83ebf7a) to head (18c1e1c).

❗ Current head 18c1e1c differs from pull request most recent head 2643b58

Please upload reports for the commit 2643b58 to get more accurate results.

Additional details and impacted files
Files Coverage Δ
common/mapq/dispatcher/dispatcher.go 90.00% <90.00%> (ø)
common/mapq/mapq.go 93.75% <93.75%> (ø)
common/mapq/types/item.go 91.66% <91.66%> (ø)
common/mapq/types/policy_collection.go 94.91% <94.91%> (ø)
common/mapq/client_impl.go 72.72% <72.72%> (ø)
common/mapq/types/policy.go 61.29% <61.29%> (ø)
common/mapq/tree/queue_tree.go 74.64% <74.64%> (ø)
common/mapq/tree/queue_tree_node.go 76.31% <76.31%> (ø)

... and 23 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 83ebf7a...2643b58. Read the comment docs.

@coveralls
Copy link

coveralls commented Jun 13, 2024 •
edited
Loading

Pull Request Test Coverage Report for Build 01900ef3-4f9c-48cc-b38e-39c97c85ad36

Details

  • 370 of 447 (82.77%) changed or added relevant lines in 8 files are covered.
  • 754 unchanged lines in 16 files lost coverage.
  • Overall coverage increased (+0.03%) to 71.487%
Changes Missing Coverage Covered Lines Changed/Added Lines %
common/mapq/mapq.go 41 43 95.35%
common/mapq/types/item.go 33 36 91.67%
common/mapq/types/policy_collection.go 69 72 95.83%
common/mapq/dispatcher/dispatcher.go 24 29 82.76%
common/mapq/client_impl.go 18 30 60.0%
common/mapq/types/policy.go 29 44 65.91%
common/mapq/tree/queue_tree.go 71 89 79.78%
common/mapq/tree/queue_tree_node.go 85 104 81.73%
Files with Coverage Reduction New Missed Lines %
service/matching/tasklist/task_list_manager.go 2 77.05%
service/history/replication/task_processor.go 2 82.76%
service/history/task/transfer_active_task_executor.go 2 72.77%
common/persistence/visibility_single_manager.go 2 99.36%
service/matching/tasklist/matcher.go 2 89.35%
service/matching/tasklist/task_reader.go 2 77.72%
common/persistence/historyManager.go 2 66.67%
common/persistence/nosql/nosql_task_store.go 3 85.52%
service/history/task/fetcher.go 4 86.08%
common/asyncworkflow/queue/consumer/default_consumer.go 6 93.82%
Totals Coverage Status
Change from base Build 01900d42-a47b-496e-814a-80e3cc64dff7: 0.03%
Covered Lines: 106902
Relevant Lines: 149540
💛 - Coveralls

@coveralls
Copy link

coveralls commented Jun 13, 2024 •
edited
Loading

Pull Request Test Coverage Report for Build 019012ba-cdeb-477a-b161-0bed5e192bb7

Details

  • 370 of 447 (82.77%) changed or added relevant lines in 8 files are covered.
  • 772 unchanged lines in 20 files lost coverage.
  • Overall coverage increased (+0.02%) to 71.478%
Changes Missing Coverage Covered Lines Changed/Added Lines %
common/mapq/mapq.go 41 43 95.35%
common/mapq/types/item.go 33 36 91.67%
common/mapq/types/policy_collection.go 69 72 95.83%
common/mapq/dispatcher/dispatcher.go 24 29 82.76%
common/mapq/client_impl.go 18 30 60.0%
common/mapq/types/policy.go 29 44 65.91%
common/mapq/tree/queue_tree.go 71 89 79.78%
common/mapq/tree/queue_tree_node.go 85 104 81.73%
Files with Coverage Reduction New Missed Lines %
common/task/weighted_round_robin_task_scheduler.go 2 89.05%
service/matching/tasklist/task_list_manager.go 2 77.05%
common/task/fifo_task_scheduler.go 2 87.63%
tools/cli/admin_db_decode_thrift.go 2 70.51%
service/history/replication/task_processor.go 2 82.76%
service/history/task/transfer_active_task_executor.go 2 72.77%
common/persistence/visibility_single_manager.go 2 99.36%
common/persistence/historyManager.go 2 66.67%
service/history/task/task.go 3 84.81%
common/persistence/statsComputer.go 3 98.21%
Totals Coverage Status
Change from base Build 01900d42-a47b-496e-814a-80e3cc64dff7: 0.02%
Covered Lines: 106888
Relevant Lines: 149540
💛 - Coveralls

@coveralls
Copy link

coveralls commented Jun 14, 2024 •
edited
Loading

Pull Request Test Coverage Report for Build 0190141d-2c12-4d43-be38-0107cc2cf921

Details

  • 377 of 442 (85.29%) changed or added relevant lines in 8 files are covered.
  • 772 unchanged lines in 18 files lost coverage.
  • Overall coverage increased (+0.02%) to 71.472%
Changes Missing Coverage Covered Lines Changed/Added Lines %
common/mapq/dispatcher/dispatcher.go 25 27 92.59%
common/mapq/mapq.go 41 43 95.35%
common/mapq/types/item.go 33 36 91.67%
common/mapq/types/policy_collection.go 69 72 95.83%
common/mapq/client_impl.go 24 30 80.0%
common/mapq/types/policy.go 29 41 70.73%
common/mapq/tree/queue_tree.go 71 89 79.78%
common/mapq/tree/queue_tree_node.go 85 104 81.73%
Files with Coverage Reduction New Missed Lines %
service/history/shard/context.go 2 78.36%
service/matching/tasklist/db.go 2 73.23%
common/peerprovider/ringpopprovider/config.go 2 81.58%
service/matching/tasklist/task_list_manager.go 2 77.05%
service/history/replication/task_processor.go 2 82.76%
common/persistence/visibility_single_manager.go 2 99.36%
service/matching/tasklist/matcher.go 2 89.35%
common/log/tag/tags.go 3 50.46%
common/persistence/nosql/nosql_task_store.go 3 85.52%
common/archiver/filestore/historyArchiver.go 4 80.95%
Totals Coverage Status
Change from base Build 01900d42-a47b-496e-814a-80e3cc64dff7: 0.02%
Covered Lines: 106875
Relevant Lines: 149535
💛 - Coveralls

Shaddoll
Shaddoll reviewed Jun 14, 2024
View reviewed changes
common/mapq/types/item.go Outdated Show resolved Hide resolved
@taylanisikdemir taylanisikdemir changed the title [WIP] Introducing MAPQ: Multi-tenant, Auto-partitioned, Persistent Queue Introducing MAPQ: Multi-tenant, Auto-partitioned, Persistent Queue Jun 24, 2024
@taylanisikdemir taylanisikdemir marked this pull request as ready for review June 24, 2024 17:48
@coveralls
Copy link

coveralls commented Jun 24, 2024 •
edited
Loading

Pull Request Test Coverage Report for Build 01904b76-7f40-4850-b643-9077d580a9ce

Details

  • 377 of 442 (85.29%) changed or added relevant lines in 8 files are covered.
  • 52 unchanged lines in 12 files lost coverage.
  • Overall coverage increased (+0.01%) to 71.547%
Changes Missing Coverage Covered Lines Changed/Added Lines %
common/mapq/dispatcher/dispatcher.go 25 27 92.59%
common/mapq/mapq.go 41 43 95.35%
common/mapq/types/item.go 33 36 91.67%
common/mapq/types/policy_collection.go 69 72 95.83%
common/mapq/client_impl.go 24 30 80.0%
common/mapq/types/policy.go 29 41 70.73%
common/mapq/tree/queue_tree.go 71 89 79.78%
common/mapq/tree/queue_tree_node.go 85 104 81.73%
Files with Coverage Reduction New Missed Lines %
common/task/weighted_round_robin_task_scheduler.go 2 89.05%
common/task/fifo_task_scheduler.go 2 84.54%
service/matching/tasklist/matcher.go 2 90.91%
service/matching/tasklist/task_reader.go 2 75.33%
service/history/handler/handler.go 3 96.2%
service/history/queue/timer_queue_processor_base.go 3 77.87%
common/persistence/statsComputer.go 3 98.21%
common/archiver/filestore/historyArchiver.go 4 80.95%
service/history/task/transfer_standby_task_executor.go 6 87.35%
service/frontend/api/handler.go 6 75.62%
Totals Coverage Status
Change from base Build 01903cd7-c1ac-49f3-a7a4-fe9da6c16ce7: 0.01%
Covered Lines: 107092
Relevant Lines: 149680
💛 - Coveralls

@taylanisikdemir
Copy link
Member Author

Merging to unblock splitting of development. Feel free to add comments. Everything in this Pr is subject to change in follow up iterations.

@taylanisikdemir taylanisikdemir merged commit de1aafc into cadence-workflow:master Jun 24, 2024
19 checks passed
@taylanisikdemir taylanisikdemir deleted the taylan/queue_playground branch June 24, 2024 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Shaddoll Shaddoll Shaddoll approved these changes

@neil-xie neil-xie Awaiting requested review from neil-xie neil-xie is a code owner

@davidporter-id-au davidporter-id-au Awaiting requested review from davidporter-id-au davidporter-id-au is a code owner

@Groxx Groxx Awaiting requested review from Groxx Groxx is a code owner

@shijiesheng shijiesheng Awaiting requested review from shijiesheng shijiesheng is a code owner

@agautam478 agautam478 Awaiting requested review from agautam478

@jakobht jakobht Awaiting requested review from jakobht jakobht is a code owner

@3vilhamster 3vilhamster Awaiting requested review from 3vilhamster 3vilhamster is a code owner

@sankari165 sankari165 Awaiting requested review from sankari165 sankari165 is a code owner

@dkrotx dkrotx Awaiting requested review from dkrotx dkrotx is a code owner

@demirkayaender demirkayaender Awaiting requested review from demirkayaender demirkayaender is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@taylanisikdemir @coveralls @Shaddoll