Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: serialize table operations by lock rather than queue #814

Merged

Conversation

ShiKaiWi
Copy link
Member

@ShiKaiWi ShiKaiWi commented Apr 9, 2023

Which issue does this PR close?

Prepare for #800

Rationale for this change

Currently, the operations on tables are serialized by the worker queue, which has a serious disadvantage -- no work steal, which may leading to hungry of some tables' operations because of one big table.

This PR will replace the worker queue with a simple lock for every table.

What changes are included in this PR?

  • Replace the worker queue with lock for every table;

Are there any user-facing changes?

Remove the configs for the write workers.

How does this change test

Existing tests.

@ShiKaiWi ShiKaiWi force-pushed the feat-recover-table-separate-from-writer-handle branch from 196a588 to 67543cc Compare April 9, 2023 15:32
@ShiKaiWi ShiKaiWi changed the title feat: recover table separate from writer handle feat: serialize table operations by lock rather than queue Apr 10, 2023
@codecov-commenter
Copy link

Codecov Report

Merging #814 (9fa1b8b) into main (a8ed20e) will decrease coverage by 0.33%.
The diff coverage is 79.79%.

❗ Current head 9fa1b8b differs from pull request most recent head 67543cc. Consider uploading reports for the commit 67543cc to get more accurate results

@@            Coverage Diff             @@
##             main     #814      +/-   ##
==========================================
- Coverage   68.02%   67.69%   -0.33%     
==========================================
  Files         300      300              
  Lines       47229    46510     -719     
==========================================
- Hits        32129    31487     -642     
+ Misses      15100    15023      -77     
Impacted Files Coverage Δ
analytic_engine/src/instance/close.rs 0.00% <0.00%> (ø)
analytic_engine/src/instance/create.rs 90.90% <ø> (-1.65%) ⬇️
analytic_engine/src/manifest/details.rs 95.62% <ø> (ø)
common_util/src/runtime/mod.rs 96.29% <ø> (ø)
components/skiplist/src/list.rs 98.87% <ø> (-0.46%) ⬇️
server/src/handlers/prom.rs 68.44% <ø> (ø)
server/src/handlers/query.rs 0.00% <ø> (ø)
table_engine/src/engine.rs 53.52% <ø> (ø)
analytic_engine/src/compaction/scheduler.rs 72.60% <14.28%> (-1.81%) ⬇️
analytic_engine/src/instance/engine.rs 61.15% <50.00%> (-2.88%) ⬇️
... and 14 more

... and 1 file with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@ShiKaiWi ShiKaiWi force-pushed the feat-recover-table-separate-from-writer-handle branch 3 times, most recently from 895dfdd to 2bfad51 Compare April 11, 2023 13:08
analytic_engine/src/instance/serializer.rs Outdated Show resolved Hide resolved
analytic_engine/src/instance/serializer.rs Outdated Show resolved Hide resolved
analytic_engine/src/instance/serializer.rs Outdated Show resolved Hide resolved
analytic_engine/src/instance/serializer.rs Outdated Show resolved Hide resolved
@ShiKaiWi ShiKaiWi force-pushed the feat-recover-table-separate-from-writer-handle branch from 2bfad51 to 243156b Compare April 13, 2023 07:25
analytic_engine/src/instance/write.rs Outdated Show resolved Hide resolved
analytic_engine/src/instance/serializer.rs Outdated Show resolved Hide resolved
analytic_engine/src/instance/serializer.rs Outdated Show resolved Hide resolved
@ShiKaiWi ShiKaiWi force-pushed the feat-recover-table-separate-from-writer-handle branch from 76e7342 to dc70cab Compare April 13, 2023 11:31
Copy link
Contributor

@jiacai2050 jiacai2050 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShiKaiWi ShiKaiWi added this pull request to the merge queue Apr 17, 2023
Merged via the queue into apache:main with commit f5b0bc9 Apr 17, 2023
@ShiKaiWi ShiKaiWi deleted the feat-recover-table-separate-from-writer-handle branch April 17, 2023 05:50
chunshao90 pushed a commit to chunshao90/ceresdb that referenced this pull request May 15, 2023
* feat: replace write group with lock

* feat: replace condvar with watch channel

* refactor: rename some names

* refactor: rename TableOpSerializer to TableOpSerialExecutor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants