Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docdb] Add a limit on number of outstanding tablet splits #6667

Closed
bmatican opened this issue Dec 15, 2020 · 2 comments
Closed

[docdb] Add a limit on number of outstanding tablet splits #6667

bmatican opened this issue Dec 15, 2020 · 2 comments
Assignees
Labels
area/docdb YugabyteDB core features

Comments

@bmatican
Copy link
Contributor

Porting point c) from #6111 (comment)

Once we have #5647, we will have the info in the master, of outstanding tablet splits, so we should also be able to limit how many tablets are still being split, so we could reliably add a limit for this. This would help to minimize the amount of work in parallel in a cluster with many tablets that will end up being split.

@bmatican
Copy link
Contributor Author

One thing we've discussed offline, is that we can do this probably easier on the TS side, to make sure we don't send extra info to the master, about new tablets to split, if we have not finished compacting all the currently being split tablets.

Notes from @ttyusupov

if (!tablet || !tablet->metadata() ||
    tablet->table_type() == TableType::TRANSACTION_STATUS_TABLE_TYPE ||
    // TODO(tsplit): Tablet splitting for colocated tables is not supported.
    tablet->metadata()->colocated() ||
    tablet->metadata()->tablet_data_state() != tablet::TabletDataState::TABLET_DATA_READY ||
    tablet->GetCurrentVersionSstFilesSize() < split_size_threshold ||
    // TODO(tsplit): We don't split not yet fully compacted post-split tablets for now, since
    // detecting effective middle key and tablet size for such tablets is not yet implemented.
    tablet->MightStillHaveParentDataAfterSplit()) {
  VLOG_WITH_FUNC(3) << Format(
      "Skipping tablet: $0, data state: $1, SST files size: $2, has key bounds: $3, has been "
      "fully compacted: $4",
      tablet->tablet_id(),
      tablet->metadata() ? AsString(tablet->metadata()->tablet_data_state()) : "NONE",
      tablet->GetCurrentVersionSstFilesSize(), tablet->doc_db().key_bounds->IsInitialized(),
      tablet->metadata()->has_been_fully_compacted());
  continue;
}

To not just skip tablets with MightStillHaveParentDataAfterSplit , but if we have them - return from this function, so we don’t trigger new split

@bmatican
Copy link
Contributor Author

was discussing with @robertsami , that if we move the control of when to do splits, to the master, then this task will be even easier

so @mikhpolitov let's hold off on this a bit, until Rob wraps up the master side work then!

robertsami added a commit that referenced this issue May 25, 2021
…ate all split decisions to yb-master

Summary:
In order to adopt the more complex multi-phase split rate strategy outlined in
#8039, we must defer
triggering of tablet splits to the master entirely. This revision moves that control to the master and adds a
basic test for automatic tablet splitting based on the existing single split size threshold mechanism.

On the master side, we inject into the tablet metrics drive info processing to synchronously
qualify/disqualify tablets as split candidates. Qualified candidates are added to a queue and split in a
background thread in the order they are seen. This approach should lend itself well to future work which will
control tablet split rates more intelligently (see #6667).

Test Plan: ybd --cxx-test integration-tests_tablet-split-itest --gtest_filter TabletSplitITest.AutomaticTabletSplitting

Reviewers: timur

Reviewed By: timur

Subscribers: zyu, amitanand, azhao, mbautin, mpolitov, ybase, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D11562
YintongMa pushed a commit to YintongMa/yugabyte-db that referenced this issue May 26, 2021
…nd delegate all split decisions to yb-master

Summary:
In order to adopt the more complex multi-phase split rate strategy outlined in
yugabyte#8039, we must defer
triggering of tablet splits to the master entirely. This revision moves that control to the master and adds a
basic test for automatic tablet splitting based on the existing single split size threshold mechanism.

On the master side, we inject into the tablet metrics drive info processing to synchronously
qualify/disqualify tablets as split candidates. Qualified candidates are added to a queue and split in a
background thread in the order they are seen. This approach should lend itself well to future work which will
control tablet split rates more intelligently (see yugabyte#6667).

Test Plan: ybd --cxx-test integration-tests_tablet-split-itest --gtest_filter TabletSplitITest.AutomaticTabletSplitting

Reviewers: timur

Reviewed By: timur

Subscribers: zyu, amitanand, azhao, mbautin, mpolitov, ybase, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D11562
@bmatican bmatican assigned hulien22 and unassigned mikhpolitov Jul 19, 2021
hulien22 added a commit that referenced this issue Aug 5, 2021
Summary:
Adding in `outstanding_tablet_split_limit` flag to limit the number of outstanding tablet splits that can be processed at the same time. In order to enforce this limit, also adding in a `processing_tablets_` set to `TabletSplitManager`. This set gets added to whenever we start a splittablet operation, and gets removed by either:
- when one of its children tablets reports and no longer has any orphaned split data (ie the split was succesful)
- or if one of the async tasks to start the split fail, in which case we call `CatalogManager::RemoveFailedProcessingTabletSplit`

Test Plan:
```
ybd --cxx-test integration-tests_tablet-split-itest --gtest_filter AutomaticTabletSplitITest.LimitNumberOfOutstandingTabletSplits
```

Reviewers: rsami

Reviewed By: rsami

Subscribers: ybase, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D12308
radekg pushed a commit to radekg/yugabyte-db that referenced this issue Aug 12, 2021
…lits

Summary:
Adding in `outstanding_tablet_split_limit` flag to limit the number of outstanding tablet splits that can be processed at the same time. In order to enforce this limit, also adding in a `processing_tablets_` set to `TabletSplitManager`. This set gets added to whenever we start a splittablet operation, and gets removed by either:
- when one of its children tablets reports and no longer has any orphaned split data (ie the split was succesful)
- or if one of the async tasks to start the split fail, in which case we call `CatalogManager::RemoveFailedProcessingTabletSplit`

Test Plan:
```
ybd --cxx-test integration-tests_tablet-split-itest --gtest_filter AutomaticTabletSplitITest.LimitNumberOfOutstandingTabletSplits
```

Reviewers: rsami

Reviewed By: rsami

Subscribers: ybase, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D12308
Master components automation moved this from To do to Done Sep 8, 2021
Tablet splitting automation moved this from To do to Done Sep 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docdb YugabyteDB core features
Projects
YBase features
  
Awaiting triage
Development

No branches or pull requests

4 participants