-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support backfilling of secondary index #448
Labels
area/docdb
YugabyteDB core features
kind/enhancement
This is an enhancement of an existing feature
priority/medium
Medium priority issue
roadmap-tracking-issue
This issue tracks a major roadmap item, and usually appears in the roadmap list.
Milestone
Comments
Any ETA for this? |
@YourTechBud sorry if we did not follow-up on this. We will be actively working on this for the next release! |
This was referenced Jul 25, 2019
This was referenced Sep 13, 2019
amitanandaiyer
added a commit
that referenced
this issue
Jan 28, 2020
Summary: High level design doc : https://github.com/yugabyte/yugabyte-db/blob/master/architecture/design/docdb-index-backfill.md This diff implements the master side changes to be done for a create index. Populating the index tablet at the appropriate time is implemented in D7104. - Implement a 4 phase index creation at the master to safely create an index and backfill the data in an online manner. Index is created with the INDEX_PERM_DELETE_ONLY permission, then updated to INDEX_PERM_WRITE_AND_DELETE, then INDEX_PERM_BACKFILLING; and finally, after the backfill is complete, set to INDEX_PERM_READ_WRITE_AND_DELETE. - At each phase, the master waits for all the tablets of the indexed table to be updated before moving on to the next phase. - Ensure that the CQL proxy/Tablet Server respects the IndexPermission and only performs the operations that are permitted. - Ensure that master failover in the middle of a index backfill is handled. A new flag `disable_index_backfill` is introduced to fence the new 4-stage index create process. This defaults to `true` right now -- until all the remaining parts needed for the backfill land. Test Plan: ./yb_build.sh --cxx-test cassandra_cpp_driver-test --gtest_filter CppCassandraDriverTest.*TestCreateIndex* ./yb_build.sh --cxx-test master_failover-itest --gtest_filter MasterFailoverTestIndexCreation.*Index* Reviewers: mihnea, sergei, hector, bogdan, rahuldesirazu, mikhail Reviewed By: rahuldesirazu, mikhail Subscribers: mikhail, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D7077
amitanandaiyer
added a commit
that referenced
this issue
Feb 6, 2020
…ng non-unique indices. Summary: High level design doc : https://github.com/yugabyte/yugabyte-db/blob/master/architecture/design/docdb-index-backfill.md Depends on D7077 for the master side implementation This diff implements: 2. BackFill implementation at the tablet level. - scan the main table as of the specified time and populate the entries for the indexed tables. - apply backfill write op(s) to the index tables at the desired hybrid-time/backfill-time. - support backfill time: tunnel it through raft and use it while applying to docdb. Handled in the next diffs: 3) Unique Indices D7668: - test create index failure on unique violation - implement reverse scan for uniqueness check - disable major compactions from cleaning up the delete markers during backfill. - Split unique index write batches into multiple chunks upon collision 4) Perf improvements. Test Plan: ybd --cxx-test cassandra_cpp_driver-test --gtest_filter CppCassandraDriverTest.*TestTableCreateIndex* Reviewers: timur, sergei, bogdan, mihnea, hector Reviewed By: hector Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D7104
This was referenced Feb 10, 2020
@robertpang |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
area/docdb
YugabyteDB core features
kind/enhancement
This is an enhancement of an existing feature
priority/medium
Medium priority issue
roadmap-tracking-issue
This issue tracks a major roadmap item, and usually appears in the roadmap list.
Jira Link: DB-2465
Upon adding new indexes to a table that already has data, this feature would enable building these indexes in an online manner, while continuing to serve other traffic. Note that this feature should work across both YSQL and YCQL APIs. It should support:
Prerequisites
Phase 1 - simple index backfill
IN PROGRESS
(target v2.2) #2301Phase 2 - manageability features
Phase 3 - constraints and unique indexes
Phase 4 - Other misc improvements
The text was updated successfully, but these errors were encountered: