[Phase 2][YSQL][Colocation] yb-admin create_database_snapshot can't deal with tablegroups #11632

frozenspider · 2022-03-02T19:16:19Z

Jira Link: DB-374

Description

On latest master (1f88fc7):

CREATE DATABASE yb1;
\c yb1

CREATE TABLEGROUP tg1;
CREATE TABLEGROUP tg2;

CREATE TABLE test_tbl (
  h INT PRIMARY KEY,
  a INT,
  b FLOAT CONSTRAINT test_tbl_uniq UNIQUE
) TABLEGROUP tg1;

CREATE INDEX test_tbl_tg2_idx  ON test_tbl (a ASC) TABLEGROUP tg2;
CREATE INDEX test_tbl_notg_idx ON test_tbl (a ASC) NO TABLEGROUP;

INSERT INTO test_tbl (h, a, b) VALUES (1, 101, 3.14);

Now try to take a snapshot:

19:15 ~/code/yugabyte [master] $ ./build/latest/bin/yb-admin create_database_snapshot ysql.yb1
Error running create_database_snapshot: Not found (yb/master/catalog_manager.cc:4435): Unable to create snapshot of database: yb1: Table with identifier 00004000000030008000000000000000.colocated.parent.uuid not found: OBJECT_NOT_FOUND (master error 3)

The text was updated successfully, but these errors were encountered:

def- · 2022-08-04T17:08:08Z

Also seeing this with LST trying to test backup/restore, disabling tablegroups for LST with backup/restore scenario.

yifanguan · 2022-08-06T17:27:58Z

Backup/restore of a database containing tablegroup fails due to the failure of yb-admin create_database_snapshot <keyspace>.
The above example can be simplified as:

CREATE TABLEGROUP tg1;

CREATE TABLE test_tbl (
  a INT
) TABLEGROUP tg1;

yifanguan · 2022-08-06T17:31:45Z

@def- Do you still have the logs? When you disable tablegroups for LST, do you still enable colocated databases?
Does the table identifier end with colocated.parent.uuid, (e.g. Table with identifier 000033e8000030008000000000000000.colocated.parent.uuid)? Thanks!

def- · 2022-08-07T11:22:11Z

No colocation used currently, but this failure happened with tablegroups (as a consequence I disabled tablegroups in backup/restore tests). Logs should still be available at https://portal.dev.yugabyte.com/universes/edaf84f8-508e-4fc0-840f-6b91d37ad21c

sanketkedia · 2022-09-20T17:25:15Z

I think @msun07 is fixing it as part of https://phabricator.dev.yugabyte.com/D19398

yifanguan · 2022-09-20T18:36:50Z

@sanketkedia @msun07 I'm also working on this including the whole backup process (create_database_snapshot, import_snapshot, platform backup script change): https://phabricator.dev.yugabyte.com/D18820, not from PITR perspective though.
I see there are some similar overlap changes in the function CollectTable(). This is fine. I can rebase onto D19398 if it is landed soon.

Summary: This diff supports backup and restore for tablegroup. Problem: Tablegroup parent name is in the format: <tablegroup_id>.tablegroup.parent.tablename, and its uuid is in the format: <tablegroup_id>.tablegroup.parent.uuid. In general, for a restore process, the table matching during `import_snapshot` step is based on table uuid or table name. Either one of them should remain the same, or we need a way to calculate the table's new name or new id on a new cluster based on the old table name or id stored in SnapshotInfoPB. Before this diff, it is hard to match a tablegroup parent table during `import_snapshot` because a tablegroup oid generated on postgres side on a new cluster can be different from the oid on the old cluster. Consider the following example: ``` CREATE TABLE tbl (a INT); -- OID: 16384 CREATE TABLEGROUP tg1; -- OID: 16387 CREATE TABLE tbl2 (a INT) TABLEGROUP tg1; -- OID: 16388 ``` Recreating these objects using `ysql_dump` follows the order: ``` CREATE TABLEGROUP tg1; -- OID: 16384 CREATE TABLE tbl (a INT); -- OID: 16385 CREATE TABLE tbl2 (a INT) TABLEGROUP tg1; -- OID: 16388 ``` With the tabelgroup parent table name/uuid format, we cannot match the parent table based on its name/uuid, and cannot calculate parent table name/uuid on the new cluster. To address the overall backup problem, major changes in this diff are: - Preserve tablegroup oid in `ysql_dump`. Use the PG binary upgrade style approach to preserve tablegroup oid so that tablegroup oid of a tablegroup remains the same on a new cluster. - Add `yb-admin import_snapshot` support for tablegroup parent table. Calculate tablegroup parent table uuid based on old tablegroup parent table uuid from old cluster. The logic is: (1) old_tablegroup_parent_table_uuid = function(old_namespace_oid, old_tablegroup_oid) (2) We know the uuid matching between the old namespace and the new namespace from an earlier step in `import_snapshot`, and tablegroup oid remains the same. (3) So new_tablegroup_parent_table_uuid = function(new_namespace_oid, old_tablegroup_oid) Sample output of import_snapshot for the above exmaple: ``` Read snapshot meta file Importing snapshot 40b4e757-797d-4ea2-be80-fc51810f92dc (COMPLETE) Target imported YSQL database name: yugabyte YSQL database being imported: yugabyte Table type: table Table being imported: yugabyte.tbl Table type: colocated table Colocated table being imported: yugabyte.000033e8000030008000000000004003.tablegroup.parent.tablename Table type: colocated table Colocated table being imported: yugabyte.tbl2 Successfully applied snapshot. Object Old ID New ID Keyspace 000033e8000030008000000000000000 000033e8000030008000000000000000 Table 000033e8000030008000000000004000 000033e8000030008000000000004000 Tablet 0 e9746f31b49d45de9ecbf3f78d3d4296 a9a5da1c2c274801b5e4d7bbe248cebc Tablet 1 c8ded40dbe2c4d55bdb7b2c499639071 3a495f69527245b08b894a244a7ccd58 Keyspace 000033e8000030008000000000000000 000033e8000030008000000000000000 ParentColocatedTable 000033e8000030008000000000004003.tablegroup.parent.uuid 000033e8000030008000000000004003.tablegroup.parent.uuid Tablet 0 37e5ee5686194284956b9aeafe439767 186254847ce8454aa1371056f58da796 Keyspace 000033e8000030008000000000000000 000033e8000030008000000000000000 ColocatedTable 000033e8000030008000000000004004 000033e8000030008000000000004001 Snapshot 40b4e757-797d-4ea2-be80-fc51810f92dc aee3464b-34f9-480c-b7b8-367426dfbec8 ``` - Add tablegroup handling in `yb_backup.py`. In yb_backup.py, add support to parse `import_snapshot` output having tablegroup parent table so that snapshot directories can be downloaded correctly for tablegroup parent table. Test Plan: ./yb_build.sh --java-test 'org.yb.pgsql.TestYsqlDump' ./yb_build.sh --java-test 'org.yb.pgsql.TestYbBackup#testTablegroup' ./yb_build.sh --java-test 'org.yb.pgsql.TestYbBackup#testMultipleTablegroups' Reviewers: mihnea, jhe, msun, myang, alex, oleg, tverona Reviewed By: alex, oleg, tverona Subscribers: zdrudi, jenkins-bot, yugaware, yql, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D18820

yifanguan · 2022-11-07T21:23:28Z

yb-admin create_database_snapshot support is resolved by commit 98980ce
Whole tablegroup backup && restore is supported by commit 60ffefc.

Summary: This diff supports backup and restore for tablegroup. Problem: Tablegroup parent name is in the format: <tablegroup_id>.tablegroup.parent.tablename, and its uuid is in the format: <tablegroup_id>.tablegroup.parent.uuid. In general, for a restore process, the table matching during `import_snapshot` step is based on table uuid or table name. Either one of them should remain the same, or we need a way to calculate the table's new name or new id on a new cluster based on the old table name or id stored in SnapshotInfoPB. Before this diff, it is hard to match a tablegroup parent table during `import_snapshot` because a tablegroup oid generated on postgres side on a new cluster can be different from the oid on the old cluster. Consider the following example: ``` CREATE TABLE tbl (a INT); -- OID: 16384 CREATE TABLEGROUP tg1; -- OID: 16387 CREATE TABLE tbl2 (a INT) TABLEGROUP tg1; -- OID: 16388 ``` Recreating these objects using `ysql_dump` follows the order: ``` CREATE TABLEGROUP tg1; -- OID: 16384 CREATE TABLE tbl (a INT); -- OID: 16385 CREATE TABLE tbl2 (a INT) TABLEGROUP tg1; -- OID: 16388 ``` With the tabelgroup parent table name/uuid format, we cannot match the parent table based on its name/uuid, and cannot calculate parent table name/uuid on the new cluster. To address the overall backup problem, major changes in this diff are: - Preserve tablegroup oid in `ysql_dump`. Use the PG binary upgrade style approach to preserve tablegroup oid so that tablegroup oid of a tablegroup remains the same on a new cluster. - Add `yb-admin import_snapshot` support for tablegroup parent table. Calculate tablegroup parent table uuid based on old tablegroup parent table uuid from old cluster. The logic is: (1) old_tablegroup_parent_table_uuid = function(old_namespace_oid, old_tablegroup_oid) (2) We know the uuid matching between the old namespace and the new namespace from an earlier step in `import_snapshot`, and tablegroup oid remains the same. (3) So new_tablegroup_parent_table_uuid = function(new_namespace_oid, old_tablegroup_oid) Sample output of import_snapshot for the above exmaple: ``` Read snapshot meta file Importing snapshot 40b4e757-797d-4ea2-be80-fc51810f92dc (COMPLETE) Target imported YSQL database name: yugabyte YSQL database being imported: yugabyte Table type: table Table being imported: yugabyte.tbl Table type: colocated table Colocated table being imported: yugabyte.000033e8000030008000000000004003.tablegroup.parent.tablename Table type: colocated table Colocated table being imported: yugabyte.tbl2 Successfully applied snapshot. Object Old ID New ID Keyspace 000033e8000030008000000000000000 000033e8000030008000000000000000 Table 000033e8000030008000000000004000 000033e8000030008000000000004000 Tablet 0 e9746f31b49d45de9ecbf3f78d3d4296 a9a5da1c2c274801b5e4d7bbe248cebc Tablet 1 c8ded40dbe2c4d55bdb7b2c499639071 3a495f69527245b08b894a244a7ccd58 Keyspace 000033e8000030008000000000000000 000033e8000030008000000000000000 ParentColocatedTable 000033e8000030008000000000004003.tablegroup.parent.uuid 000033e8000030008000000000004003.tablegroup.parent.uuid Tablet 0 37e5ee5686194284956b9aeafe439767 186254847ce8454aa1371056f58da796 Keyspace 000033e8000030008000000000000000 000033e8000030008000000000000000 ColocatedTable 000033e8000030008000000000004004 000033e8000030008000000000004001 Snapshot 40b4e757-797d-4ea2-be80-fc51810f92dc aee3464b-34f9-480c-b7b8-367426dfbec8 ``` - Add tablegroup handling in `yb_backup.py`. In yb_backup.py, add support to parse `import_snapshot` output having tablegroup parent table so that snapshot directories can be downloaded correctly for tablegroup parent table. Test Plan: ./yb_build.sh --java-test 'org.yb.pgsql.TestYsqlDump' ./yb_build.sh --java-test 'org.yb.pgsql.TestYbBackup#testTablegroup' ./yb_build.sh --java-test 'org.yb.pgsql.TestYbBackup#testMultipleTablegroups' Reviewers: mihnea, jhe, msun, myang, alex, oleg, tverona Reviewed By: alex, oleg, tverona Subscribers: zdrudi, jenkins-bot, yugaware, yql, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D18820

frozenspider added the area/ysql Yugabyte SQL (YSQL) label Mar 2, 2022

frozenspider added this to Backlog in YSQL via automation Mar 2, 2022

frozenspider added this to To do in Colocation via automation Mar 2, 2022

frozenspider assigned m-iancu Mar 2, 2022

frozenspider mentioned this issue Mar 9, 2022

[YSQL] Database-level multi-tenancy with tablegroups #11665

Open

tverona1 changed the title ~~[YSQL][Colocation] yb-admin create_database_snapshot can't deal with tablegroups~~ [Phase 2][YSQL][Colocation] yb-admin create_database_snapshot can't deal with tablegroups May 17, 2022

yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue labels May 17, 2022

yifanguan self-assigned this Aug 6, 2022

tverona1 unassigned m-iancu Aug 8, 2022

yifanguan moved this from To do to In progress in Colocation Aug 30, 2022

def- mentioned this issue Sep 20, 2022

[DocDB][Backup][Tablegroup][Materialized View] Create backup on a database containing tablegroups fails with OBJECT_NOT_FOUND (master error 3) #14081

Closed

yifanguan closed this as completed Nov 7, 2022

YSQL automation moved this from Backlog to Done Nov 7, 2022

Colocation automation moved this from In progress to Done Nov 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Phase 2][YSQL][Colocation] yb-admin create_database_snapshot can't deal with tablegroups #11632

[Phase 2][YSQL][Colocation] yb-admin create_database_snapshot can't deal with tablegroups #11632

frozenspider commented Mar 2, 2022 •

edited by yugabyte-ci

Loading

def- commented Aug 4, 2022 •

edited

Loading

yifanguan commented Aug 6, 2022 •

edited

Loading

yifanguan commented Aug 6, 2022

def- commented Aug 7, 2022

sanketkedia commented Sep 20, 2022

yifanguan commented Sep 20, 2022

yifanguan commented Nov 7, 2022

[Phase 2][YSQL][Colocation] yb-admin create_database_snapshot can't deal with tablegroups #11632

[Phase 2][YSQL][Colocation] yb-admin create_database_snapshot can't deal with tablegroups #11632

Comments

frozenspider commented Mar 2, 2022 • edited by yugabyte-ci Loading

Description

def- commented Aug 4, 2022 • edited Loading

yifanguan commented Aug 6, 2022 • edited Loading

yifanguan commented Aug 6, 2022

def- commented Aug 7, 2022

sanketkedia commented Sep 20, 2022

yifanguan commented Sep 20, 2022

yifanguan commented Nov 7, 2022

frozenspider commented Mar 2, 2022 •

edited by yugabyte-ci

Loading

def- commented Aug 4, 2022 •

edited

Loading

yifanguan commented Aug 6, 2022 •

edited

Loading