Skip to content

Commit

Permalink
[#19192] XClusterDDLRepl: Support Drop Relations
Browse files Browse the repository at this point in the history
Summary:
Adding in initial support for drop table/index + some related objects.

Since dropped objects are not captured by pg_event_trigger_ddl_commands and only captured by
pg_event_trigger_dropped_objects, I am adding in a new event trigger on sql_drop to capture and
check dropped objects. For the time being this is just doing simple checks (checking for temporary
objects) as the trigger happens after the delete has occured, so it is difficult to do lookups. If
necessary in the future, we could expand pg_event_trigger_dropped_objects to have additional data,
but that is not needed at the moment.

As part of this, I am pulling out should_replicate_ddl to a global variable that is per DDL. So now,
if a DDL has any object that is dropped/created/modified and we need to replicate it, then we will
replicate the entire command.
Adding in a new event trigger on ddl_command_start to reset this variable and also moving prechecks
here (eg multicommand check).

Note: For the time being until ghi #22320, we will block drops of any relations in a colocated
database. We could do as I mentioned above and add is_colocated to pg_event_trigger_dropped_objects,
but that would be just be removed in a soon to come diff, so opting for a simpler temp solution.
Jira: DB-7981

Test Plan:
```
ybd --java-test "org.yb.pgsql.TestPgRegressYbExtensionsYbXclusterDdlReplication"

ybd --cxx-test xcluster_ddl_replication_pgregress-test --gtest_filter "XClusterPgRegressDDLReplicationTest.PgRegressCreateDropTable"
ybd --cxx-test xcluster_ddl_replication_pgregress-test --gtest_filter "XClusterPgRegressDDLReplicationTest.PgRegressCreateDropTable2"
ybd --cxx-test xcluster_ddl_replication_pgregress-test --gtest_filter "XClusterPgRegressDDLReplicationTest.PgRegressCreateTableUnsupported"
ybd --cxx-test xcluster_ddl_replication_pgregress-test --gtest_filter "XClusterPgRegressDDLReplicationTest.PgRegressCreateDropPartitionedTable"
ybd --cxx-test xcluster_ddl_replication_pgregress-test --gtest_filter "XClusterPgRegressDDLReplicationTest.PgRegressCreateDropTablePartitions"
ybd --cxx-test xcluster_ddl_replication_pgregress-test --gtest_filter "XClusterPgRegressDDLReplicationTest.PgRegressCreateDropTablePartitions2"
```

Reviewers: hsunder, xCluster

Reviewed By: hsunder

Subscribers: ycdcxcluster, ybase, yql

Differential Revision: https://phorge.dev.yugabyte.com/D34898
  • Loading branch information
hulien22 committed Aug 9, 2024
1 parent ba23acf commit 924dada
Show file tree
Hide file tree
Showing 38 changed files with 2,517 additions and 241 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ CREATE TEMP TABLE temp_foo(i int PRIMARY KEY);
-- Verify that colocated tables are blocked.
CREATE TABLE coloc_foo(i int PRIMARY KEY);
ERROR: Colocated objects are not yet supported by yb_xcluster_ddl_replication
To manually replicate, run DDL with SET yb_xcluster_ddl_replication.enable_manual_ddl_replication = true
To manually replicate, run DDL on the source followed by the target with SET yb_xcluster_ddl_replication.enable_manual_ddl_replication = true
SELECT yb_data FROM yb_xcluster_ddl_replication.ddl_queue ORDER BY start_time;
yb_data
---------
Expand All @@ -18,7 +18,7 @@ CREATE TABLE non_coloc_foo(i int PRIMARY KEY) WITH (COLOCATION = false);
SELECT yb_data FROM yb_xcluster_ddl_replication.ddl_queue ORDER BY start_time;
yb_data
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
{"user": "yugabyte", "query": "CREATE TABLE non_coloc_foo(i int PRIMARY KEY) WITH (COLOCATION = false);", "schema": "public", "version": 1, "command_tag": "CREATE TABLE", "new_rel_map": [{"rel_name": "non_coloc_foo", "relfile_oid": 16414}]}
{"user": "yugabyte", "query": "CREATE TABLE non_coloc_foo(i int PRIMARY KEY) WITH (COLOCATION = false);", "schema": "public", "version": 1, "command_tag": "CREATE TABLE", "new_rel_map": [{"rel_name": "non_coloc_foo", "relfile_oid": 16418}]}
(1 row)

SELECT * FROM yb_xcluster_ddl_replication.replicated_ddls ORDER BY start_time;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
CALL TEST_reset();
SET yb_xcluster_ddl_replication.replication_role = DISABLED;
CREATE SCHEMA create_index;
SET search_path TO create_index;
-- Test temp table and index.
SET yb_xcluster_ddl_replication.replication_role = SOURCE;
CREATE TEMP TABLE temp_foo(i int PRIMARY KEY, a int);
CREATE INDEX foo_idx_temp on temp_foo(a);
DROP INDEX foo_idx_temp;
DROP TABLE temp_foo;
SELECT yb_data FROM yb_xcluster_ddl_replication.ddl_queue ORDER BY start_time;
yb_data
---------
(0 rows)

SET yb_xcluster_ddl_replication.replication_role = BIDIRECTIONAL;
-- Create base table.
CREATE TABLE foo(i int PRIMARY KEY, a int, b text, c int);
-- Create indexes.
CREATE INDEX foo_idx_simple ON foo(a);
CREATE UNIQUE INDEX foo_idx_unique ON foo(b);
CREATE INDEX foo_idx_filtered ON foo(c ASC, a) WHERE a > c;
-- Test that role is captured properly.
CREATE ROLE new_role SUPERUSER;
SET ROLE new_role;
CREATE INDEX foo_idx_include ON foo(lower(b)) INCLUDE (a) SPLIT INTO 2 TABLETS;
SET ROLE NONE;
SELECT yb_data FROM yb_xcluster_ddl_replication.ddl_queue ORDER BY start_time;
yb_data
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
{"user": "yugabyte", "query": "CREATE TABLE foo(i int PRIMARY KEY, a int, b text, c int);", "schema": "create_index", "version": 1, "command_tag": "CREATE TABLE", "new_rel_map": [{"rel_name": "foo", "relfile_oid": 16451}]}
{"user": "yugabyte", "query": "CREATE INDEX foo_idx_simple ON foo(a);", "schema": "create_index", "version": 1, "command_tag": "CREATE INDEX", "new_rel_map": [{"rel_name": "foo_idx_simple", "relfile_oid": 16456}]}
{"user": "yugabyte", "query": "CREATE UNIQUE INDEX foo_idx_unique ON foo(b);", "schema": "create_index", "version": 1, "command_tag": "CREATE INDEX", "new_rel_map": [{"rel_name": "foo_idx_unique", "relfile_oid": 16457}]}
{"user": "yugabyte", "query": "CREATE INDEX foo_idx_filtered ON foo(c ASC, a) WHERE a > c;", "schema": "create_index", "version": 1, "command_tag": "CREATE INDEX", "new_rel_map": [{"rel_name": "foo_idx_filtered", "relfile_oid": 16458}]}
{"user": "new_role", "query": "CREATE INDEX foo_idx_include ON foo(lower(b)) INCLUDE (a) SPLIT INTO 2 TABLETS;", "schema": "create_index", "version": 1, "command_tag": "CREATE INDEX", "new_rel_map": [{"rel_name": "foo_idx_include", "relfile_oid": 16459}]}
(5 rows)

SELECT * FROM yb_xcluster_ddl_replication.replicated_ddls ORDER BY start_time;
start_time | query_id | yb_data
------------+----------+----------------------------------------------------------------------------------------------
1 | 1 | {"query": "CREATE TABLE foo(i int PRIMARY KEY, a int, b text, c int);"}
2 | 1 | {"query": "CREATE INDEX foo_idx_simple ON foo(a);"}
3 | 1 | {"query": "CREATE UNIQUE INDEX foo_idx_unique ON foo(b);"}
4 | 1 | {"query": "CREATE INDEX foo_idx_filtered ON foo(c ASC, a) WHERE a > c;"}
5 | 1 | {"query": "CREATE INDEX foo_idx_include ON foo(lower(b)) INCLUDE (a) SPLIT INTO 2 TABLETS;"}
(5 rows)

-- Now drop these indexes.
-- Drop two indexes by themselves.
DROP INDEX foo_idx_unique;
DROP INDEX foo_idx_filtered;
-- Drop base table and cascade deletion of other indexes.
DROP TABLE foo;
SELECT yb_data FROM yb_xcluster_ddl_replication.ddl_queue ORDER BY start_time;
yb_data
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
{"user": "yugabyte", "query": "CREATE TABLE foo(i int PRIMARY KEY, a int, b text, c int);", "schema": "create_index", "version": 1, "command_tag": "CREATE TABLE", "new_rel_map": [{"rel_name": "foo", "relfile_oid": 16451}]}
{"user": "yugabyte", "query": "CREATE INDEX foo_idx_simple ON foo(a);", "schema": "create_index", "version": 1, "command_tag": "CREATE INDEX", "new_rel_map": [{"rel_name": "foo_idx_simple", "relfile_oid": 16456}]}
{"user": "yugabyte", "query": "CREATE UNIQUE INDEX foo_idx_unique ON foo(b);", "schema": "create_index", "version": 1, "command_tag": "CREATE INDEX", "new_rel_map": [{"rel_name": "foo_idx_unique", "relfile_oid": 16457}]}
{"user": "yugabyte", "query": "CREATE INDEX foo_idx_filtered ON foo(c ASC, a) WHERE a > c;", "schema": "create_index", "version": 1, "command_tag": "CREATE INDEX", "new_rel_map": [{"rel_name": "foo_idx_filtered", "relfile_oid": 16458}]}
{"user": "new_role", "query": "CREATE INDEX foo_idx_include ON foo(lower(b)) INCLUDE (a) SPLIT INTO 2 TABLETS;", "schema": "create_index", "version": 1, "command_tag": "CREATE INDEX", "new_rel_map": [{"rel_name": "foo_idx_include", "relfile_oid": 16459}]}
{"user": "yugabyte", "query": "DROP INDEX foo_idx_unique;", "schema": "create_index", "version": 1, "command_tag": "DROP INDEX"}
{"user": "yugabyte", "query": "DROP INDEX foo_idx_filtered;", "schema": "create_index", "version": 1, "command_tag": "DROP INDEX"}
{"user": "yugabyte", "query": "DROP TABLE foo;", "schema": "create_index", "version": 1, "command_tag": "DROP TABLE"}
(8 rows)

SELECT * FROM yb_xcluster_ddl_replication.replicated_ddls ORDER BY start_time;
start_time | query_id | yb_data
------------+----------+----------------------------------------------------------------------------------------------
1 | 1 | {"query": "CREATE TABLE foo(i int PRIMARY KEY, a int, b text, c int);"}
2 | 1 | {"query": "CREATE INDEX foo_idx_simple ON foo(a);"}
3 | 1 | {"query": "CREATE UNIQUE INDEX foo_idx_unique ON foo(b);"}
4 | 1 | {"query": "CREATE INDEX foo_idx_filtered ON foo(c ASC, a) WHERE a > c;"}
5 | 1 | {"query": "CREATE INDEX foo_idx_include ON foo(lower(b)) INCLUDE (a) SPLIT INTO 2 TABLETS;"}
6 | 1 | {"query": "DROP INDEX foo_idx_unique;"}
7 | 1 | {"query": "DROP INDEX foo_idx_filtered;"}
8 | 1 | {"query": "DROP TABLE foo;"}
(8 rows)
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
CALL TEST_reset();
-- Verify that temporary objects are not captured.
SET yb_xcluster_ddl_replication.replication_role = SOURCE;
CREATE TEMP TABLE temp_foo(i int PRIMARY KEY);
DROP TABLE temp_foo;
SELECT yb_data FROM yb_xcluster_ddl_replication.ddl_queue ORDER BY start_time;
yb_data
---------
(0 rows)

SET yb_xcluster_ddl_replication.replication_role = BIDIRECTIONAL;
-- Verify that regular tables are captured.
CREATE TABLE foo(i int PRIMARY KEY);
-- Check with manual replication flags enabled, ddl string is captured with flag.
SET yb_xcluster_ddl_replication.enable_manual_ddl_replication = 1;
CREATE TABLE manual_foo(i int PRIMARY KEY);
SET yb_xcluster_ddl_replication.enable_manual_ddl_replication = 0;
-- Verify that extra info is captured.
CREATE TABLE extra_foo(i int PRIMARY KEY) WITH (COLOCATION = false) SPLIT INTO 1 TABLETS;
-- Verify that info for unique constraint indexes are also captured.
CREATE TABLE unique_foo(i int PRIMARY KEY, u text UNIQUE);
SELECT yb_data FROM yb_xcluster_ddl_replication.ddl_queue ORDER BY start_time;
yb_data
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
{"user": "yugabyte", "query": "CREATE TABLE foo(i int PRIMARY KEY);", "schema": "public", "version": 1, "command_tag": "CREATE TABLE", "new_rel_map": [{"rel_name": "foo", "relfile_oid": 16412}]}
{"user": "yugabyte", "query": "CREATE TABLE manual_foo(i int PRIMARY KEY);", "schema": "public", "version": 1, "command_tag": "CREATE TABLE", "manual_replication": true}
{"user": "yugabyte", "query": "CREATE TABLE extra_foo(i int PRIMARY KEY) WITH (COLOCATION = false) SPLIT INTO 1 TABLETS;", "schema": "public", "version": 1, "command_tag": "CREATE TABLE", "new_rel_map": [{"rel_name": "extra_foo", "relfile_oid": 16422}]}
{"user": "yugabyte", "query": "CREATE TABLE unique_foo(i int PRIMARY KEY, u text UNIQUE);", "schema": "public", "version": 1, "command_tag": "CREATE TABLE", "new_rel_map": [{"rel_name": "unique_foo", "relfile_oid": 16427}, {"rel_name": "unique_foo_u_key", "relfile_oid": 16432}]}
(4 rows)

SELECT * FROM yb_xcluster_ddl_replication.replicated_ddls ORDER BY start_time;
start_time | query_id | yb_data
------------+----------+--------------------------------------------------------------------------------------------------------
1 | 1 | {"query": "CREATE TABLE foo(i int PRIMARY KEY);"}
2 | 1 | {"query": "CREATE TABLE extra_foo(i int PRIMARY KEY) WITH (COLOCATION = false) SPLIT INTO 1 TABLETS;"}
3 | 1 | {"query": "CREATE TABLE unique_foo(i int PRIMARY KEY, u text UNIQUE);"}
(3 rows)

-- Now test dropping these tables.
DROP TABLE foo;
-- Check with manual replication flags enabled, ddl string is captured with flag.
SET yb_xcluster_ddl_replication.enable_manual_ddl_replication = 1;
DROP TABLE manual_foo;
SET yb_xcluster_ddl_replication.enable_manual_ddl_replication = 0;
DROP TABLE extra_foo;
DROP TABLE unique_foo;
SELECT yb_data FROM yb_xcluster_ddl_replication.ddl_queue ORDER BY start_time;
yb_data
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
{"user": "yugabyte", "query": "CREATE TABLE foo(i int PRIMARY KEY);", "schema": "public", "version": 1, "command_tag": "CREATE TABLE", "new_rel_map": [{"rel_name": "foo", "relfile_oid": 16412}]}
{"user": "yugabyte", "query": "CREATE TABLE manual_foo(i int PRIMARY KEY);", "schema": "public", "version": 1, "command_tag": "CREATE TABLE", "manual_replication": true}
{"user": "yugabyte", "query": "CREATE TABLE extra_foo(i int PRIMARY KEY) WITH (COLOCATION = false) SPLIT INTO 1 TABLETS;", "schema": "public", "version": 1, "command_tag": "CREATE TABLE", "new_rel_map": [{"rel_name": "extra_foo", "relfile_oid": 16422}]}
{"user": "yugabyte", "query": "CREATE TABLE unique_foo(i int PRIMARY KEY, u text UNIQUE);", "schema": "public", "version": 1, "command_tag": "CREATE TABLE", "new_rel_map": [{"rel_name": "unique_foo", "relfile_oid": 16427}, {"rel_name": "unique_foo_u_key", "relfile_oid": 16432}]}
{"user": "yugabyte", "query": "DROP TABLE foo;", "schema": "public", "version": 1, "command_tag": "DROP TABLE"}
{"user": "yugabyte", "query": "DROP TABLE manual_foo;", "schema": "public", "version": 1, "command_tag": "DROP TABLE", "manual_replication": true}
{"user": "yugabyte", "query": "DROP TABLE extra_foo;", "schema": "public", "version": 1, "command_tag": "DROP TABLE"}
{"user": "yugabyte", "query": "DROP TABLE unique_foo;", "schema": "public", "version": 1, "command_tag": "DROP TABLE"}
(8 rows)

SELECT * FROM yb_xcluster_ddl_replication.replicated_ddls ORDER BY start_time;
start_time | query_id | yb_data
------------+----------+--------------------------------------------------------------------------------------------------------
1 | 1 | {"query": "CREATE TABLE foo(i int PRIMARY KEY);"}
2 | 1 | {"query": "CREATE TABLE extra_foo(i int PRIMARY KEY) WITH (COLOCATION = false) SPLIT INTO 1 TABLETS;"}
3 | 1 | {"query": "CREATE TABLE unique_foo(i int PRIMARY KEY, u text UNIQUE);"}
4 | 1 | {"query": "DROP TABLE foo;"}
5 | 1 | {"query": "DROP TABLE extra_foo;"}
6 | 1 | {"query": "DROP TABLE unique_foo;"}
(6 rows)

-- Test mix of temp and regular tables.
SET yb_xcluster_ddl_replication.replication_role = SOURCE;
CREATE TEMP TABLE temp_foo(i int PRIMARY KEY);
SET yb_xcluster_ddl_replication.replication_role = BIDIRECTIONAL;
CREATE TABLE foo(i int PRIMARY KEY);
DROP TABLE temp_foo, foo; -- should fail
ERROR: Unsupported DROP command, found mix of temporary and persisted objects in DDL command.
To manually replicate, run DDL on the source followed by the target with SET yb_xcluster_ddl_replication.enable_manual_ddl_replication = true
DROP TABLE foo, temp_foo; -- should fail
ERROR: Unsupported DROP command, found mix of temporary and persisted objects in DDL command.
To manually replicate, run DDL on the source followed by the target with SET yb_xcluster_ddl_replication.enable_manual_ddl_replication = true
DROP TABLE temp_foo;
DROP TABLE foo;

This file was deleted.

Loading

0 comments on commit 924dada

Please sign in to comment.