Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Connector-V2][Postgres-cdc]Support for Postgres cdc #5986

Merged
merged 17 commits into from
Jan 18, 2024

Conversation

Carl-Zhou-CN
Copy link
Member

@Carl-Zhou-CN Carl-Zhou-CN commented Dec 10, 2023

Purpose of this pull request

Support for Postgres cdc

Does this PR introduce any user-facing change?

How was this patch tested?

new e2e test

Check list

@jobmission
Copy link

Very nice need this feature.

@Carl-Zhou-CN Carl-Zhou-CN marked this pull request as ready for review December 18, 2023 11:56
@Carl-Zhou-CN Carl-Zhou-CN added this to the 2.3.4 milestone Dec 20, 2023
@Carl-Zhou-CN
Copy link
Member Author

Carl-Zhou-CN commented Dec 21, 2023

@hailin0 @ic4y @EricJoy2048 @Hisoka-X @TyrantLucifer Please help review, thank you

@EricJoy2048
Copy link
Member

@hailin0 @ic4y @EricJoy2048 @Hisoka-X @TyrantLucifer Please help review, thank you

We have found some serious bugs in PG CDC during testing. Please wait for us to fix them before reviewing this PR.

@Carl-Zhou-CN
Copy link
Member Author

Carl-Zhou-CN commented Jan 15, 2024

Wait for compatibility with the following optimizations #6106 #5784 #6098 #5785

@hailin0
Copy link
Member

hailin0 commented Jan 16, 2024

waiting for ci passed

EricJoy2048
EricJoy2048 previously approved these changes Jan 18, 2024
Hisoka-X
Hisoka-X previously approved these changes Jan 18, 2024
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add new connector into seatunnel-dist/pom.xml

@Hisoka-X Hisoka-X dismissed their stale review January 18, 2024 02:00

please add new connector into seatunnel-dist/pom.xml

@EricJoy2048 EricJoy2048 merged commit 97438b9 into apache:dev Jan 18, 2024
7 checks passed
@W-dragan
Copy link

W-dragan commented Feb 6, 2024

I found that there is currently no CDC that supports PG partition tables. For example, I created a people table,
CREATE TABLE public.people (
id int4 NOT NULL,
birth_date date NOT NULL,
country_code bpchar(2) NOT NULL,
"name" text NULL,
CONSTRAINT people_pk PRIMARY KEY (id, birth_date)
)

Then create 3 more partitions

--Publicpeople_ Y2000 definition

CREATE TABLE public.people_y2000 PARTITION OF public.people FOR VALUES FROM ('2000-01-01') TO ('2001-01-01');

--Publicpeople_ Y2000 foreign keys

--Publicpeople_ Y2001 definition

CREATE TABLE public.people_y2001 PARTITION OF public.people FOR VALUES FROM ('2001-01-01') TO ('2002-01-01');

--Publicpeople_ Y2001 foreign keys

--Publicpeople_ Y2002 definition

CREATE TABLE public.people_y2002 PARTITION OF public.people FOR VALUES FROM ('2002-01-01') TO ('2003-01-01');

--Publicpeople_ Y2002 foreign keys
When I configure the use of the People table in my source configuration, I cannot query the table because it is a partition type. Even if I modify the code to forcibly query the table, I cannot perform CDC operations on it
@Carl-Zhou-CN @EricJoy2048

chaorongzhi pushed a commit to chaorongzhi/seatunnel that referenced this pull request Aug 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants