Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Connector-V2] Jdbc DB2 support upsert SQL #7879

Merged
merged 1 commit into from
Oct 21, 2024

Conversation

shashwatsai
Copy link
Contributor

@shashwatsai shashwatsai commented Oct 20, 2024

Purpose of this pull request

Currently Seatunnel JDBC Connector with DB2 Dialect, does not support upsert scenarios, as the implementation is unavailable. While analysing our requirements within Visa's use-cases, we have approximately 60% of jobs that heavily depend on this.

We want to introduce the implementation for upsert support for DB2 dialect, along with the e2e Integration test case.

solves: #4434, as Dameng DB, implementation is already present.
How do I reopen the issue?

Does this PR introduce any user-facing change?

No,
User does not have to provide the query parameter in sink, user has to provide database, table and generate_sink_sql, which are already available in the configurations. Please refer corresponding [Seatunnel-web] PR: apache/seatunnel-web#233

Sample:

env {
  parallelism = 1
  job.mode = "BATCH"
}

source {
  # This is a example source plugin **only for test and demonstrate the feature source plugin**
  Jdbc {
    driver = com.ibm.db2.jcc.DB2Driver
    url = "jdbc:db2://db2-e2e:50000/E2E"
    user = "db2inst1"
    password = "123456"
    query = """
    select * from "E2E".SOURCE;
    """
  }
}

sink {
  Jdbc {
    driver = com.ibm.db2.jcc.DB2Driver
    url = "jdbc:db2://db2-e2e:50000/E2E"
    user = "db2inst1"
    password = "123456"
    database = "E2E"
    table = "SINK"
    # The primary keys of the table, which will be used to generate the upsert sql
    generate_sink_sql = true
    primary_keys = [
      C_INT
    ]
  }

How was this patch tested?

We internally deployed, and tested the scenarios, we need to introduce a minor change in Seatunnel-web module as well, to delegate generation of sink sql to the engine, as required for the upsert use-case.

Check list

Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @shashwatsai , LGTM just except one minior problem.

database = "E2E"
table = "SINK"
# The primary keys of the table, which will be used to generate the upsert sql
generate_sink_sql = true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please force set enable_upsert to true to avoid the test case broken when the default value changed.

Add E2E for upsert db2 scenario
@shashwatsai shashwatsai force-pushed the ISSUE-4434-db2-upsert-support branch from ce8ab9b to a6e8108 Compare October 21, 2024 05:09
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if ci passes. Thanks @shashwatsai !

@shashwatsai
Copy link
Contributor Author

@Hisoka-X , thanks for your time in reviewing the same.

@hailin0 , @liugddx, @arshadmohammad can you please help with the review.

Copy link
Collaborator

@arshadmohammad arshadmohammad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1

@Hisoka-X Hisoka-X changed the title [Feature] DB2 support upsert SQL [Feature][Connector-V2] Jdbc DB2 support upsert SQL Oct 21, 2024
@Hisoka-X Hisoka-X merged commit 1399193 into apache:dev Oct 21, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][jdbc] dameng + DB2 support upsert SQL
3 participants