Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/doc-mysql-connector-version' int…
Browse files Browse the repository at this point in the history 
…o doc-mysql-connector-version
  • Loading branch information
@zhilinli123
zhilinli123 committed Aug 14, 2023
2 parents 7a1cdbe + 29a3d96 commit 91edda0
Show file tree
Hide file tree
Showing 104 changed files with 2,336 additions and 866 deletions.
2 changes: 0 additions & 2 deletions .github/workflows/backend.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,6 @@ on:
branches:
- dev
paths-ignore:
- 'docs/**'
- '**/*.md'
- 'seatunnel-ui/**'

concurrency:
Expand Down
2 changes: 1 addition & 1 deletion DISCLAIMER
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Apache SeaTunnel (incubating) is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC.
Apache SeaTunnel is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC.
Incubation is required of all newly accepted projects until a further review indicates that the infrastructure,
communications, and decision making process have stabilized in a manner consistent with other successful ASF projects.
While incubation status is not necessarily a reflection of the completeness or stability of the code,
Expand Down
3 changes: 1 addition & 2 deletions config/seatunnel.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@

seatunnel:
engine:
history-job-expire-minutes: 1440
backup-count: 1
queue-type: blockingqueue
print-execution-info-interval: 60
Expand All @@ -26,8 +27,6 @@ seatunnel:
checkpoint:
interval: 10000
timeout: 60000
max-concurrent: 1
tolerable-failure: 2
storage:
type: hdfs
max-retained: 3
Expand Down
47 changes: 47 additions & 0 deletions docs/en/connector-v2/formats/kafka-compatible-kafkaconnect-json.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Kafka source compatible kafka-connect-json

Seatunnel connector kafka supports parsing data extracted through kafka connect source, especially data extracted from kafka connect jdbc and kafka connect debezium

# How to use

## Kafka output to mysql

```bash
env {
execution.parallelism = 1
job.mode = "BATCH"
}

source {
Kafka {
bootstrap.servers = "localhost:9092"
topic = "jdbc_source_record"
result_table_name = "kafka_table"
start_mode = earliest
schema = {
fields {
id = "int"
name = "string"
description = "string"
weight = "string"
}
},
format = COMPATIBLE_KAFKA_CONNECT_JSON
}
}


sink {
Jdbc {
driver = com.mysql.cj.jdbc.Driver
url = "jdbc:mysql://localhost:3306/seatunnel"
user = st_user
password = seatunnel
generate_sink_sql = true
database = seatunnel
table = jdbc_sink
primary_keys = ["id"]
}
}
```

50 changes: 27 additions & 23 deletions docs/en/connector-v2/sink/PostgreSql.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,34 +36,34 @@ semantics (using XA transaction guarantee).
## Data Type Mapping

| PostgreSQL Data type | SeaTunnel Data type |
|----------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|
| BOOL<br/> | BOOLEAN |
| _BOOL<br/> | ARRAY&LT;BOOLEAN&GT; |
| BYTEA<br/> | BYTES |
| _BYTEA<br/> | ARRAY&LT;TINYINT&GT; |
| INT2<br/>SMALLSERIAL<br/>INT4<br/>SERIAL<br/> | INT |
| _INT2<br/>_INT4<br/> | ARRAY&LT;INT&GT; |
| INT8<br/>BIGSERIAL<br/> | BIGINT |
| _INT8<br/> | ARRAY&LT;BIGINT&GT; |
| FLOAT4<br/> | FLOAT |
| _FLOAT4<br/> | ARRAY&LT;FLOAT&GT; |
| FLOAT8<br/> | DOUBLE |
| _FLOAT8<br/> | ARRAY&LT;DOUBLE&GT; |
| NUMERIC(Get the designated column's specified column size>0) | DECIMAL(Get the designated column's specified column size,Gets the number of digits in the specified column to the right of the decimal point) |
| NUMERIC(Get the designated column's specified column size<0) | DECIMAL(38, 18) |
| BPCHAR<br/>CHARACTER<br/>VARCHAR<br/>TEXT<br/>GEOMETRY<br/>GEOGRAPHY | STRING |
| _BPCHAR<br/>_CHARACTER<br/>_VARCHAR<br/>_TEXT | ARRAY&LT;STRING&GT; |
| TIMESTAMP<br/> | TIMESTAMP |
| TIME<br/> | TIME |
| DATE<br/> | DATE |
| OTHER DATA TYPES | NOT SUPPORTED YET |
| PostgreSQL Data type | SeaTunnel Data type |
|-----------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|
| BOOL<br/> | BOOLEAN |
| _BOOL<br/> | ARRAY&LT;BOOLEAN&GT; |
| BYTEA<br/> | BYTES |
| _BYTEA<br/> | ARRAY&LT;TINYINT&GT; |
| INT2<br/>SMALLSERIAL<br/>INT4<br/>SERIAL<br/> | INT |
| _INT2<br/>_INT4<br/> | ARRAY&LT;INT&GT; |
| INT8<br/>BIGSERIAL<br/> | BIGINT |
| _INT8<br/> | ARRAY&LT;BIGINT&GT; |
| FLOAT4<br/> | FLOAT |
| _FLOAT4<br/> | ARRAY&LT;FLOAT&GT; |
| FLOAT8<br/> | DOUBLE |
| _FLOAT8<br/> | ARRAY&LT;DOUBLE&GT; |
| NUMERIC(Get the designated column's specified column size>0) | DECIMAL(Get the designated column's specified column size,Gets the number of digits in the specified column to the right of the decimal point) |
| NUMERIC(Get the designated column's specified column size<0) | DECIMAL(38, 18) |
| BPCHAR<br/>CHARACTER<br/>VARCHAR<br/>TEXT<br/>GEOMETRY<br/>GEOGRAPHY<br/>JSON<br/>JSONB | STRING |
| _BPCHAR<br/>_CHARACTER<br/>_VARCHAR<br/>_TEXT | ARRAY&LT;STRING&GT; |
| TIMESTAMP<br/> | TIMESTAMP |
| TIME<br/> | TIME |
| DATE<br/> | DATE |
| OTHER DATA TYPES | NOT SUPPORTED YET |

## Options

| Name | Type | Required | Default | Description |
|-------------------------------------------|---------|----------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| url | String | Yes | - | The URL of the JDBC connection. Refer to a case: jdbc:postgresql://localhost:5432/test |
| url | String | Yes | - | The URL of the JDBC connection. Refer to a case: jdbc:postgresql://localhost:5432/test <br/> if you would use json or jsonb type insert please add jdbc url stringtype=unspecified option |
| driver | String | Yes | - | The jdbc class name used to connect to the remote data source,<br/> if you use PostgreSQL the value is `org.postgresql.Driver`. |
| user | String | No | - | Connection instance user name |
| password | String | No | - | Connection instance password |
Expand Down Expand Up @@ -124,6 +124,7 @@ transform {
sink {
jdbc {
# if you would use json or jsonb type insert please add jdbc url stringtype=unspecified option
url = "jdbc:postgresql://localhost:5432/test"
driver = "org.postgresql.Driver"
user = root
Expand All @@ -142,6 +143,7 @@ sink {
```
sink {
Jdbc {
# if you would use json or jsonb type insert please add jdbc url stringtype=unspecified option
url = "jdbc:postgresql://localhost:5432/test"
driver = org.postgresql.Driver
user = root
Expand All @@ -161,6 +163,7 @@ sink {
```
sink {
jdbc {
# if you would use json or jsonb type insert please add jdbc url stringtype=unspecified option
url = "jdbc:postgresql://localhost:5432/test"
driver = "org.postgresql.Driver"
Expand All @@ -183,6 +186,7 @@ sink {
```
sink {
jdbc {
# if you would use json or jsonb type insert please add jdbc url stringtype=unspecified option
url = "jdbc:postgresql://localhost:5432/test"
driver = "org.postgresql.Driver"
user = root
Expand Down
5 changes: 5 additions & 0 deletions docs/en/connector-v2/sink/Redis.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ Used to write data to Redis.
| mode | string | no | single |
| nodes | list | yes when mode=cluster | - |
| format | string | no | json |
| expire | long | no | -1 |
| common-options | | no | - |

### host [string]
Expand Down Expand Up @@ -120,6 +121,10 @@ Connector will generate data as the following and write it to redis:

```

### expire [long]

Set redis expiration time, the unit is second. The default value is -1, keys do not automatically expire by default.

### common options

Sink plugin common parameters, please refer to [Sink Common Options](common-options.md) for details
Expand Down
Loading

0 comments on commit 91edda0

Please sign in to comment.