Skip to content

Commit

Permalink
[Feature][Doc] Add doc about connector-v2 schema feature (#3296)
Browse files Browse the repository at this point in the history
  • Loading branch information
liugddx authored Nov 8, 2022
1 parent 3e6caf7 commit dce1cb7
Show file tree
Hide file tree
Showing 2 changed files with 64 additions and 1 deletion.
62 changes: 62 additions & 0 deletions docs/en/concept/schema-feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Intro to schema feature

## Why we need schema

Some NoSQL databases or message queue are not strongly limited schema, so the schema cannot be obtained through the api. At this time, a schema needs to be defined to convert to SeaTunnelRowType and obtain data.

## What type supported at now

| Data type | Description |
| :-------- | :----------------------------------------------------------- |
| string | string |
| boolean | boolean |
| tinyint | -128 to 127 regular. 0 to 255 unsigned*. Specify the maximum number of digits in parentheses. |
| smallint | -32768 to 32767 General. 0 to 65535 unsigned*. Specify the maximum number of digits in parentheses. |
| int | All numbers from -2,147,483,648 to 2,147,483,647 are allowed. |
| bigint | All numbers between -9,223,372,036,854,775,808 and 9,223,372,036,854,775,807 are allowed. |
| float | Float-precision numeric data from -1.79E+308 to 1.79E+308. |
| double | Double precision floating point. Handle most decimals. |
| decimal | DOUBLE type stored as a string, allowing a fixed decimal point. |
| null | null |
| bytes | bytes. |
| date | Only the date is stored. From January 1, 0001 to December 31, 9999. |
| time | Only store time. Accuracy is 100 nanoseconds. |
| timestamp | Stores a unique number that is updated whenever a row is created or modified. timestamp is based on the internal clock and does not correspond to real time. There can only be one timestamp variable per table. |
| row | Row type,can be nested. |

## How to use schema

`schema` defines the format of the data,it contains`fields` properties. `fields` define the field properties,it's a K-V key-value pair, the Key is the field name and the value is field type. Here is an example.

```
source {
FakeSource {
parallelism = 2
result_table_name = "fake"
row.num = 16
schema = {
fields {
id = bigint
c_map = "map<string, smallint>"
c_array = "array<tinyint>"
c_string = string
c_boolean = boolean
c_tinyint = tinyint
c_smallint = smallint
c_int = int
c_bigint = bigint
c_float = float
c_double = double
c_decimal = "decimal(2, 1)"
c_bytes = bytes
c_date = date
c_timestamp = timestamp
}
}
}
}
```

## When we should use it or not

If there is a `schema` configuration project in Options,the connector can then customize the schema. Like `Fake` `Pulsar` `Http` source connector etc.
3 changes: 2 additions & 1 deletion docs/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,8 @@ const sidebars = {
"label": "Concept",
"items": [
"concept/config",
"concept/connector-v2-features"
"concept/connector-v2-features",
'concept/schema-feature'
]
},
{
Expand Down

0 comments on commit dce1cb7

Please sign in to comment.