Skip to content

Commit

Permalink
Merge branch 'dev' into influxdb_sink_connector
Browse files Browse the repository at this point in the history
  • Loading branch information
531651225 authored Nov 7, 2022
2 parents 4bd1da2 + 3e6caf7 commit 33c9ca7
Show file tree
Hide file tree
Showing 30 changed files with 2,435 additions and 139 deletions.
2 changes: 1 addition & 1 deletion docs/en/connector-v2/sink/IoTDB.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ Enable rpc compression in `IoTDB` client

### connection_timeout_in_ms [int]

The maximum time (in ms) to wait when connect `IoTDB`
The maximum time (in ms) to wait when connecting to `IoTDB`

### common options

Expand Down
128 changes: 128 additions & 0 deletions docs/en/connector-v2/sink/StarRocks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# StarRocks

> StarRocks sink connector
## Description
Used to send data to StarRocks. Both support streaming and batch mode.
The internal implementation of StarRocks sink connector is cached and imported by stream load in batches.
## Key features

- [ ] [exactly-once](../../concept/connector-v2-features.md)
- [ ] [schema projection](../../concept/connector-v2-features.md)

## Options

| name | type | required | default value |
|-----------------------------|------------------------------|----------|-----------------|
| node_urls | list | yes | - |
| username | string | yes | - |
| password | string | yes | - |
| database | string | yes | - |
| table | string | no | - |
| labelPrefix | string | no | - |
| batch_max_rows | long | no | 1024 |
| batch_max_bytes | int | no | 5 * 1024 * 1024 |
| batch_interval_ms | int | no | - |
| max_retries | int | no | - |
| retry_backoff_multiplier_ms | int | no | - |
| max_retry_backoff_ms | int | no | - |
| sink.properties.* | starrocks stream load config | no | - |

### node_urls [list]

`StarRocks` cluster address, the format is `["fe_ip:fe_http_port", ...]`

### username [string]

`StarRocks` user username

### password [string]

`StarRocks` user password

### database [string]

The name of StarRocks database

### table [string]

The name of StarRocks table

### labelPrefix [string]

the prefix of StarRocks stream load label

### batch_max_rows [string]

For batch writing, when the number of buffers reaches the number of `batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches `batch_interval_ms`, the data will be flushed into the StarRocks

### batch_max_bytes [string]

For batch writing, when the number of buffers reaches the number of `batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches `batch_interval_ms`, the data will be flushed into the StarRocks

### batch_interval_ms [string]

For batch writing, when the number of buffers reaches the number of `batch_max_rows` or the byte size of `batch_max_bytes` or the time reaches `batch_interval_ms`, the data will be flushed into the StarRocks

### max_retries [string]

The number of retries to flush failed

### retry_backoff_multiplier_ms [string]

Using as a multiplier for generating the next delay for backoff

### max_retry_backoff_ms [string]

The amount of time to wait before attempting to retry a request to `StarRocks`

### sink.properties.* [starrocks stream load config]

the parameter of the stream load `data_desc`
The way to specify the parameter is to add the prefix `sink.properties.` to the original stream load parameter name.
For example, the way to specify `strip_outer_array` is: `sink.properties.strip_outer_array`.

#### Supported import data formats

The supported formats include CSV and JSON. Default value: CSV

## Example
Use JSON format to import data
```
sink {
StarRocks {
nodeUrls = ["e2e_starRocksdb:8030"]
username = root
password = ""
database = "test"
table = "e2e_table_sink"
batch_max_rows = 10
sink.properties.format = "JSON"
sink.properties.strip_outer_array = true
}
}
```

Use CSV format to import data
```
sink {
StarRocks {
nodeUrls = ["e2e_starRocksdb:8030"]
username = root
password = ""
database = "test"
table = "e2e_table_sink"
batch_max_rows = 10
sink.properties.format = "CSV"
sink.properties.column_separator = "\\x01",
sink.properties.row_delimiter = "\\x02"
}
}
```

## Changelog

### next version

- Add StarRocks Sink Connector
8 changes: 8 additions & 0 deletions docs/en/start-v2/docker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
sidebar_position: 3
---

# Set Up with Docker

<!-- TODO -->
WIP
Loading

0 comments on commit 33c9ca7

Please sign in to comment.