Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve][doc] Add doc of reader config to pulsar-io-debezium #16891

Merged
merged 5 commits into from
Sep 29, 2022
Merged

[improve][doc] Add doc of reader config to pulsar-io-debezium #16891

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7bf5b1b
add reader config doc
freeznet Jul 31, 2022
339f1a6
update to the versioned doc
freeznet Jul 31, 2022
2a8aa0b
Update site2/docs/io-debezium-source.md
freeznet Aug 2, 2022
f9dc81f
Update site2/docs/io-debezium-source.md
freeznet Aug 2, 2022
af1ce78
revert changes to 2.10.1 and 2.9.3
freeznet Sep 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 28 additions & 1 deletion site2/docs/io-debezium-source.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,9 @@ The configuration of Debezium source connector has the following properties.
| `database.history.pulsar.topic` | true | null | The name of the database history topic where the connector writes and recovers DDL statements. <br /><br />**Note: this topic is for internal use only and should not be used by consumers.** |
| `database.history.pulsar.service.url` | true | null | Pulsar cluster service URL for history topic. |
| `offset.storage.topic` | true | null | Record the last committed offsets that the connector successfully completes. |
| `json-with-envelope` | false | false | Present the message only consist of payload.
| `json-with-envelope` | false | false | Present the message only consist of payload. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| `json-with-envelope` | false | false | Present the message only consist of payload. |
| `json-with-envelope` | false | false | Present the message that consists of payload only. |

Is this what it means? The original one has a grammar issue.

| `database.history.pulsar.reader.config` | false | null | The configs of the reader for the database schema history topic, in the form of a JSON string with key-value pairs. |
| `offset.storage.reader.config` | false | null | The configs of the reader for the kafka connector offsets topic, in the form of a JSON string with key-value pairs. |

### Converter Options

Expand All @@ -54,7 +56,32 @@ Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED)`, and the message consist
| `mongodb.password` | true | null | Password to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication. |
| `mongodb.task.id` | true | null | The taskId of the MongoDB connector that attempts to use a separate task for each replica set. |

### Customize the Reader config for the metadata topics

The Debezium Connector exposes `database.history.pulsar.reader.config` and `offset.storage.reader.config` to configure the reader of database schema history topic and the kafka connector offsets topic. For example, it can be used to configure the subscription name and other reader configurations. The available configuration can be found at [ReaderConfigurationData](https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ReaderConfigurationData.java).
freeznet marked this conversation as resolved.
Show resolved Hide resolved

For example, to configure the subscription name for both Reader, you can add the following configuration:
freeznet marked this conversation as resolved.
Show resolved Hide resolved

* JSON

```json
{
"configs": {
"database.history.pulsar.reader.config": "{\"subscriptionName\":\"history-reader\"}",
"offset.storage.reader.config": "{\"subscriptionName\":\"offset-reader\"}",
}
}
```

* YAML

```yaml

configs:
database.history.pulsar.reader.config: "{\"subscriptionName\":\"history-reader\"}"
offset.storage.reader.config: "{\"subscriptionName\":\"offset-reader\"}"

```

## Example of MySQL

Expand Down
29 changes: 28 additions & 1 deletion site2/website/versioned_docs/version-2.10.1/io-debezium-source.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ The configuration of Debezium source connector has the following properties.
| `database.history.pulsar.topic` | true | null | The name of the database history topic where the connector writes and recovers DDL statements. <br /><br />**Note: this topic is for internal use only and should not be used by consumers.** |
| `database.history.pulsar.service.url` | true | null | Pulsar cluster service URL for history topic. |
| `offset.storage.topic` | true | null | Record the last committed offsets that the connector successfully completes. |
| `json-with-envelope` | false | false | Present the message only consist of payload.
| `json-with-envelope` | false | false | Present the message only consist of payload. |
| `database.history.pulsar.reader.config` | false | null | The configs of the reader for the database schema history topic, in the form of a JSON string with key-value pairs. |
| `offset.storage.reader.config` | false | null | The configs of the reader for the kafka connector offsets topic, in the form of a JSON string with key-value pairs. |

### Converter Options

Expand All @@ -55,7 +57,32 @@ Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED)`, and the message consist
| `mongodb.password` | true | null | Password to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication. |
| `mongodb.task.id` | true | null | The taskId of the MongoDB connector that attempts to use a separate task for each replica set. |

### Customize the Reader config for the metadata topics

The Debezium Connector exposes `database.history.pulsar.reader.config` and `offset.storage.reader.config` to configure the reader of database schema history topic and the kafka connector offsets topic. For example, it can be used to configure the subscription name and other reader configurations. The available configuration can be found at [ReaderConfigurationData](https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ReaderConfigurationData.java).

For example, to configure the subscription name for both Reader, you can add the following configuration:

* JSON

```json
{
"configs": {
"database.history.pulsar.reader.config": "{\"subscriptionName\":\"history-reader\"}",
"offset.storage.reader.config": "{\"subscriptionName\":\"offset-reader\"}",
}
}
```

* YAML

```yaml

configs:
database.history.pulsar.reader.config: "{\"subscriptionName\":\"history-reader\"}"
offset.storage.reader.config: "{\"subscriptionName\":\"offset-reader\"}"

```

## Example of MySQL

Expand Down
30 changes: 29 additions & 1 deletion site2/website/versioned_docs/version-2.9.3/io-debezium-source.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,10 @@ The configuration of Debezium source connector has the following properties.
| `database.history.pulsar.topic` | true | null | The name of the database history topic where the connector writes and recovers DDL statements. <br /><br />**Note: this topic is for internal use only and should not be used by consumers.** |
| `database.history.pulsar.service.url` | true | null | Pulsar cluster service URL for history topic. |
| `offset.storage.topic` | true | null | Record the last committed offsets that the connector successfully completes. |
| `json-with-envelope` | false | false | Present the message only consist of payload.
| `json-with-envelope` | false | false | Present the message only consist of payload.|
| `database.history.pulsar.reader.config` | false | null | The configs of the reader for the database schema history topic, in the form of a JSON string with key-value pairs. |
| `offset.storage.reader.config` | false | null | The configs of the reader for the kafka connector offsets topic, in the form of a JSON string with key-value pairs. |


### Converter Options

Expand All @@ -55,7 +58,32 @@ Schema.AUTO_CONSUME(), KeyValueEncodingType.SEPARATED)`, and the message consist
| `mongodb.password` | true | null | Password to be used when connecting to MongoDB. This is required only when MongoDB is configured to use authentication. |
| `mongodb.task.id` | true | null | The taskId of the MongoDB connector that attempts to use a separate task for each replica set. |

### Customize the Reader config for the metadata topics

The Debezium Connector exposes `database.history.pulsar.reader.config` and `offset.storage.reader.config` to configure the reader of database schema history topic and the kafka connector offsets topic. For example, it can be used to configure the subscription name and other reader configurations. The available configuration can be found at [ReaderConfigurationData](https://github.com/apache/pulsar/blob/master/pulsar-client/src/main/java/org/apache/pulsar/client/impl/conf/ReaderConfigurationData.java).

For example, to configure the subscription name for both Reader, you can add the following configuration:

* JSON

```json
{
"configs": {
"database.history.pulsar.reader.config": "{\"subscriptionName\":\"history-reader\"}",
"offset.storage.reader.config": "{\"subscriptionName\":\"offset-reader\"}",
}
}
```

* YAML

```yaml

configs:
database.history.pulsar.reader.config: "{\"subscriptionName\":\"history-reader\"}"
offset.storage.reader.config: "{\"subscriptionName\":\"offset-reader\"}"

```

## Example of MySQL

Expand Down