Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Connector-V2] [Doc] Add TableSourceFactory and TableSinkFactory Doc #3343

Merged
merged 4 commits into from
Nov 11, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions seatunnel-connectors-v2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,41 @@ In the current version, it is recommended to implement ``SinkAggregatedCommitter
provide strong consistency guarantee in Flink/Spark. At the same time, commit should be idempotent, and save engine
retry can work normally.

### TableSourceFactory and TableSinkFactory

In order to automatically create the Source Connector and Sink Connector and Transform Connector, we need the connector to return the parameters needed to create them and the verification rules for each parameter. For Source Connector and Sink Connector, we define `TableSourceFactory` and `TableSinkFactory`
supported by the current connector and the required parameters. We define TableSourceFactory and TableSinkFactory,
It is recommended to put it in the same directory as the implementation class of SeaTunnelSource or SeaTunnelSink for easy searching.

- `factoryIdentifier` is used to indicate the name of the current Factory. This value should be the same as the
value returned by `getPluginName`, so that if Factory is used to create Source/Sink in the future,
A seamless switch can be achieved.
- `createSink` and `createSource` are the methods for creating Source and Sink respectively,
and do not need to be implemented at present.
- `optionRule` returns the parameter logic, which is used to indicate which parameters of our connector are supported,
which parameters are required, which parameters are optional, and which parameters are exclusive, which parameters are bundledRequired.
This method will be used when we visually create the connector logic, and it will also be used to generate a complete parameter
object according to the parameters configured by the user, and then the connector developer does not need to judge whether the parameters
exist one by one in the Config, and use it directly That's it.
You can refer to existing implementations, such as `org.apache.seatunnel.connectors.seatunnel.elasticsearch.source.ElasticsearchSourceFactory`.
There is support for configuring Schema for many Sources, so a common Option is used.
If you need a schema, you can refer to `org.apache.seatunnel.connectors.seatunnel.common.schema.SeaTunnelSchema.SCHEMA`.

Don't forget to add `@AutoService(Factory.class)` to the class. This Factory is the parent class of TableSourceFactory and TableSinkFactory.

### **Options**

When we implement TableSourceFactory and TableSinkFactory, the corresponding Option will be created.
Each Option corresponds to a configuration, but different configurations will have different types.
Common types can be created by directly calling the corresponding method.
But if our parameter type is an object, we can use POJO to represent parameters of object type,
and need to use `org.apache.seatunnel.api.configuration.util.OptionMark` on each parameter to indicate that this is A child Option.
`OptionMark` has two parameters, `name` is used to declare the parameter name corresponding to the field.
If it is empty, we will convert the small camel case corresponding to java to underscore by default, such as: `myUserPassword` -> `my_user_password` .
In most cases, the default is empty. `description` is used to indicate the description of the current parameter.
This parameter is optional. It is recommended to be consistent with the documentation. For specific examples,
please refer to `org.apache.seatunnel.connectors.seatunnel.assertion.sink.AssertSinkFactory`.

## **Result**

All Connector implementations should be under the ``seatunnel-connectors-v2``, and the examples that can be referred to
Expand Down
22 changes: 22 additions & 0 deletions seatunnel-connectors-v2/README.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,28 @@ Sink可以根据组件属性进行选择,到底是只实现`SinkCommitter`或`

当前版本推荐将实现SinkAggregatedCommitter作为首选,可以在Flink/Spark中提供较强的一致性保证,同时commit应该要实现幂等性,保存引擎重试能够正常运作。

### TableSourceFactory 和 TableSinkFactory

为了实现自动化的创建Source或者Sink,我们需要连接器能够声明并返回创建他们所需要的参数列表和每个参数的校验规则。为了实现这个目标,我们定义了TableSourceFactory和TableSinkFactory,
建议将其放在和SeaTunnelSource或SeaTunnelSink实现类同一目录下,方便寻找。

- `factoryIdentifier` 用于表明当前Factory的名称,这个值应该和`getPluginName`返回的值一致,这样后续如果使用Factory来创建Source/Sink,
就能实现无缝切换。
- `createSink` 和 `createSource` 分别是创建Source和Sink的方法,目前不用实现。
- `optionRule` 返回的是参数逻辑,用于表示我们的连接器参数哪些支持,哪些参数是必须(required)的,哪些参数是可选(optional)的,哪些参数是互斥(exclusive)的,哪些参数是绑定(bundledRequired)的。
这个方法会在我们可视化创建连接器逻辑的时候用到,同时也会用于根据用户配置的参数生成完整的参数对象,然后连接器开发者就不用在Config里面一个个判断参数是否存在,直接使用即可。
可以参考现有的实现,比如`org.apache.seatunnel.connectors.seatunnel.elasticsearch.source.ElasticsearchSourceFactory`。针对很多Source都有支持配置Schema,所以采用了通用的Option,
需要Schema则可以引用`org.apache.seatunnel.connectors.seatunnel.common.schema.SeaTunnelSchema.SCHEMA`。

别忘记添加`@AutoService(Factory.class)` 到类上面。这个Factory即TableSourceFactory 和 TableSinkFactory的父类。

### Option

当我们实现TableSourceFactory 和 TableSinkFactory时,会创建对应的Option,每一个Option对应的就是一个配置,但是不同的配置会有不同的类型,普通类型直接调用对应的方法即可创建。
但是如果我们参数类型是一个对象,我们就可以使用POJO来表示对象类型的参数,同时需要在每个参数上使用`org.apache.seatunnel.api.configuration.util.OptionMark`来表明这是一个子Option。
`OptionMark`有两个参数,`name`用于声明字段对应的参数名称,如果为空的话,我们会默认将java对应的小驼峰转换成下划线进行表达,如:`myUserPassword`->`my_user_password`。
在大多数情况下,默认为空即可。`description`用于表示当前参数的描述,这个参数是可选的,建议和文档上的保持一致。具体例子可以参考`org.apache.seatunnel.connectors.seatunnel.assertion.sink.AssertSinkFactory`。

## 实现

现阶段所有的连接器实现及可参考的示例都在seatunnel-connectors-v2下,用户可自行查阅参考。