Skip to content

Commit

Permalink
[Improve][Connector-V2][Hive] Improve config check logic (apache#3886)
Browse files Browse the repository at this point in the history
* Whether the partition directory contains the partition name. If not, the partition loading method is different to ensure successful partition loading

* partition_dir_expression parameter adds the validation of the partition name to prevent the partition from failing to load

* update change log.

* Fix the problem of duplicate hive partition loading.

* partition_dir_expression parameter adds the validation of the partition name to prevent the partition from failing to load

* revert

* revert

* [Improve][Connector-V2][Hive] Improve config check logic, add the limit of partition expression

Co-authored-by: zhaoliang01 <zhaoliang01@58.com>
Co-authored-by: tyrantlucifer <tyrantlucifer@gmail.com>
  • Loading branch information
3 people authored and harveyyue committed Feb 7, 2023
1 parent cdf76c1 commit c35e106
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 1 deletion.
2 changes: 1 addition & 1 deletion docs/en/connector-v2/sink/Hive.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,4 +166,4 @@ sink {
### Next version

- [Improve] Support kerberos authentication ([3840](https://github.com/apache/incubator-seatunnel/pull/3840))

- [Improve] Added partition_dir_expression validation logic ([3886](https://github.com/apache/incubator-seatunnel/pull/3886))
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
import org.apache.seatunnel.common.config.CheckResult;
import org.apache.seatunnel.common.constants.PluginType;
import org.apache.seatunnel.common.exception.CommonErrorCode;
import org.apache.seatunnel.connectors.seatunnel.file.config.BaseSinkConfig;
import org.apache.seatunnel.connectors.seatunnel.file.config.FileFormat;
import org.apache.seatunnel.connectors.seatunnel.file.config.HadoopConf;
import org.apache.seatunnel.connectors.seatunnel.file.hdfs.sink.BaseHdfsFileSink;
Expand Down Expand Up @@ -83,6 +84,11 @@ public void prepare(Config pluginConfig) throws PrepareFailException {
String.format("PluginName: %s, PluginType: %s, Message: %s",
getPluginName(), PluginType.SINK, result.getMsg()));
}
if (pluginConfig.hasPath(BaseSinkConfig.PARTITION_DIR_EXPRESSION.key())) {
throw new HiveConnectorException(SeaTunnelAPIErrorCode.CONFIG_VALIDATION_FAILED, String.format(
"Hive sink connector does not support setting %s", BaseSinkConfig.PARTITION_DIR_EXPRESSION.key()
));
}
Pair<String[], Table> tableInfo = HiveConfig.getTableInfo(pluginConfig);
dbName = tableInfo.getLeft()[0];
tableName = tableInfo.getLeft()[1];
Expand Down

0 comments on commit c35e106

Please sign in to comment.