feat: Implementation Datax doriswriter plugin #6107
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Kyofin
changed the title
Kyofin
changed the title
|
Hi @huzk8 ,this PR #6111 has been merged, please move the |
morningman
added
area/datax
kind/feature
hf200012
reviewed
Jul 1, 2021
| public List<Configuration> split(int mandatoryNumber) { | ||
| List<Configuration> configurations = new ArrayList<>(mandatoryNumber); | ||
|
|
||
| for(int i = 0; i < mandatoryNumber; ++i) { |
| } | ||
|
|
||
| private String getStreamLoadLabel() { | ||
| return UUID.randomUUID().toString(); |
There was a problem hiding this comment.
Data import of datax can be uniformly started with datax_doris_writer_
eg："datax_doris_writer_" + UUID.randomUUID().toString();
| throw new IOException("None of the host in `beLoadUrl` could be connected."); | ||
| } | ||
| final String loadUrl = host + "/api/" + this.keys.getDatabase() + "/" + this.keys.getTable() + "/_stream_load"; | ||
| LOG.debug(String.format("Start to join batch data: rows[%d] bytes[%d] label[%s].", flushData.getRows().size(), flushData.getBytes(), flushData.getLabel())); |
There was a problem hiding this comment.
It is recommended to use info here to facilitate troubleshooting
debug -> info
| if (null == loadResult || !loadResult.containsKey(keyStatus)) { | ||
| throw new IOException("Unable to flush data to doris: unknown result status."); | ||
| } | ||
| LOG.debug("StreamLoad response:\n" + JSON.toJSONString(loadResult)); |
There was a problem hiding this comment.
It is recommended to use info here to facilitate troubleshooting
debug -> info
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Implementation datax doriswriter plugin.
To compile this plugin , you need to download Alibaba/Datax source and put plugin to that maven project.
Types of changes
Checklist
Further comments
Should I write a doc about how to install this plugin in Datax?