[Improve][Connector-V2-kafka] Support for dynamic discover topic & partition in streaming mode #3125
Conversation
| @Override | ||
| public void open() { | ||
| this.adminClient = initAdminClient(this.metadata.getProperties()); | ||
| ParallelSource parallelSource = context.getParallelSource(); |
There was a problem hiding this comment.
It's not right the connector to use translation module class. You can use ScheduledThreadPoolExecutor to execute topic find with fixed rate. Then close ScheduledThreadPoolExecutor when KafkaSourceSplitEnumerator closed.
There was a problem hiding this comment.
@Hisoka-X Well, I have this idea. I think that if the search thread fails, the consumer thread can be ended. If the search thread fails, do we need to affect the consumer thread
There was a problem hiding this comment.
It should be, if not, user will think program running fine.
There was a problem hiding this comment.
The problem I'm facing now is that if I want to stop the source, it seems that I can only interrupt at pollNext，Because I can't get too much valid information in context，Do you have any other good suggestions，
Carl-Zhou-CN
force-pushed
the
isses-2958
branch
from
6062de0 to
6ad6dfe
Compare
|
@Hisoka-X Please help to review |
.../java/org/apache/seatunnel/connectors/seatunnel/kafka/source/KafkaSourceSplitEnumerator.java
Outdated
Show resolved
Hide resolved
|
Retry CI after #3183 merged |
ok |
| ## partition-discovery.interval-millis [long] | ||
|
|
||
| The interval for dynamically discovering topics and partitions. | ||
|
|
There was a problem hiding this comment.
Please add changed log reference https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/connector-v2/source/Redis.md#next-version
|
@Carl-Zhou-CN Please resolve conflict. Thanks! |
…rtition in streaming mode
…rtition in streaming mode
Carl-Zhou-CN
force-pushed
the
isses-2958
branch
from
06ce652 to
adc31df
Compare
|
@Hisoka-X @EricJoy2048 Please help to review. |
…rtition in streaming mode
…rtition in streaming mode
# Conflicts: # docs/en/connector-v2/source/kafka.md
…rtition in streaming mode
…rtition in streaming mode
…rtition in streaming mode
Purpose of this pull request
Check list
New License Guide
close #2958