Thread blocked on get key inside a watcher #1089
Comments
|
@q12321q do you have any time to investigate further ? |
|
Hello @lburgazzoli, in a non blocking thread the test works in 0.7.3. My guess is that the switch to Vertx produce this regression. The watcher notification and the key get are in the event loop (i.e. single thread) and the first one wait for the second hence thread is blocked. This is just a supposition as I'm not very familiar with Vertx. Thanks a lot for your help |
|
I can't look at the issue now and probably for a few days so I would love if you can help me on this. |
|
Thanks a lot for having a look at this issue. |
|
@q12321q May I ask how you can avoid this problem? I am also encountering this problem now. |
|
I guess you should use an async call so don't use client.getKVClient().get(keyToGet).get() but i.e. client.getKVClient().get(keyToGet).accept(...) |
|
I also have the same problem in reactor,but reactor throws BlockingException, I have an idea, can you have async method execute it. |
|
I don't know the method of |
|
Try with something like this: public void testWatchAndGet(final Client client) throws Exception {
final ByteSequence key = randomByteSequence();
final ByteSequence value = randomByteSequence();
final AtomicReference<KeyValue> ref = new AtomicReference<>();
final Consumer<WatchResponse> consumer = response -> {
for (WatchEvent event : response.getEvents()) {
if (event.getEventType() == EventType.PUT) {
ByteSequence key1 = event.getKeyValue().getKey();
client.getKVClient().get(key1).whenComplete((r, t) -> {
if (!r.getKvs().isEmpty()) {
ref.set(r.getKvs().get(0));
}
});
}
}
};
try (Watcher watcher = client.getWatchClient().watch(key, consumer)) {
client.getKVClient().put(key, value).get();
await().atMost(TIME_OUT_SECONDS, TimeUnit.SECONDS).untilAsserted(() -> assertThat(ref.get()).isNotNull());
assertThat(ref.get()).isNotNull();
assertThat(ref.get().getKey()).isEqualTo(key);
assertThat(ref.get().getValue()).isEqualTo(value);
}
} |
I have tested this code, the result is as follows |
What is the issue ? |
I think the thread 'vetx.x-eventloop-0' should not be single, but I test it, and it is single |
|
this is new test |
|
I still don't get what is the issue as vert.x use event loops so things may end up on the the same thread. |
yes, I know that I will reference to vertx |
|
@lburgazzoli for me the issue is that the migration to vert.x force us not to block on some call backs. It's new requirement and a undocumented breaking change. |
Unfortunately I'm the only maintainer of the project and I asked many time to get some help but no one step up to help so I do what I think it is right and what helps me reducing the maintenance cost. Every time I've opened issues related to similar changes for discussion, I have received no feedback so I assume people do not care or do not use jetcd. So I'm sorry for the breaking change but given I'm alone and I don't have much time, I can't do much.
I think this is something you have to take into account anyway as in general the expectation is that you should use the callback methods to handle responses. That is why we return a CompletableFuture and not a blocking one |
|
Oh sorry if my tone was too direct! I didn't open the issue to blame. Thanks a lot for your work and my bad to not have enough time to help you more in this issue. My feeling here is that we may have to wrap the watcher's callbacks in a thread pool. The Maybe we should document this behavior as I feel that we won't be the only ones to have the issue. What do you think? |
|
thanks for your response, if possible, I am glad to seek the question. |
No worries at all
I don't think we should wrap the watcher's callbacks in a thread pool as if the pool become full then you may end blocking anyway so that should be handled by the application code (i.e. put responses in a queue). Honestly the watch should be implemented using the Flow APIs but I had to hold back on this because of people are asking for keep supporting Java 8 ... |
|
I also encountered this problem. I think callback processing is a good solution |
|
This issue is stale because it has been open 60 days with no activity. |
|
I have the similiar problem, my etcd client sometime stucks, and never comes back, while the internet is working perfectly. And in some of cases, for example, I used 4 thread to submit request to get data by the client, and 2 of them is blocked for no reason, but rest of them is still working. I am still trying to figure out what happens, but it seems to be not easy to reproduce this problem. Since then, it happens frequently on our production machine, which is on a private enviroment. But not able to reproduce on our local machine. |
Is there any new progress on this issue? @moremind @lburgazzoli |
|
@Wackerle if you think this is an issue, please open a new one or it will be forgotten and more important, any help would really be appreciated as unfortunately, I don't have much time to dedicate to troubleshoot issues like this one., |
Versions
Describe the bug
I don't know if it's a bug or a miss usage of the jetcd sdk but since we migrate from jetcd version
0.5.7to0.7.3when we do a get key within a watcher notification we get the exceptionio.vertx.core.VertxException: Thread blockedand indeed the get key is stuck.To Reproduce
Here how I reproduce:
Here is the full exception:
Expected behavior
I expect the test not to be stuck and the get key return its result.
Additional context
NA
The text was updated successfully, but these errors were encountered: