-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
consumer group specific offset seeking for AbstractConsumerSeekAware #2302
Comments
I don't see a behavior you are describing:
As you see I have two In the unit test I do:
So, after sending some data, I just call
This confirms that seeking really happens only in one consumer group and it does not effect other groups on the same topic. |
Correct; seeks only affect the current group. |
@artembilan @garyrussell
TestI have modified the code provided above slightly.
...
// Existing code
@KafkaListener(id = "seekExample", topics = "seekExample", concurrency = "3")
public void listen(String payload) {
...
}
// Added code in the same class Listener
@KafkaListener(id = "seekExample3", topics = "seekExample", concurrency = "3")
public void listen3(String payload) {
System.out.println("Listener3 received: " + payload);
}
...
...
// Existing code
for (int i = 0; i < 30; i++) { // Change: 10 -> 30 for both two listeners(seekExample and seekExample3) to be able to seek offsets
this.template.send("seekExample", i % 3, "some_key", "test#" + i);
}
...
...
Listener received: test#0
Listener3 received: test#3
Listener2 received: test#16
Listener2 received: test#19
Listener2 received: test#22
Listener received: test#29
Listener2 received: test#25
Listener2 received: test#28
Listener received: test#28
Listener received: test#21
Listener received: test#24
Listener received: test#27
Listener2 received: test#23
Listener2 received: test#26
Listener3 received: test#16
========
Listener received: test#0
Listener received: test#3
Listener received: test#24
Listener received: test#27
Listener received: test#2
Listener3 received: test#1
Listener received: test#5
Listener3 received: test#4
Listener received: test#8
Listener3 received: test#7
Listener received: test#11
Listener3 received: test#10
... IMHO, If we want to seek offset for a specific consumer group only, we can use the following methods:
If you have any other solutions for seeking offsets based on a specific consumer group ID, please let me know. I would appreciate hearing them. Thank you! |
I am no longer involved with the project, but what you are suggesting is incorrect. Each listener method is invoked by a different listener container and, therefore, on different threads. So, if there is a problem, it is not related to any thread-based state. |
@bky373 Thanks for reporting. As @garyrussell pointed out, this looks like it is non-thread-state related, but it looks like some bug (Thanks, Gary, for chiming in!! :) ). We will look at this today. Do you have any sample application for us to reproduce? (that would be easier). Otherwise, we can look into creating one since you provided some snippets. |
Oh I didn't know that! Thanks for your comment!! 🙇
You are totally right. Threads are different. I didn't mean to say that it's an issue with thread state. I just wanted to report that in a class that have listeners with different consumer group IDs and implements AbstractConsumerSeekAware, it's difficult to find the offset by specifying the consumer group ID. |
Sure! the code is so simple so I'll leave it in the comments here. @SpringBootApplication
public class KafkaGh2302Application {
public static void main(String[] args) {
SpringApplication.run(KafkaGh2302Application.class, args);
}
@Bean
public NewTopic topic() {
return new NewTopic("seekExample", 3, (short) 1);
}
@Component
public static class Listener extends AbstractConsumerSeekAware {
@KafkaListener(id = "seekExample", topics = "seekExample", concurrency = "3")
public void listen(String payload) {
System.out.println("Listener received: " + payload);
}
@KafkaListener(id = "seekExample3", topics = "seekExample", concurrency = "3")
public void listen3(String payload) {
System.out.println("Listener3 received: " + payload);
}
public void seekToStart() {
getSeekCallbacks().forEach((tp, callback) -> callback.seekToBeginning(tp.topic(), tp.partition()));
}
}
@Component
public static class Listener2 extends AbstractConsumerSeekAware {
@KafkaListener(id = "seekExample2", topics = "seekExample", concurrency = "3")
public void listen(String payload) {
System.out.println("Listener2 received: " + payload);
}
public void seekToStart() {
getSeekCallbacks().forEach((tp, callback) -> callback.seekToBeginning(tp.topic(), tp.partition()));
}
}
} @SpringBootTest
@EmbeddedKafka(bootstrapServersProperty = "spring.kafka.bootstrap-servers")
@DirtiesContext
class KafkaGh2302ApplicationTest {
@Autowired
KafkaGh2302Application.Listener listener;
@Autowired
KafkaTemplate<String, String> template;
@Test
void contextLoads() throws InterruptedException {
for (int i = 0; i < 50; i++) {
this.template.send("seekExample", i % 3, "some_key", "test#" + i);
}
Thread.sleep(1000);
System.out.println("====================================");
this.listener.seekToStart();
Thread.sleep(10000);
}
} |
I think the best course of action is to have a single consumer ( |
@sobychacko FYI, the thread's associated group is available in spring-kafka/spring-kafka/src/main/java/org/springframework/kafka/support/KafkaUtils.java Lines 110 to 117 in 4a5a849
|
@sobychacko @garyrussell As you mentioned, the method listeners within the class will apply the callback identically regardless of consumer group ID. So it seems necessary to execute callbacks differently for each consumer group since the intended behavior may vary between consumer groups. (Of course, we can work around this for now by keeping our classes separate.) I'll also keep looking for ways to do it. I'm so grateful for your help! |
We will try to make some changes to accommodate this before the GA. |
@bky373 After looking at this further, we realized this is a bit more involved from the framework perspective since we need to introduce some breaking changes at the API level. Therefore, we recommend your workaround in this and prior versions of Spring Kafka (since we are so close to the |
Thank you for taking the time to research and respond! I'm curious to know what you think of the solution.
In either case, I'm hesitant to say, as it would be a big change, |
@bky373 We had an internal discussion on this with @artembilan yesterday. We need to make some changes similar to your line of thinking. Some API methods in |
Yes, thank you! Personally, I'd like to take this on and work on it a bit more. Off the top of my head, as you said, if we can get the consumerGroupId from the |
@bky373 Feel free to work on it. Before you start coding, if you want us to confirm the design, please continue discussing it here, and we can review it. Thanks! |
Hi, I apologize for reaching out after such a long time. Before diving into the details, let me briefly summarize the problem since it has been a while.
Here are the approaches I've considered: 1. Passing
2. Setting seek allow flag per
Thank you for reading through this long message. Feel free to share any thoughts or feedback. |
@sobychacko |
sure @bky373. Sorry for the delay. We will get back to you soon on this. |
@bky373 I like your second approach, as this is a minimally invasive set of changes and doesn't require any API changes in @artembilan do you have any thoughts on changing Also, I wonder if there is a valid use case that might benefit others where they need to drive seeking offsets based on the group-id? |
Well, this new
So, just don't mix up many listeners in a single class. Sorry for some rude language, but if we go this way, I'd prefer |
Each approaches have pros and cons. While it is an easier solution to add this as a new flag, adding a top-level property like this to
|
There's another option; add a new ( default boolean seekByGroupId() {
return false;
} Then, use No breaking API changes, no But it would only work when seeking on the listener thread. |
Thanks, @garyrussell, for that great insight. |
Thanks, Gary! I see the logic in
Which probably has to be swapped to make that |
@sobychacko
I think this might be the case. @Component
public class DeliveryListener extends AbstractConsumerSeekAware {
...
@KafkaListener(groupId = "delivery-status-group", topics = "delivery-topic")
void listenForStatusUpdates(String message) {
// Update delivery status in DB
updateService.update(message)
}
@KafkaListener(groupId = "delivery-notification-group", topics = "delivery-topic")
void listenForNotifications(String message) {
// Notify the customer
notificationService.notify(message)
}
@KafkaListener(groupId = "delivery-analytics-group", topics = "delivery-topic")
void listenForAnalytics(String message) {
// Process the delivery message for analytics
analyticsService.analyze(message)
}
} If the error only occurs in the In fact, in cases like the above, it's probably best to separate the classes, as the internal logic would be rather complicated (which is what I want most!). |
@artembilan |
@garyrussell |
@bky373 I think it is better to go with Gary's suggestion on this. Can you think about a design along the lines of what he suggested? |
@sobychacko Yes, I'll look into this and get back to you. |
b93ff92 Here's what I did. @sobychacko @artembilan
|
I think approach 1 is cleaner. I suggest making that a formal PR and submitting it so we can do more reviews. |
Alright! I'm going to create PR. |
@artembilan @sobychacko @garyrussell I've come to provide an additional example regarding the problem mentioned above. (I'd just like to leave this content as a reference). [Background]
[One of the solutions]
[Where the problem was found]
[Code Examples]
public class BaseListener extends AbstractConsumerSeekAware {
public void seekToEarliest() {
this.getSeekCallbacks()
.forEach((tp, cb) -> cb.seekToBeginning(tp.topic(), tp.partition()));
}
}
@Component
public class FirstListener extends BaseListener {
@KafkaListener(topics = “my-topic”, groupId = “my-group-id-1”)
public void listen(String message) {
System.out.println("[FirstListener] received: ” + message);
}
}
@Component
public class SecondListener extends BaseListener {
@KafkaListener(topics = “my-topic”, groupId = “my-group-id-2”)
public void listen(String message) {
System.out.println("[SecondListener] received: ” + message);
}
}
@SpringBootApplication
public class SpringKafkaPlaygroundApp {
public static void main(String[] args) {
SpringApplication.run(SpringKafkaPlaygroundApp.class, args);
}
@Bean
public CommandLineRunner run(List<BaseListener> listeners,
KafkaTemplate<String, String> kafkaTemplate) {
return args -> {
kafkaTemplate.send("my-topic", "my-data-1");
kafkaTemplate.send("my-topic", "my-data-2");
kafkaTemplate.send("my-topic", "my-data-3");
Thread.sleep(5000);
listeners.forEach(l -> l.seekToEarliest());
}
}
public class BaseListener extends AbstractConsumerSeekAware {
public void seekToEarliest(String groupId) {
if ("my-group-id-1".equals(groupId)) {
this.getSeekCallbacks()
.forEach((tp, cb) -> cb.seekToBeginning(tp.topic(), tp.partition()));
}
}
}
public class BaseListener extends AbstractConsumerSeekAware {
public void seekToEarliest(String groupId) {
this.getTopicsAndCallbacks()
.forEach((tp, callbacks) -> {
callbacks.stream()
.filter(cb -> groupId.equals(cb.getGroupId()))
.forEach(cb -> cb.seekToBeginning(tp.topic(), tp.partition()));
});
}
}
That's all for now. After version 3.3 was released, I reviewed the code again and wanted to say thanks again for all the brainstorming we did. It really helped me solve the problem. Thanks! |
Expected Behavior
We want to be able to seek offset for specific consumer group by using AbstractConsumerSeekAware.
Current Behavior
regarding to below implementation it is clear that we can seek offset for all assigned partitions in a topic regardless of different consumer group ids.
Context
For our use case there might be more than one group instance which is assigned same partition in a topic. Below example might be useful to describe our case:
Is there any way to seek offset in a partition but only for specific group id?
The text was updated successfully, but these errors were encountered: