Existing messages are not shown in topic Data table

[FilatovM]

After producing messages to topic they are not shown in UI (all messages or some of them). They are already committed but can't be displayed until the page refreshed several times or some kind of filters or sorts applied. The number of messages displayed correctly, but "No data available" displayed in the table instead of messages.

Sometimes only part of messages are missing, in other cases all messages committed recently are absent.

[AlexisSouquiere]

@FilatovM on which version ? 0.24.0 or dev ?

[FilatovM]

It's 0.24.0

[AlexisSouquiere]

Ok then this issue should be fixed on the dev branch (and the new version that will be released soon)
Be careful there are breaking changes between 0.24.0 and dev on the roles/groups settings

[mrMigles]

Hi @AlexisSouquiere,
Thank you for the answer!
May we know when will the new version be released? Approximately 😄

[mrMigles]

Hello,
I just reproduced this issue with latest dev branch. It's reproducing only for distributed Kafka cluster (In my case it's 3 brokers cluster).

1. create topic with 10 partitions and replication factor 3
2. produce few records key 1,value 1, etc. You need to make sure they are distributed to different partitions with different leaders brokers (seems like it's important).
3. Try to open topic on Data tab and good luck to see all records you added once.

Looks like you usually see data from partitions which have the same leader broker. But I didn't look through the code yet to prove it, may be someone know what can be a reason of such behavior?

[mrMigles]

Found out the cause. The problem in default consumer poll here https://github.com/tchiotludo/akhq/blob/dev/src/main/java/org/akhq/repositories/RecordRepository.java#L164
it doesn't guarantee that poll returns data from all partition once, and in our case it returns some messages from partition 1 and 2 (which leaders on the same broker), but doesn't return messages from partition 3 in the same poll result, next poll call returns them (but for us it doesn't mater, because we can't see some data in UI).

The issue is not reproducing in newest sorting, because in this case the new consumer is created per each partition.

Looks like previously we had the same logic for oldest also, but then it has been rewritten after newest optimization #1069.

I locally resolved the issue in fast mode, just call consume for each partition I have:

        partitions.forEach((partition, offset) -> {
            consumer.unsubscribe();
            consumer.assign(Collections.singleton(partition));
            consumer.seek(partition, offset);
            log.trace(
                "Consume [topic: {}] [partition: {}] [start: {}]",
                partition.topic(),
                partition.partition(),
                offset
            );
            ConsumerRecords<byte[], byte[]> records = this.poll(consumer);
            for (ConsumerRecord<byte[], byte[]> record : records) {
                Record current = newRecord(record, options, topic);
                if (matchFilters(options, current)) {
                    filterMessageLength(current);
                    list.add(current);
                }
            }
        });

but looks like it should be more generic and smart logic. Probably it is better to use the same approach as for Newest sort used.

@tchiotludo Hi, what do you think about that?

[nparfait]

I have noticed this issue also. Multi broker cluster (MSK on AWS). If i switch to sort by oldest it appears to display messages. I have had this several times, not sure if it always works with Oldest.

[neoscaler]

Also experiencing this. In our case we see messages with oldest sorting, but not using newest.

[warpian]

I am also experiencing this, when I switch back to akhq 0.23.0 all messages are displayed just fine.

[tchiotludo]

please try with latest dev branch, there is major enhancement on that part, if it's still not working, please reopen

[itarasenko]

I have the same isuue on 0.23.0, 0.24.0 and dev! "No data available" when search by Value or sort by Oldest. Even if serch for a record added one munutes ago. Topic contains 260K records, 5 partitions, replication factor =3. Schema registry is used, format is avro.