getting frequent non-zero number of bytes remaining error during DescribeGroups

[tarun-discovery]

Describe the bug

Got non-zero number of bytes remaining: 10

Kafka Version

• What version(s) of Kafka are you testing against? v0.4.39
• What version of kafka-go are you using? v0.4.39

To Reproduce
I am not sure way to reproduce as this intermittent

package main

import (
    "github.com/segmentio/kafka-go"
)

func main() {
   dc, err := bc.Client.DescribeGroups(ctx, &kafkago.DescribeGroupsRequest{Addr: kafkago.TCP(bc.Broker), GroupIDs: []string{gId}})
		if err != nil {
			logger.Errorf(ctx, "describeGroup error for group %s, region %s: %v", gId, bc.Region, err)
			continue
		}
}

Expected Behavior
It should return a valid describegroupresponse

Observed Behavior
Got non-zero number of bytes remaining: 10

Additional Context

Add any other context about the problem here.

[amortezaei]

1. Seems like you used the same version number for both kafka and kafka-go. Can you clarify the kafka version you tested against?
2. Can you please provide more details on this specific case such as cluster setup ie number of brokers, the topic details, and consumer group details.

[tarun-discovery]

the consumer is a java client
i am using kafka-go to fetch consumer details from broker which is on latest kafka version
my kafka-go is v0.4.44

[tarun-discovery]

any news on this? its been months

[beebeeep]

Have same problem, tried kafka-go v0.4.39 and v0.4.47, against kafka 3.6, kafka-broker-api-versions:

	Produce(0): 0 to 9 [usable: 9],
	Fetch(1): 0 to 15 [usable: 15],
	ListOffsets(2): 0 to 8 [usable: 8],
	Metadata(3): 0 to 12 [usable: 12],
	LeaderAndIsr(4): 0 to 7 [usable: 7],
	StopReplica(5): 0 to 4 [usable: 4],
	UpdateMetadata(6): 0 to 8 [usable: 8],
	ControlledShutdown(7): 0 to 3 [usable: 3],
	OffsetCommit(8): 0 to 8 [usable: 8],
	OffsetFetch(9): 0 to 8 [usable: 8],
	FindCoordinator(10): 0 to 4 [usable: 4],
	JoinGroup(11): 0 to 9 [usable: 9],
	Heartbeat(12): 0 to 4 [usable: 4],
	LeaveGroup(13): 0 to 5 [usable: 5],
	SyncGroup(14): 0 to 5 [usable: 5],
	DescribeGroups(15): 0 to 5 [usable: 5],
	ListGroups(16): 0 to 4 [usable: 4],
	SaslHandshake(17): 0 to 1 [usable: 1],
	ApiVersions(18): 0 to 3 [usable: 3],
	CreateTopics(19): 0 to 7 [usable: 7],
	DeleteTopics(20): 0 to 6 [usable: 6],
	DeleteRecords(21): 0 to 2 [usable: 2],
	InitProducerId(22): 0 to 4 [usable: 4],
	OffsetForLeaderEpoch(23): 0 to 4 [usable: 4],
	AddPartitionsToTxn(24): 0 to 4 [usable: 4],
	AddOffsetsToTxn(25): 0 to 3 [usable: 3],
	EndTxn(26): 0 to 3 [usable: 3],
	WriteTxnMarkers(27): 0 to 1 [usable: 1],
	TxnOffsetCommit(28): 0 to 3 [usable: 3],
	DescribeAcls(29): 0 to 3 [usable: 3],
	CreateAcls(30): 0 to 3 [usable: 3],
	DeleteAcls(31): 0 to 3 [usable: 3],
	DescribeConfigs(32): 0 to 4 [usable: 4],
	AlterConfigs(33): 0 to 2 [usable: 2],
	AlterReplicaLogDirs(34): 0 to 2 [usable: 2],
	DescribeLogDirs(35): 0 to 4 [usable: 4],
	SaslAuthenticate(36): 0 to 2 [usable: 2],
	CreatePartitions(37): 0 to 3 [usable: 3],
	CreateDelegationToken(38): 0 to 3 [usable: 3],
	RenewDelegationToken(39): 0 to 2 [usable: 2],
	ExpireDelegationToken(40): 0 to 2 [usable: 2],
	DescribeDelegationToken(41): 0 to 3 [usable: 3],
	DeleteGroups(42): 0 to 2 [usable: 2],
	ElectLeaders(43): 0 to 2 [usable: 2],
	IncrementalAlterConfigs(44): 0 to 1 [usable: 1],
	AlterPartitionReassignments(45): 0 [usable: 0],
	ListPartitionReassignments(46): 0 [usable: 0],
	OffsetDelete(47): 0 [usable: 0],
	DescribeClientQuotas(48): 0 to 1 [usable: 1],
	AlterClientQuotas(49): 0 to 1 [usable: 1],
	DescribeUserScramCredentials(50): 0 [usable: 0],
	AlterUserScramCredentials(51): 0 [usable: 0],
	DescribeQuorum(55): UNSUPPORTED,
	AlterPartition(56): 0 to 3 [usable: 3],
	UpdateFeatures(57): 0 to 1 [usable: 1],
	Envelope(58): 0 [usable: 0],
	DescribeCluster(60): 0 [usable: 0],
	DescribeProducers(61): 0 [usable: 0],
	UnregisterBroker(64): UNSUPPORTED,
	DescribeTransactions(65): 0 [usable: 0],
	ListTransactions(66): 0 [usable: 0],
	AllocateProducerIds(67): 0 [usable: 0],
	ConsumerGroupHeartbeat(68): UNSUPPORTED

As far as I can tell, the issue is intermittent and only appears for some for some groups. As far as I've seen, all problematic groups were active (had active consumers). I was unable to see any pattern so far.

[beebeeep]

Ok, so far I've noticed that kafka-go (library and latest topicctl) is issuing DescribeGroups v4 response when describing problematic group. Meantime, native java tool kafka-consumer-groups.sh (which doesnt have any problems) are using DescribeGroups v5. It feels like it's malformed v4 response from Kafka, because WireShark is also failing to decode it correctly claiming that length is invalid.