Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Broker certificate could not be verified. #1610

Open
6 of 7 tasks
sbv-csis opened this issue Jul 13, 2023 · 1 comment
Open
6 of 7 tasks

Broker certificate could not be verified. #1610

sbv-csis opened this issue Jul 13, 2023 · 1 comment

Comments

@sbv-csis
Copy link

Description

When using openssl s_client I can get the connection to be verified - but when using confluent_kafka it keeps telling me to set a variable I've already set:

Works:


% openssl s_client -connect redpanda-0.redpanda.redpanda.svc.cluster.local:9093  -CAfile /tmp/ca.crt -verify_hostname redpanda-0.redpanda.redpanda.svc.cluster.local
CONNECTED(00000003)
depth=1 CN = redpanda-default-root-certificate
verify return:1
depth=0 
verify return:1
---
Certificate chain
 0 s:
   i:CN = redpanda-default-root-certificate
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Jul 12 10:18:16 2023 GMT; NotAfter: Jul 10 10:18:16 2028 GMT
---
Server certificate
.....
subject=
issuer=CN = redpanda-default-root-certificate
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 1302 bytes and written 801 bytes
Verification: OK
Verified peername: *.redpanda.redpanda.svc.cluster.local
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

But when trying to use confluent-kafka to produce:

>>> from confluent_kafka import Producer
>>> c = {'bootstrap.servers': 'redpanda-0.redpanda.redpanda.svc.cluster.local:9093,redpanda-1.redpanda.redpanda.svc.cluster.local:9093,redpanda-2.redpanda.redpanda.svc.cluster.local:9093', 'security.protocol': 'SASL_SSL', 'sasl.mechanisms': 'SCRAM-SHA-256', 'sasl.username': 'therightusernamehere', 'sasl.password': 'therightpwhere', 'ssl.ca.location': '/tmp/ca.crt'}
>>> p = Producer(c)
>>> p.produce('t1', 'test'.encode())
>>> %3|1689252662.299|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 3ms in state SSL_HANDSHAKE)

And with full debug:

>>> c['debug'] = 'all'
>>> p = Producer(c)
%7|1689252905.302|SASL|rdkafka#producer-2| [thrd:app]: Selected provider SCRAM (builtin) for SASL mechanism SCRAM-SHA-256
%7|1689252905.302|OPENSSL|rdkafka#producer-2| [thrd:app]: Using statically linked OpenSSL version OpenSSL 3.0.8 7 Feb 2023 (0x30000080, librdkafka built with 0x30000080)
%7|1689252905.303|SSL|rdkafka#producer-2| [thrd:app]: Loading CA certificate(s) from file /tmp/ca.crt
%7|1689252905.303|WAKEUPFD|rdkafka#producer-2| [thrd:app]: sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabled low-latency ops queue wake-ups
%7|1689252905.303|BROKER|rdkafka#producer-2| [thrd:app]: sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Added new broker with NodeId -1
%7|1689252905.303|BRKMAIN|rdkafka#producer-2| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1689252905.304|BRKMAIN|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enter main broker thread
%7|1689252905.304|WAKEUPFD|rdkafka#producer-2| [thrd:app]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabled low-latency ops queue wake-ups
%7|1689252905.304|BROKER|rdkafka#producer-2| [thrd:app]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Added new broker with NodeId -1
%7|1689252905.304|WAKEUPFD|rdkafka#producer-2| [thrd:app]: sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabled low-latency ops queue wake-ups
%7|1689252905.304|BROKER|rdkafka#producer-2| [thrd:app]: sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Added new broker with NodeId -1
%7|1689252905.304|BRKMAIN|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enter main broker thread
%7|1689252905.304|CONNECT|rdkafka#producer-2| [thrd:app]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1689252905.304|BRKMAIN|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enter main broker thread
%7|1689252905.304|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received CONNECT op
%7|1689252905.304|INIT|rdkafka#producer-2| [thrd:app]: librdkafka v2.1.1 (0x20101ff) rdkafka#producer-2 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer,http,oidc, STRIP STATIC_LINKING GCC GXX PKGCONFIG INSTALL GNULD LIBDL PLUGINS ZLIB SSL ZSTD CURL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER OAUTHBEARER_OIDC CRC32C_HW, debug 0xfffff)
%7|1689252905.304|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]: Client configuration:
%7|1689252905.304|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: Broadcasting state change
%7|1689252905.304|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: broker in state TRY_CONNECT connecting
%7|1689252905.304|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1689252905.304|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: Broadcasting state change
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   client.software.name = confluent-kafka-python
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   client.software.version = 2.1.1-rdkafka-2.1.1
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   metadata.broker.list = redpanda-0.redpanda.redpanda.svc.cluster.local:9093,redpanda-1.redpanda.redpanda.svc.cluster.local:9093,redpanda-2.redpanda.redpanda.svc.cluster.local:9093
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   debug = generic,broker,topic,metadata,feature,queue,msg,protocol,cgrp,security,fetch,interceptor,plugin,consumer,admin,eos,mock,assignor,conf,all
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   error_cb = 0x7f6b383e3860
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   opaque = 0x7f6b3904fa60
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   security.protocol = sasl_ssl
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   ssl.ca.location = /tmp/ca.crt
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   sasl.mechanisms = SCRAM-SHA-256
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   sasl.username = [redacted]
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   sasl.password = [redacted]
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   dr_msg_cb = 0x7f6b383e12c0
%4|1689252905.304|TERMINATE|rdkafka#producer-1| [thrd:app]: Producer terminating with 1 message (4 bytes) still in queue or transit: use flush() to wait for outstanding message delivery
%7|1689252905.305|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Connecting to ipv4#10.244.13.5:9093 (sasl_ssl) with socket 3
%7|1689252905.305|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Connected to ipv4#10.244.13.5:9093
%7|1689252905.305|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker changed state CONNECT -> SSL_HANDSHAKE
%7|1689252905.305|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: Broadcasting state change
%7|1689252905.305|ENDPOINT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabled endpoint identification using hostname redpanda-1.redpanda.redpanda.svc.cluster.local
>>> %7|1689252905.308|SSLVERIFY|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker SSL certificate verified
%7|1689252905.308|CONNECTED|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Connected (#1)
%7|1689252905.308|FEATURE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1689252905.308|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker changed state SSL_HANDSHAKE -> APIVERSION_QUERY
%7|1689252905.308|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: Broadcasting state change
%7|1689252905.308|SEND|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Sent ApiVersionRequest (v3, 66 bytes @ 0, CorrId 1)
%7|1689252905.350|RECV|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received ApiVersionResponse (v3, 281 bytes, CorrId 1, rtt 41.06ms)
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker API support:
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey Produce (0) Versions 0..7
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey Fetch (1) Versions 4..11
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey ListOffsets (2) Versions 0..4
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey Metadata (3) Versions 0..7
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey OffsetFetch (9) Versions 1..7
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey OffsetDeleteRequest (47) Versions 0..0
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey FindCoordinator (10) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey ListGroups (16) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey ApiVersion (18) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey JoinGroup (11) Versions 0..6
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey Heartbeat (12) Versions 0..4
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey LeaveGroup (13) Versions 0..4
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey SyncGroup (14) Versions 0..4
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey CreateTopics (19) Versions 0..6
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey OffsetCommit (8) Versions 1..8
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DescribeConfigs (32) Versions 0..4
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey AlterConfigs (33) Versions 0..2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DeleteTopics (20) Versions 0..5
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DescribeGroups (15) Versions 0..5
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey SaslHandshake (17) Versions 0..1
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey SaslAuthenticate (36) Versions 0..2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey IncrementalAlterConfigsRequest (44) Versions 0..1
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DeleteGroups (42) Versions 0..2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DescribeAcls (29) Versions 0..2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DescribeLogDirs (35) Versions 0..2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey CreateAcls (30) Versions 0..2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DeleteAcls (31) Versions 0..2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey InitProducerId (22) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey AddPartitionsToTxn (24) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey TxnOffsetCommit (28) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey AddOffsetsToTxn (25) Versions 0..1
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey EndTxn (26) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey CreatePartitions (37) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey OffsetForLeaderEpoch (23) Versions 0..4
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey AlterPartitionReassignmentsRequest (45) Versions 0..0
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey ListPartitionReassignmentsRequest (46) Versions 0..0
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DescribeProducers (61) Versions 0..0
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DescribeTransactions (65) Versions 0..0
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey ListTransactions (66) Versions 0..0
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature MsgVer1: Produce (2..2) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature MsgVer1: Fetch (2..2) NOT supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Disabling feature MsgVer1
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature MsgVer2: Produce (3..3) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature MsgVer2: Fetch (4..4) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature MsgVer2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature ApiVersion: ApiVersion (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature ApiVersion
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerGroupCoordinator: FindCoordinator (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature BrokerGroupCoordinator
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerBalancedConsumer: FindCoordinator (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerBalancedConsumer: OffsetCommit (1..2) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerBalancedConsumer: OffsetFetch (1..1) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerBalancedConsumer: JoinGroup (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerBalancedConsumer: SyncGroup (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerBalancedConsumer: Heartbeat (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerBalancedConsumer: LeaveGroup (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature BrokerBalancedConsumer
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature ThrottleTime: Produce (1..2) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature ThrottleTime: Fetch (1..2) NOT supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Disabling feature ThrottleTime
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature Sasl: JoinGroup (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature Sasl
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature SaslHandshake: SaslHandshake (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature SaslHandshake
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature LZ4: FindCoordinator (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature LZ4
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature OffsetTime: ListOffsets (1..1) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature OffsetTime
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature IdempotentProducer: InitProducerId (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature IdempotentProducer
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature ZSTD: Produce (7..7) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature ZSTD: Fetch (10..10) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature ZSTD
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature SaslAuthReq: SaslHandshake (1..1) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature SaslAuthReq: SaslAuthenticate (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature SaslAuthReq
%7|1689252905.350|FEATURE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Updated enabled protocol features to ApiVersion,BrokerBalancedConsumer,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq
%7|1689252905.350|AUTH|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Auth in state APIVERSION_QUERY (handshake supported)
%7|1689252905.350|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker changed state APIVERSION_QUERY -> AUTH_HANDSHAKE
%7|1689252905.351|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: Broadcasting state change
%7|1689252905.351|SEND|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Sent SaslHandshakeRequest (v1, 36 bytes @ 0, CorrId 2)
%7|1689252905.351|RECV|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received SaslHandshakeResponse (v1, 36 bytes, CorrId 2, rtt 0.34ms)
%7|1689252905.351|SASLMECHS|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker supported SASL mechanisms: SCRAM-SHA-256,SCRAM-SHA-512
%7|1689252905.351|AUTH|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Auth in state AUTH_HANDSHAKE (handshake supported)
%7|1689252905.351|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker changed state AUTH_HANDSHAKE -> AUTH_REQ
%7|1689252905.351|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: Broadcasting state change
%7|1689252905.351|SASL|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Initializing SASL client: service name kafka, hostname redpanda-1.redpanda.redpanda.svc.cluster.local, mechanisms SCRAM-SHA-256, provider SCRAM (builtin)
%7|1689252905.351|SASLSCRAM|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: SASL SCRAM client in state client-first-message
%7|1689252905.351|SASL|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Send SASL Kafka frame to broker (43 bytes)
%7|1689252905.351|SEND|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Sent SaslAuthenticateRequest (v0, 68 bytes @ 0, CorrId 3)
%7|1689252905.351|RECV|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received SaslAuthenticateResponse (v0, 358 bytes, CorrId 3, rtt 0.25ms)
%7|1689252905.351|SASL|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received SASL frame from broker (350 bytes)
%7|1689252905.351|SASLSCRAM|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: SASL SCRAM client in state server-first-message
%7|1689252905.379|SASL|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Send SASL Kafka frame to broker (250 bytes)
%7|1689252905.379|SEND|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Sent SaslAuthenticateRequest (v0, 275 bytes @ 0, CorrId 4)
%7|1689252905.420|RECV|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received SaslAuthenticateResponse (v0, 54 bytes, CorrId 4, rtt 40.99ms)
%7|1689252905.420|SASL|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received SASL frame from broker (46 bytes)
%7|1689252905.420|SASLSCRAM|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: SASL SCRAM client in state client-final-message
%7|1689252905.420|SCRAMAUTH|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: SASL SCRAM authentication successful on server: verifying ServerSignature
%7|1689252905.420|SCRAMAUTH|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Authenticated as therealusername using SCRAM-SHA-256
%7|1689252905.420|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker changed state AUTH_REQ -> UP
%7|1689252905.420|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: Broadcasting state change
%7|1689252905.420|METADATA|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Request metadata for brokers only: connected
%7|1689252905.420|SEND|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Sent MetadataRequest (v7, 26 bytes @ 0, CorrId 5)
%7|1689252905.420|RECV|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received MetadataResponse (v7, 240 bytes, CorrId 5, rtt 0.39ms)
%7|1689252905.420|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: ===== Received metadata: connected =====
%7|1689252905.420|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: ClusterId: redpanda.9dc56407-546e-44ca-83ed-f45b2aabefb9, ControllerId: 0
%7|1689252905.420|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: 3 brokers, 0 topics
%7|1689252905.420|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   Broker #0/3: redpanda-0.redpanda.redpanda.svc.cluster.local.:9093 NodeId 0
%7|1689252905.420|WAKEUPFD|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local.:9093/0: Enabled low-latency ops queue wake-ups
%7|1689252905.420|BROKER|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local.:9093/0: Added new broker with NodeId 0
%7|1689252905.421|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   Broker #1/3: redpanda-1.redpanda.redpanda.svc.cluster.local.:9093 NodeId 1
%7|1689252905.421|WAKEUPFD|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Enabled low-latency ops queue wake-ups
%7|1689252905.421|BRKMAIN|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local.:9093/0: Enter main broker thread
%7|1689252905.421|BROKER|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Added new broker with NodeId 1
%7|1689252905.421|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   Broker #2/3: redpanda-2.redpanda.redpanda.svc.cluster.local.:9093 NodeId 2
%7|1689252905.421|WAKEUPFD|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local.:9093/2: Enabled low-latency ops queue wake-ups
%7|1689252905.421|BRKMAIN|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Enter main broker thread
%7|1689252905.421|BROKER|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local.:9093/2: Added new broker with NodeId 2
%7|1689252905.421|CLUSTERID|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: ClusterId update "" -> "redpanda.9dc56407-546e-44ca-83ed-f45b2aabefb9"
%7|1689252905.421|CONTROLLERID|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: ControllerId update -1 -> 0
%7|1689252905.421|BROADCAST|rdkafka#producer-2| [thrd:main]: Broadcasting state change
%7|1689252905.421|BRKMAIN|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local.:9093/2: Enter main broker thread

>>> p.produce('t1', 'test'.encode())
%7|1689252908.620|TOPIC|rdkafka#producer-2| [thrd:app]: New local topic: t1
%7|1689252908.620|TOPPARNEW|rdkafka#producer-2| [thrd:app]: NEW t1 [-1] 0x561366fb9340 refcnt 0x561366fb93d0 (at rd_kafka_topic_new0:471)
>>> %7|1689252909.304|NOINFO|rdkafka#producer-2| [thrd:main]: Topic t1 metadata information unknown
%7|1689252909.304|NOINFO|rdkafka#producer-2| [thrd:main]: Topic t1 partition count is zero: should refresh metadata
%7|1689252909.304|METADATA|rdkafka#producer-2| [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
%7|1689252909.304|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Request metadata for 1 topic(s): refresh unavailable topics
%7|1689252909.304|SEND|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Sent MetadataRequest (v7, 30 bytes @ 0, CorrId 6)
%7|1689252909.305|RECV|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received MetadataResponse (v7, 285 bytes, CorrId 6, rtt 0.35ms)
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: ClusterId: redpanda.9dc56407-546e-44ca-83ed-f45b2aabefb9, ControllerId: 0
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: 3 brokers, 1 topics
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   Topic t1 with 1 partitions
%7|1689252909.305|STATE|rdkafka#producer-2| [thrd:main]: Topic t1 changed state unknown -> exists
%7|1689252909.305|PARTCNT|rdkafka#producer-2| [thrd:main]: Topic t1 partition count changed from 0 to 1
%7|1689252909.305|TOPPARNEW|rdkafka#producer-2| [thrd:main]: NEW t1 [0] 0x7f6b08007c40 refcnt 0x7f6b08007cd0 (at rd_kafka_topic_partition_cnt_update:908)
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]:   Topic t1 partition 0 Leader 1 Epoch -1
%7|1689252909.305|BRKDELGT|rdkafka#producer-2| [thrd:main]: t1 [0]: delegate to broker sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1 (rktp 0x7f6b08007c40, term 0, ref 2)
%7|1689252909.305|BRKDELGT|rdkafka#producer-2| [thrd:main]: t1 [0]: delegating to broker sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1 for partition with 0 messages (0 bytes) queued
%7|1689252909.305|BRKMIGR|rdkafka#producer-2| [thrd:main]: Migrating topic t1 [0] 0x7f6b08007c40 from (none) to sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1 (sending PARTITION_JOIN to sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1)
%7|1689252909.305|PARTCNT|rdkafka#producer-2| [thrd:main]: Partitioning 1 unassigned messages in topic t1 to 1 partitions
%7|1689252909.305|PARTITIONER|rdkafka#producer-2| [thrd:main]: t1 [0] is the new sticky partition
%7|1689252909.305|TOPBRK|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Topic t1 [0]: joining broker (rktp 0x7f6b08007c40, 0 message(s) queued)
%7|1689252909.305|FETCHADD|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Added t1 [0] to active list (1 entries, opv 0, 0 messages queued): joining
%7|1689252909.305|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: Broadcasting state change
%7|1689252909.305|UAS|rdkafka#producer-2| [thrd:main]: 1/1 messages were partitioned in topic t1
%7|1689252909.305|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) in xmit queue (1 added from partition queue)
%7|1689252909.305|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) queued but broker not up
%7|1689252909.305|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Broker changed state INIT -> TRY_CONNECT
%7|1689252909.305|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: Broadcasting state change
%7|1689252909.305|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: broker in state TRY_CONNECT connecting
%7|1689252909.305|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Broker changed state TRY_CONNECT -> CONNECT
%7|1689252909.305|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: Broadcasting state change
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   Broker #0/3: redpanda-0.redpanda.redpanda.svc.cluster.local.:9093 NodeId 0
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   Broker #1/3: redpanda-1.redpanda.redpanda.svc.cluster.local.:9093 NodeId 1
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   Broker #2/3: redpanda-2.redpanda.redpanda.svc.cluster.local.:9093 NodeId 2
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: 1/1 requested topic(s) seen in metadata
%7|1689252909.306|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Connecting to ipv4#10.244.13.5:9093 (sasl_ssl) with socket 10
%7|1689252909.306|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) in xmit queue (0 added from partition queue)
%7|1689252909.306|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) queued but broker not up
%7|1689252909.306|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) in xmit queue (0 added from partition queue)
%7|1689252909.306|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) queued but broker not up
%7|1689252909.306|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Connected to ipv4#10.244.13.5:9093
%7|1689252909.306|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Broker changed state CONNECT -> SSL_HANDSHAKE
%7|1689252909.306|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: Broadcasting state change
%7|1689252909.306|ENDPOINT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Enabled endpoint identification using hostname redpanda-1.redpanda.redpanda.svc.cluster.local.
%7|1689252909.307|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) in xmit queue (0 added from partition queue)
%7|1689252909.307|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) queued but broker not up
%7|1689252909.307|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) in xmit queue (0 added from partition queue)
%7|1689252909.307|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) queued but broker not up
%7|1689252909.309|FAIL|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: SSL handshake failed: ssl/statem/statem_clnt.c:1890:tls_post_process_server_certificate error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 3ms in state SSL_HANDSHAKE) (_SSL)

Checklist

Please provide the following information:

  • confluent-kafka-python and librdkafka version ( and confluent_kafka.libversion()):
>>> import confluent_kafka
>>> confluent_kafka.version()
('2.1.1', 33620224)
>>> confluent_kafka.libversion()
('2.1.1', 33620479)
  • Apache Kafka broker version: Redpanda v23.1.13
  • Client configuration: {'bootstrap.servers': 'redpanda-0.redpanda.redpanda.svc.cluster.local:9093,redpanda-1.redpanda.redpanda.svc.cluster.local:9093,redpanda-2.redpanda.redpanda.svc.cluster.local:9093', 'security.protocol': 'SASL_SSL', 'sasl.mechanisms': 'SCRAM-SHA-256', 'sasl.username': 'therightusernamehere', 'sasl.password': 'therightpwhere', 'ssl.ca.location': '/tmp/ca.crt'}
  • Operating system: Ubuntu - Linux 5.10.176+ kafkatest, refactoring, and bug fixes #1 SMP Sat May 6 15:10:33 UTC 2023 x86_64 GNU/Linux
  • Provide client logs (with 'debug': '..' as necessary)
  • Provide broker log excerpts (nothing in the broker logs)
  • Critical issue
@sbv-csis
Copy link
Author

Sounds like it could be related to confluentinc/librdkafka#4348

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@sbv-csis