- feat: update go directive to 1.18 by @dnwe in #2713
- feat: return KError instead of errors in AlterConfigs and DescribeConfig by @zhuliquan in #2472
- fix: don't waste time for backoff on member id required error by @lzakharov in #2759
- fix: prevent ConsumerGroup.Close infinitely locking by @maqdev in #2717
- chore(deps): bump golang.org/x/net from 0.17.0 to 0.18.0 by @dependabot in #2716
- chore(deps): bump golang.org/x/sync to v0.5.0 by @dependabot in #2718
- chore(deps): bump github.com/pierrec/lz4/v4 from 4.1.18 to 4.1.19 by @dependabot in #2739
- chore(deps): bump golang.org/x/crypto from 0.15.0 to 0.17.0 by @dependabot in #2748
- chore(deps): bump the golang-org-x group with 1 update by @dependabot in #2734
- chore(deps): bump the golang-org-x group with 2 updates by @dependabot in #2764
- chore(deps): bump github.com/pierrec/lz4/v4 from 4.1.19 to 4.1.21 by @dependabot in #2763
- chore(deps): bump golang.org/x/crypto from 0.15.0 to 0.17.0 in /examples/exactly_once by @dependabot in #2749
- chore(deps): bump golang.org/x/crypto from 0.15.0 to 0.17.0 in /examples/consumergroup by @dependabot in #2750
- chore(deps): bump golang.org/x/crypto from 0.15.0 to 0.17.0 in /examples/sasl_scram_client by @dependabot in #2751
- chore(deps): bump golang.org/x/crypto from 0.15.0 to 0.17.0 in /examples/interceptors by @dependabot in #2752
- chore(deps): bump golang.org/x/crypto from 0.15.0 to 0.17.0 in /examples/http_server by @dependabot in #2753
- chore(deps): bump github.com/eapache/go-resiliency from 1.4.0 to 1.5.0 by @dependabot in #2745
- chore(deps): bump golang.org/x/crypto from 0.15.0 to 0.17.0 in /examples/txn_producer by @dependabot in #2754
- chore(deps): bump go.opentelemetry.io/otel/sdk from 1.19.0 to 1.22.0 in /examples/interceptors by @dependabot in #2767
- chore(deps): bump the golang-org-x group with 1 update by @dependabot in #2793
- chore(deps): bump go.opentelemetry.io/otel/exporters/stdout/stdoutmetric from 0.42.0 to 1.23.1 in /examples/interceptors by @dependabot in #2792
- fix(examples): housekeeping of code and deps by @dnwe in #2720
- fix(test): retry MockBroker Listen for EADDRINUSE by @dnwe in #2721
Full Changelog: https://github.com/IBM/sarama/compare/v1.42.1...v1.42.2
- fix: make fetchInitialOffset use correct protocol by @dnwe in #2705
- fix(config): relax ClientID validation after 1.0.0 by @dnwe in #2706
Full Changelog: https://github.com/IBM/sarama/compare/v1.42.0...v1.42.1
- Asynchronously close brokers during a RefreshBrokers by @bmassemin in #2693
- Fix data race on Broker.done channel by @prestona in #2698
- fix: data race in Broker.AsyncProduce by @lzakharov in #2678
- Fix default retention time value in offset commit by @prestona in #2700
- fix(txmgr): ErrOffsetsLoadInProgress is retriable by @dnwe in #2701
- chore(ci): improve ossf scorecard result by @dnwe in #2685
- chore(ci): add kafka 3.6.0 to FVT and versions by @dnwe in #2692
- chore(ci): ossf scorecard.yml by @dnwe in #2683
- fix(ci): always run CodeQL on every commit by @dnwe in #2689
- chore(doc): add OpenSSF Scorecard badge by @dnwe in #2691
Full Changelog: https://github.com/IBM/sarama/compare/v1.41.3...v1.42.0
- fix: pre-compile regex for parsing kafka version by @qshuai in #2663
- fix(client): ignore empty Metadata responses when refreshing by @HaoSunUber in #2672
- chore(deps): bump the golang-org-x group with 2 updates by @dependabot in #2661
- chore(deps): bump golang.org/x/net from 0.16.0 to 0.17.0 by @dependabot in #2671
- fix(docs): correct topic name in rebalancing strategy example by @maksadbek in #2657
Full Changelog: https://github.com/IBM/sarama/compare/v1.41.2...v1.41.3
- perf: Alloc records in batch by @ronanh in #2646
- fix(consumer): guard against nil client by @dnwe in #2636
- fix(consumer): don't retry session if ctx canceled by @dnwe in #2642
- fix: use least loaded broker to refresh metadata by @HaoSunUber in #2645
- chore(deps): bump the golang-org-x group with 1 update by @dependabot in #2641
- @HaoSunUber made their first contribution in #2645
Full Changelog: https://github.com/IBM/sarama/compare/v1.41.1...v1.41.2
- fix(proto): handle V3 member metadata and empty owned partitions by @dnwe in #2618
- fix: make clear that error is configuration issue not server error by @hindessm in #2628
- fix(client): force Event Hubs to use V1_0_0_0 by @dnwe in #2633
- fix: add retries to alter user scram creds by @hindessm in #2632
- chore(lint): bump golangci-lint and tweak config by @dnwe in #2620
- fix(doc): add missing doc for mock consumer by @hsweif in #2386
- chore(proto): doc CreateTopics/JoinGroup fields by @dnwe in #2627
- chore(gh): add new style issue templates by @dnwe in #2624
Full Changelog: https://github.com/IBM/sarama/compare/v1.41.0...v1.41.1
Note: this version of Sarama has had a big overhaul in its adherence to the use of the right Kafka protocol versions for the given Config Version. It has also bumped the default Version set in Config (where one is not supplied) to 2.1.0. This is in preparation for Kafka 4.0 dropping support for protocol versions older than 2.1. If you are using Sarama against Kafka clusters older than v2.1.0, or using it against Azure EventHubs then you will likely have to change your application code to pin to the appropriate Version.
- chore(config): make DefaultVersion V2_0_0_0 by @dnwe in #2572
- chore(config): make DefaultVersion V2_1_0_0 by @dnwe in #2574
- Implement resolve_canonical_bootstrap_servers_only by @gebn in #2156
- feat: sleep when throttled (KIP-219) by @hindessm in #2536
- feat: add isValidVersion to protocol types by @dnwe in #2538
- fix(consumer): use newer LeaveGroup as appropriate by @dnwe in #2544
- Add support for up to version 4 List Groups API by @prestona in #2541
- fix(producer): use newer ProduceReq as appropriate by @dnwe in #2546
- fix(proto): ensure req+resp requiredVersion match by @dnwe in #2548
- chore(proto): permit CreatePartitionsRequest V1 by @dnwe in #2549
- chore(proto): permit AlterConfigsRequest V1 by @dnwe in #2550
- chore(proto): permit DeleteGroupsRequest V1 by @dnwe in #2551
- fix(proto): correct JoinGroup usage for wider version range by @dnwe in #2553
- fix(consumer): use full range of FetchRequest vers by @dnwe in #2554
- fix(proto): use range of OffsetCommitRequest vers by @dnwe in #2555
- fix(proto): use full range of MetadataRequest by @dnwe in #2556
- fix(proto): use fuller ranges of supported proto by @dnwe in #2558
- fix(proto): use full range of SyncGroupRequest by @dnwe in #2565
- fix(proto): use full range of ListGroupsRequest by @dnwe in #2568
- feat(proto): support for Metadata V6-V10 by @dnwe in #2566
- fix(proto): use full ranges for remaining proto by @dnwe in #2570
- feat(proto): add remaining protocol for V2.1 by @dnwe in #2573
- feat: add new error for MockDeleteTopicsResponse by @javiercri in #2475
- feat(gzip): switch to klauspost/compress gzip by @dnwe in #2600
- fix: correct unsupported version check by @hindessm in #2528
- fix: avoiding burning cpu if all partitions are paused by @napallday in #2532
- extend throttling metric scope by @hindessm in #2533
- Fix printing of final metrics by @prestona in #2545
- fix(consumer): cannot automatically fetch newly-added partitions unless restart by @napallday in #2563
- bug: implement unsigned modulus for partitioning with crc32 hashing by @csm8118 in #2560
- fix: avoid logging value of proxy.Dialer by @prestona in #2569
- fix(test): add missing closes to admin client tests by @dnwe in #2594
- fix(test): ensure some more clients are closed by @dnwe in #2595
- fix(examples): sync exactly_once and consumergroup by @dnwe in #2614
- fix(fvt): fresh metrics registry for each test by @dnwe in #2616
- fix(test): flaky test TestFuncOffsetManager by @napallday in #2609
- chore(deps): bump the golang-org-x group with 1 update by @dependabot in #2542
- chore(deps): bump the golang-org-x group with 1 update by @dependabot in #2561
- chore(deps): bump module github.com/pierrec/lz4/v4 to v4.1.18 by @dnwe in #2589
- chore(deps): bump module github.com/jcmturner/gokrb5/v8 to v8.4.4 by @dnwe in #2587
- chore(deps): bump github.com/eapache/go-xerial-snappy digest to c322873 by @dnwe in #2586
- chore(deps): bump module github.com/klauspost/compress to v1.16.7 by @dnwe in #2588
- chore(deps): bump github.com/eapache/go-resiliency from 1.3.0 to 1.4.0 by @dependabot in #2598
- fix(fvt): ensure fully-replicated at test start by @hindessm in #2531
- chore: rollup fvt kafka to latest three by @dnwe in #2537
- Merge the two CONTRIBUTING.md's by @prestona in #2543
- fix(test): test timing error by @hindessm in #2552
- chore(ci): tidyup and improve actions workflows by @dnwe in #2557
- fix(test): shutdown MockBroker by @dnwe in #2571
- chore(proto): match HeartbeatResponse version by @dnwe in #2576
- chore(test): ensure MockBroker closed within test by @dnwe in #2575
- chore(test): ensure all mockresponses use version by @dnwe in #2578
- chore(ci): use latest Go in actions by @dnwe in #2580
- chore(test): speedup some slow tests by @dnwe in #2579
- chore(test): use modern protocol versions in FVT by @dnwe in #2581
- chore(test): fix a couple of leaks by @dnwe in #2591
- feat(fvt): experiment with per-kafka-version image by @dnwe in #2592
- chore(ci): replace toxiproxy client dep by @dnwe in #2593
- feat(fvt): add healthcheck, depends_on and --wait by @dnwe in #2601
- fix(fvt): handle msgset vs batchset by @dnwe in #2603
- fix(fvt): Metadata version in ensureFullyReplicated by @dnwe in #2612
- fix(fvt): versioned cfg for invalid topic producer by @dnwe in #2613
- chore(fvt): tweak to work across more versions by @dnwe in #2615
- feat(fvt): test wider range of kafkas by @dnwe in #2605
- fix(example): check if msg channel is closed by @ioanzicu in #2479
- chore: use go install for installing sarama tools by @vigith in #2599
- @gebn made their first contribution in #2156
- @prestona made their first contribution in #2543
- @ioanzicu made their first contribution in #2479
- @csm8118 made their first contribution in #2560
- @javiercri made their first contribution in #2475
- @vigith made their first contribution in #2599
Full Changelog: https://github.com/IBM/sarama/compare/v1.40.1...v1.41.0
- Use buffer pools for decompression by @ronanh in #2484
- feat: support for Kerberos authentication with a credentials cache. by @mrogaski in #2457
- Fix some retry issues by @hindessm in #2517
- fix: admin retry logic by @hindessm in #2519
- Add some retry logic to more admin client functions by @hindessm in #2520
- fix: concurrent issue on updateMetadataMs by @napallday in #2522
- fix(test): allow testing of skipped test without IsTransactional panic by @hindessm in #2525
- chore(deps): bump the golang-org-x group with 2 updates by @dependabot in #2509
- chore(deps): bump github.com/klauspost/compress from 1.15.14 to 1.16.6 by @dependabot in #2513
- chore(deps): bump github.com/stretchr/testify from 1.8.1 to 1.8.3 by @dependabot in #2512
- chore(ci): migrate probot-stale to actions/stale by @dnwe in #2496
- chore(ci): bump golangci version, cleanup, depguard config by @EladLeev in #2504
- Clean up some typos and docs/help mistakes by @hindessm in #2514
- chore(ci): add simple apidiff workflow by @dnwe in #2497
- chore(ci): bump actions/setup-go from 3 to 4 by @dependabot in #2508
- fix(comments): PauseAll and ResumeAll by @napallday in #2523
- @EladLeev made their first contribution in #2504
- @hindessm made their first contribution in #2514
- @ronanh made their first contribution in #2484
- @mrogaski made their first contribution in #2457
Full Changelog: https://github.com/IBM/sarama/compare/v1.40.0...v1.40.1
Note: this is the first release after the transition of Sarama ownership from Shopify to IBM in #2461
- chore: migrate module to github.com/IBM/sarama by @dnwe in #2492
- fix: restore (*OffsetCommitRequest) AddBlock func by @dnwe in #2494
- fix(consumer): don't retry FindCoordinator forever by @dnwe in #2427
- fix(metrics): fix race condition when calling Broker.Open() twice by @vincentbernat in #2428
- fix: use version 4 of DescribeGroupsRequest only if kafka broker vers… …ion is >= 2.4 by @faillefer in #2451
- Fix HighWaterMarkOffset of mocks partition consumer by @gr8web in #2447
- fix: prevent data race in balance strategy by @napallday in #2453
- chore(deps): bump golang.org/x/net from 0.5.0 to 0.7.0 by @dependabot in #2452
- chore: add kafka 3.3.2 by @dnwe in #2434
- chore(ci): remove Shopify/shopify-cla-action by @dnwe in #2489
- chore: bytes.Equal instead bytes.Compare by @testwill in #2485
- @dependabot made their first contribution in #2452
- @gr8web made their first contribution in #2447
- @testwill made their first contribution in #2485
Full Changelog: https://github.com/IBM/sarama/compare/v1.38.1...v1.40.0
- fix(example): correct
records-numberparam in txn producer readme by @diallo-han in #2420 - fix: use newConsumer method in newConsumerGroup method by @Lumotheninja in #2424
- chore(deps): bump module github.com/klauspost/compress to v1.15.14 by @dnwe in #2410
- chore(deps): bump module golang.org/x/net to v0.5.0 by @dnwe in #2413
- chore(deps): bump module github.com/stretchr/testify to v1.8.1 by @dnwe in #2411
- chore(deps): bump module github.com/xdg-go/scram to v1.1.2 by @dnwe in #2412
- chore(deps): bump module golang.org/x/sync to v0.1.0 by @dnwe in #2414
- chore(deps): bump github.com/eapache/go-xerial-snappy digest to bf00bc1 by @dnwe in #2418
- @diallo-han made their first contribution in #2420
- @Lumotheninja made their first contribution in #2424
Full Changelog: https://github.com/IBM/sarama/compare/v1.38.0...v1.38.1
- feat(producer): improve memory usage of zstd encoder by using our own pool management by @rtreffer in #2375
- feat(proto): implement and use MetadataRequest v7 by @dnwe in #2388
- feat(metrics): add protocol-requests-rate metric by @auntan in #2373
- fix(proto): track and supply leader epoch to FetchRequest by @dnwe in #2389
- fix(example): improve arg name used for tls skip verify by @michaeljmarshall in #2385
- fix(zstd): default back to GOMAXPROCS concurrency by @bgreenlee in #2404
- fix(producer): add nil check while producer is retrying by @hsweif in #2387
- fix(producer): return errors for every message in retryBatch to avoid producer hang forever by @cch123 in #2378
- fix(metrics): fix race when accessing metric registry by @vincentbernat in #2409
- chore(deps): bump golang.org/x/net to v0.4.0 by @dnwe in #2403
- chore(ci): replace set-output command in GH Action by @dnwe in #2390
- chore(ci): include kafka 3.3.1 in testing matrix by @dnwe in #2406
- @michaeljmarshall made their first contribution in #2385
- @bgreenlee made their first contribution in #2404
- @hsweif made their first contribution in #2387
- @cch123 made their first contribution in #2378
Full Changelog: https://github.com/IBM/sarama/compare/v1.37.2...v1.38.0
- fix: ensure updateMetaDataMs is 64-bit aligned by @dnwe in #2356
- fix: bump go.mod specification to go 1.17 by @dnwe in #2357
Full Changelog: https://github.com/IBM/sarama/compare/v1.37.1...v1.37.2
- fix: support existing deprecated Rebalance.Strategy field usage by @spongecaptain in #2352
- fix(test): consumer group rebalance strategy compatibility by @Jacob-bzx in #2353
- fix(producer): replace time.After with time.Timer to avoid high memory usage by @Jacob-bzx in #2355
- @spongecaptain made their first contribution in #2352
Full Changelog: https://github.com/IBM/sarama/compare/v1.37.0...v1.37.1
- Due to a change in github.com/klauspost/compress v1.15.10, Sarama v1.37.0 requires Go 1.17 going forward, unfortunately due to an oversight this wasn't reflected in the go.mod declaration at time of release.
- feat(consumer): support multiple balance strategies by @Jacob-bzx in #2339
- feat(producer): transactional API by @ryarnyah in #2295
- feat(mocks): support key in MockFetchResponse. by @Skandalik in #2328
- fix: avoid panic when Metadata.RefreshFrequency is 0 by @Jacob-bzx in #2329
- fix(consumer): avoid pushing unrelated responses to paused children by @pkoutsovasilis in #2317
- fix: prevent metrics leak with cleanup by @auntan in #2340
- fix: race condition(may panic) when closing consumer group by @Jacob-bzx in #2331
- fix(consumer): default ResetInvalidOffsets to true by @dnwe in #2345
- Validate the
Configwhen creating a mock producer/consumer by @joewreschnig in #2327
- chore(deps): bump module github.com/pierrec/lz4/v4 to v4.1.16 by @dnwe in #2335
- chore(deps): bump golang.org/x/net digest to bea034e by @dnwe in #2333
- chore(deps): bump golang.org/x/sync digest to 7f9b162 by @dnwe in #2334
- chore(deps): bump golang.org/x/net digest to f486391 by @dnwe in #2348
- chore(deps): bump module github.com/shopify/toxiproxy/v2 to v2.5.0 by @dnwe in #2336
- chore(deps): bump module github.com/klauspost/compress to v1.15.11 by @dnwe in #2349
- chore(deps): bump module github.com/pierrec/lz4/v4 to v4.1.17 by @dnwe in #2350
- chore(ci): bump kafka-versions to latest by @dnwe in #2346
- chore(ci): bump go-versions to N and N-1 by @dnwe in #2347
- @Jacob-bzx made their first contribution in #2329
- @pkoutsovasilis made their first contribution in #2317
- @Skandalik made their first contribution in #2328
- @auntan made their first contribution in #2340
- @ryarnyah made their first contribution in #2295
Full Changelog: https://github.com/IBM/sarama/compare/v1.36.0...v1.37.0
- feat: add option to propagate OffsetOutOfRange error by @dkolistratova in #2252
- feat(producer): expose ProducerMessage.byteSize() function by @k8scat in #2315
- feat(metrics): track consumer fetch request rates by @dnwe in #2299
- fix(consumer): avoid submitting empty fetch requests when paused by @raulnegreiros in #2143
- chore(deps): bump module github.com/klauspost/compress to v1.15.9 by @dnwe in #2304
- chore(deps): bump golang.org/x/net digest to c7608f3 by @dnwe in #2301
- chore(deps): bump golangci/golangci-lint-action action to v3 by @dnwe in #2311
- chore(deps): bump golang.org/x/net digest to 07c6da5 by @dnwe in #2307
- chore(deps): bump github actions versions (major) by @dnwe in #2313
- chore(deps): bump module github.com/jcmturner/gofork to v1.7.6 by @dnwe in #2305
- chore(deps): bump golang.org/x/sync digest to 886fb93 by @dnwe in #2302
- chore(deps): bump module github.com/jcmturner/gokrb5/v8 to v8.4.3 by @dnwe in #2303
- chore: add kafka 3.1.1 to the version matrix by @dnwe in #2300
- Migrate off probot-CLA to new GitHub Action by @cursedcoder in #2294
- Forgot to remove cla probot by @cursedcoder in #2297
- chore(lint): re-enable a small amount of go-critic by @dnwe in #2312
- @cursedcoder made their first contribution in #2294
- @dkolistratova made their first contribution in #2252
- @k8scat made their first contribution in #2315
Full Changelog: https://github.com/IBM/sarama/compare/v1.35.0...v1.36.0
- fix: fix metadata retry backoff invalid when get metadata failed by @Stephan14 in #2256
- fix(balance): sort and de-deplicate memberIDs by @dnwe in #2285
- fix: prevent DescribeLogDirs hang in admin client by @zerowidth in #2269
- fix: include assignment-less members in SyncGroup by @dnwe in #2292
- chore(deps): bump module github.com/stretchr/testify to v1.8.0 by @dnwe in #2284
- chore(deps): bump module github.com/eapache/go-resiliency to v1.3.0 by @dnwe in #2283
- chore(deps): bump golang.org/x/net digest to 1185a90 by @dnwe in #2279
- chore(deps): bump module github.com/pierrec/lz4/v4 to v4.1.15 by @dnwe in #2281
- chore(deps): bump module github.com/klauspost/compress to v1.15.8 by @dnwe in #2280
- chore: rename
anyfunc to avoid identifier by @dnwe in #2272 - chore: add and test against kafka 3.2.0 by @dnwe in #2288
- chore: document Fetch protocol fields by @dnwe in #2289
- chore(ci): fix redirect with GITHUB_STEP_SUMMARY by @dnwe in #2286
- fix(test): permit ECONNRESET in TestInitProducerID by @dnwe in #2287
- fix: ensure empty or devel version valid by @dnwe in #2291
- @zerowidth made their first contribution in #2269
Full Changelog: https://github.com/IBM/sarama/compare/v1.34.1...v1.35.0
- fix(examples): check session.Context().Done() in examples/consumergroup by @zxc111 in #2240
- fix(protocol): move AuthorizedOperations into GroupDescription of DescribeGroupsResponse by @aiquestion in #2247
- fix(protocol): tidyup DescribeGroupsResponse by @dnwe in #2248
- fix(consumer): range balance strategy not like reference by @njhartwell in #2245
- chore(ci): experiment with using tparse by @dnwe in #2236
- chore(deps): bump thirdparty dependencies to latest releases by @dnwe in #2242
Full Changelog: https://github.com/IBM/sarama/compare/v1.34.0...v1.34.1
- KIP-345: support static membership by @aiquestion in #2230
- fix: KIP-368 use receiver goroutine to process all sasl v1 responses by @k-wall in #2234
- chore(deps): bump module github.com/pierrec/lz4 to v4 by @dnwe in #2231
- chore(deps): bump golang.org/x/net digest to 2e3eb7b by @dnwe in #2232
- @aiquestion made their first contribution in #2230
Full Changelog: https://github.com/IBM/sarama/compare/v1.33.0...v1.34.0
Note: with this change, the user of Sarama is required to use Go 1.13's errors.Is etc (rather then ==) when forming conditionals returned by this library.
- feat: make
ErrOutOfBrokerswrap the underlying error that prevented connections to the brokers by @k-wall in #2131
- feat(message): add UnmarshalText method to CompressionCodec by @vincentbernat in #2172
- KIP-368 : Allow SASL Connections to Periodically Re-Authenticate by @k-wall in #2197
- feat: add batched CreateACLs func to ClusterAdmin by @nkostoulas in #2191
- fix: TestRecordBatchDecoding failing sporadically by @k-wall in #2154
- feat(test): add an fvt for broker deadlock by @dnwe in #2144
- fix: avoid starvation in subscriptionManager by @dnwe in #2109
- fix: remove "Is your cluster reachable?" from msg by @dnwe in #2165
- fix: remove trailing fullstop from error strings by @dnwe in #2166
- fix: return underlying sasl error message by @dnwe in #2164
- fix: potential data race on a global variable by @pior in #2171
- fix: AdminClient | CreateACLs | check for error in response, return error if needed by @omris94 in #2185
- producer: ensure that the management message (fin) is never "leaked" by @niamster in #2182
- fix: prevent RefreshBrokers leaking old brokers by @k-wall in #2203
- fix: prevent RefreshController leaking controller by @k-wall in #2204
- fix: prevent AsyncProducer retryBatch from leaking by @k-wall in #2208
- fix: prevent metrics leak when authenticate fails by @Stephan14 in #2205
- fix: prevent deadlock between subscription manager and consumer goroutines by @niamster in #2194
- fix: prevent idempotent producer epoch exhaustion by @ladislavmacoun in #2178
- fix(test): mockbroker offsetResponse vers behavior by @dnwe in #2213
- fix: cope with OffsetsLoadInProgress on Join+Sync by @dnwe in #2214
- fix: make default MaxWaitTime 500ms by @dnwe in #2227
- chore(deps): bump xdg-go/scram and klauspost/compress by @dnwe in #2170
- fix(test): skip TestReadOnlyAndAllCommittedMessages by @dnwe in #2161
- fix(test): remove t.Parallel() by @dnwe in #2162
- chore(ci): bump along to Go 1.17+1.18 and bump golangci-lint by @dnwe in #2183
- chore: switch to multi-arch compatible docker images by @dnwe in #2210
- Remediate a number go-routine leaks (mainly test issues) by @k-wall in #2198
- chore: retract v1.32.0 due to #2150 by @dnwe in #2199
- chore: bump functional test timeout to 12m by @dnwe in #2200
- fix(admin): make DeleteRecords err consistent by @dnwe in #2226
- @k-wall made their first contribution in #2154
- @pior made their first contribution in #2171
- @omris94 made their first contribution in #2185
- @vincentbernat made their first contribution in #2172
- @niamster made their first contribution in #2182
- @ladislavmacoun made their first contribution in #2178
- @nkostoulas made their first contribution in #2191
Full Changelog: https://github.com/IBM/sarama/compare/v1.32.0...v1.33.0
- chore: retract v1.32.0 due to #2150 by @dnwe in #2199
- Fix deadlock when closing Broker in brokerProducer by @slaunay in #2133
- chore: refresh dependencies to latest by @dnwe in #2159
- fix: rework RebalancingMultiplePartitions test by @dnwe in #2130
- fix(test): use Sarama transactional producer by @dnwe in #1939
- chore: enable t.Parallel() wherever possible by @dnwe in #2138
- chore: restrict to 1 testbinary at once by @dnwe in #2145
- chore: restrict to 1 parallel test at once by @dnwe in #2146
- Remove myself from codeowners by @bai in #2147
- chore: add retractions for known bad versions by @dnwe in #2160
Full Changelog: https://github.com/IBM/sarama/compare/v1.31.1...v1.32.0
- #2126 - @bai - Populate missing kafka versions
- #2124 - @bai - Add Kafka 3.1.0 to CI matrix, migrate to bitnami kafka image
- #2123 - @bai - Update klauspost/compress to 0.14
- #2122 - @dnwe - fix(test): make it simpler to re-use toxiproxy
- #2119 - @bai - Add Kafka 3.1.0 version number
- #2005 - @raulnegreiros - feat: add methods to pause/resume consumer's consumption
- #2051 - @seveas - Expose the TLS connection state of a broker connection
- #2117 - @wuhuizuo - feat: add method MockApiVersionsResponse.SetApiKeys
- #2110 - @dnwe - fix: ensure heartbeats only stop after cleanup
- #2113 - @mosceo - Fix typo
- feat: expose IncrementalAlterConfigs API in admin.go by @fengyinqiao in #2088
- feat: allow AsyncProducer to have MaxOpenRequests inflight produce requests per broker by @xujianhai666 in #1686
- Support request pipelining in AsyncProducer by @slaunay in #2094
- fix(test): add fluent interface for mocks where missing by @grongor in #2080
- fix(test): test for ConsumePartition with OffsetOldest by @grongor in #2081
- fix: set HWMO during creation of partitionConsumer (fix incorrect HWMO before first fetch) by @grongor in #2082
- fix: ignore non-nil but empty error strings in Describe/Alter client quotas responses by @agriffaut in #2096
- fix: skip over KIP-482 tagged fields by @dnwe in #2107
- fix: clear preferredReadReplica if broker shutdown by @dnwe in #2108
- fix(test): correct wrong offsets in mock Consumer by @grongor in #2078
- fix: correct bugs in DescribeGroupsResponse by @dnwe in #2111
- chore: bump runtime and test dependencies by @dnwe in #2100
- docs: refresh README.md for Kafka 3.0.0 by @dnwe in #2099
- Fix typo by @mosceo in #2084
- @grongor made their first contribution in #2080
- @fengyinqiao made their first contribution in #2088
- @xujianhai666 made their first contribution in #1686
- @mosceo made their first contribution in #2084
Full Changelog: https://github.com/IBM/sarama/compare/v1.30.1...v1.31.0
- feat(zstd): pass level param through to compress/zstd encoder by @lizthegrey in #2045
- fix: set min-go-version to 1.16 by @troyanov in #2048
- logger: fix debug logs' formatting directives by @utrack in #2054
- fix: stuck on the batch with zero records length by @pachmu in #2057
- fix: only update preferredReadReplica if valid by @dnwe in #2076
- chore: add release notes configuration by @dnwe in #2046
- chore: confluent platform version bump by @lizthegrey in #2070
- ℹ️ from Sarama 1.30.x onward the minimum version of Go toolchain required is 1.16.x
- @troyanov made their first contribution in #2048
- @lizthegrey made their first contribution in #2045
- @utrack made their first contribution in #2054
- @pachmu made their first contribution in #2057
Full Changelog: https://github.com/IBM/sarama/compare/v1.30.0...v1.30.1
regression: enabling rackawareness causes severe throughput drops (#2071) — fixed in v1.30.1 via #2076
ℹ️ Note: from Sarama 1.30.0 the minimum version of Go toolchain required is 1.16.x
- #1983 - @zifengyu - allow configure AllowAutoTopicCreation argument in metadata refresh
- #2000 - @matzew - Using xdg-go module for SCRAM
- #2003 - @gdm85 - feat: add counter metrics for consumer group join/sync and their failures
- #1992 - @zhaomoran - feat: support SaslHandshakeRequest v0 for SCRAM
- #2006 - @faillefer - Add support for DeleteOffsets operation
- #1909 - @agriffaut - KIP-546 Client quota APIs
- #1633 - @aldelucca1 - feat: allow balance strategies to provide initial state
- #1275 - @dnwe - log: add a DebugLogger that proxies to Logger
- #2018 - @dnwe - feat: use DebugLogger reference for goldenpath log
- #2019 - @dnwe - feat: add logging & a metric for producer throttle
- #2023 - @dnwe - feat: add Controller() to ClusterAdmin interface
- #2025 - @dnwe - feat: support ApiVersionsRequest V3 protocol
- #2028 - @dnwe - feat: send ApiVersionsRequest on broker open
- #2034 - @bai - Add support for kafka 3.0.0
- #1990 - @doxsch - fix: correctly pass ValidateOnly through to CreatePartitionsRequest
- #1988 - @LubergAlexander - fix: correct WithCustomFallbackPartitioner implementation
- #2001 - @HurSungYun - docs: inform AsyncProducer Close pitfalls
- #1973 - @qiangmzsx - fix: metrics still taking up too much memory when metrics.UseNilMetrics=true
- #2007 - @bai - Add support for Go 1.17
- #2009 - @dnwe - fix: enable nilerr linter and fix iferr checks
- #2010 - @dnwe - chore: enable exportloopref and misspell linters
- #2013 - @faillefer - fix(test): disable encoded response/request check when map contains multiple elements
- #2015 - @bai - Change default branch to main
- #1718 - @crivera-fastly - fix: correct the error handling in client.InitProducerID()
- #1984 - @null-sleep - fix(test): bump confluentPlatformVersion from 6.1.1 to 6.2.0
- #2016 - @dnwe - chore: replace deprecated Go calls
- #2017 - @dnwe - chore: delete legacy vagrant script
- #2020 - @dnwe - fix(test): remove testLogger from TrackLeader test
- #2024 - @dnwe - chore: bump toxiproxy container to v2.1.5
- #2033 - @bai - Update dependencies
- #2031 - @gdm85 - docs: do not mention buffered messages in sync producer Close method
- #2035 - @dnwe - chore: populate the missing kafka versions
- #2038 - @dnwe - feat: add a fuzzing workflow to github actions
- @zifengyu made their first contribution in #1983
- @doxsch made their first contribution in #1990
- @LubergAlexander made their first contribution in #1988
- @HurSungYun made their first contribution in #2001
- @gdm85 made their first contribution in #2003
- @qiangmzsx made their first contribution in #1973
- @zhaomoran made their first contribution in #1992
- @faillefer made their first contribution in #2006
- @crivera-fastly made their first contribution in #1718
- @null-sleep made their first contribution in #1984
Full Changelog: https://github.com/IBM/sarama/compare/v1.29.1...v1.30.0
- #1966 - @ajanikow - KIP-339: Add Incremental Config updates API
- #1964 - @ajanikow - Add DelegationToken ResourceType
- #1962 - @hanxiaolin - fix(consumer): call interceptors when MaxProcessingTime expire
- #1971 - @KerryJava - fix kafka-producer-performance throughput panic
- #1968 - @dnwe - chore: bump golang.org/x versions
- #1956 - @joewreschnig - Allow checking the entire
ProducerMessagein the mock producers - #1963 - @dnwe - fix: ensure backoff timer is re-used
- #1949 - @dnwe - fix: explicitly use uint64 for payload length
- #1917 - @arkady-emelyanov - KIP-554: Add Broker-side SCRAM Config API
- #1869 - @wyndhblb - zstd: encode+decode performance improvements
- #1541 - @izolight - add String, (Un)MarshalText for acl types.
- #1921 - @bai - Add support for Kafka 2.8.0
- #1936 - @dnwe - fix(consumer): follow preferred broker
- #1933 - @ozzieba - Use gofork for encoding/asn1 to fix ASN errors during Kerberos authentication
- #1929 - @celrenheit - Handle isolation level in Offset(Request|Response) and require stable offset in FetchOffset(Request|Response)
- #1926 - @dnwe - fix: correct initial CodeQL findings
- #1925 - @bai - Test out CodeQL
- #1923 - @bestgopher - Remove redundant switch-case, fix doc typos
- #1922 - @bai - Update go dependencies
- #1898 - @mmaslankaprv - Parsing only known control batches value
- #1887 - @withshubh - Fix: issues affecting code quality
Note that with this release we change RoundRobinBalancer strategy to match Java client behavior. See #1788 for details.
- #1870 - @kvch - Update Kerberos library to latest major
- #1876 - @bai - Update docs, reference pkg.go.dev
- #1846 - @wclaeys - Do not ignore Consumer.Offsets.AutoCommit.Enable config on Close
- #1747 - @XSAM - fix: mock sync producer does not handle the offset while sending messages
- #1863 - @bai - Add support for Kafka 2.7.0 + update lz4 and klauspost/compress dependencies
- #1788 - @kzinglzy - feat[balance_strategy]: announcing a new round robin balance strategy
- #1862 - @bai - Fix CI setenv permissions issues
- #1832 - @ilyakaznacheev - Update Godoc link to pkg.go.dev
- #1822 - @danp - KIP-392: Allow consumers to fetch from closest replica
#1750 - @krantideep95 Adds missing mock responses for mocking consumer group
#1817 - reverts #1785 - Add private method to Client interface to prevent implementation
#1775 - @d1egoaz - Adds a Producer Interceptor example #1781 - @justin-chen - Refresh brokers given list of seed brokers #1784 - @justin-chen - Add randomize seed broker method #1790 - @d1egoaz - remove example binary #1798 - @bai - Test against Go 1.15 #1785 - @justin-chen - Add private method to Client interface to prevent implementation #1802 - @uvw - Support Go 1.13 error unwrapping
#1791 - @stanislavkozlovski - bump default version to 1.0.0
#1466 - @rubenvp8510 - Expose kerberos fast negotiation configuration #1695 - @KJTsanaktsidis - Use docker-compose to run the functional tests #1699 - @wclaeys - Consumer group support for manually comitting offsets #1714 - @bai - Bump Go to version 1.14.3, golangci-lint to 1.27.0 #1726 - @d1egoaz - Include zstd on the functional tests #1730 - @d1egoaz - KIP-42 Add producer and consumer interceptors #1738 - @varun06 - fixed variable names that are named same as some std lib package names #1741 - @varun06 - updated zstd dependency to latest v1.10.10 #1743 - @varun06 - Fixed declaration dependencies and other lint issues in code base #1763 - @alrs - remove deprecated tls options from test #1769 - @bai - Add support for Kafka 2.6.0
#1697 - @kvch - Use gofork for encoding/asn1 to fix ASN errors during Kerberos authentication #1744 - @alrs - Fix isBalanced Function Signature
- #1701 - @d1egoaz - Set server name only for the current broker
- #1694 - @dnwe - testfix: set KAFKA_HEAP_OPTS for zk and kafka
- #1692 - @d1egoaz - Set tls ServerName to fix issue: either ServerName or InsecureSkipVerify must be specified in the tls.Config
This release has been marked as not ready for production and may be unstable, please use v1.26.4.
- #1560 - @iyacontrol - add sync pool for gzip 1-9
- #1605 - @dnwe - feat: protocol support for V11 fetch w/ rackID
- #1617 - @sladkoff / @dwi-di / @random-dwi - Add support for alter/list partition reassignements APIs
- #1632 - @bai - Add support for Go 1.14
- #1640 - @random-dwi - Feature/fix list partition reassignments
- #1646 - @mimaison - Add DescribeLogDirs to admin client
- #1667 - @bai - Add support for kafka 2.5.0
- #1594 - @sladkoff - Sets ConfigEntry.Default flag in addition to the ConfigEntry.Source for Kafka versions > V1_1_0_0
- #1601 - @alrs - fix: remove use of testing.T.FailNow() inside goroutine
- #1602 - @d1egoaz - adds a note about consumer groups Consume method
- #1607 - @darklore - Fix memory leak when Broker.Open and Broker.Close called repeatedly
- #1613 - @wblakecaldwell - Updated "retrying" log message when BackoffFunc implemented
- #1614 - @alrs - produce_response.go: Remove Unused Functions
- #1619 - @alrs - tools/kafka-producer-performance: prune unused flag variables
- #1639 - @agriffaut - Handle errors with no message but error code
- #1643 - @kzinglzy - fix
config.net.keepalive - #1644 - @KJTsanaktsidis - Fix brokers continually allocating new Session IDs
- #1645 - @Stephan14 - Remove broker(s) which no longer exist in metadata
- #1650 - @lavoiesl - Return the response error in heartbeatLoop
- #1661 - @KJTsanaktsidis - Fix "broker received out of order sequence" when brokers die
- #1666 - @KevinJCross - Bugfix: Allow TLS connections to work over socks proxy.
Improvements:
- Add requests-in-flight metric (1539)
- Fix misleading example for cluster admin (1595)
- Replace Travis with GitHub Actions, linters housekeeping (1573)
- Allow BalanceStrategy to provide custom assignment data (1592)
Bug Fixes:
- Adds back Consumer.Offsets.CommitInterval to fix API (1590)
- Fix error message s/CommitInterval/AutoCommit.Interval (1589)
New Features:
Improvements:
- Add SASL AuthIdentity to SASL frames (authzid) (1585).
Bug Fixes:
- Sending messages with ZStd compression enabled fails in multiple ways (1252).
- Use the broker for any admin on BrokerConfig (1571).
- Set DescribeConfigRequest Version field (1576).
- ConsumerGroup flooding logs with client/metadata update req (1578).
- MetadataRequest version in DescribeCluster (1580).
- Fix deadlock in consumer group handleError (1581)
- Fill in the Fetch{Request,Response} protocol (1582).
- Retry topic request on ControllerNotAvailable (1586).
New Features:
Improvements:
- Allow the Consumer to disable auto-commit offsets (1164).
- Produce records with consistent timestamps (1455).
Bug Fixes:
- Fix incorrect SetTopicMetadata name mentions (1534).
- Fix client.tryRefreshMetadata Println (1535).
- Fix panic on calling updateMetadata on closed client (1531).
- Fix possible faulty metrics in TestFuncProducing (1545).
New Features:
- Add DescribeLogDirs Request/Response pair (1520).
Bug Fixes:
- Fix ClusterAdmin returning invalid controller ID on DescribeCluster (1518).
- Fix issue with consumergroup not rebalancing when new partition is added (1525).
- Ensure consistent use of read/write deadlines (1529).
New Features:
- Add sticky partition assignor (1416).
- Switch from cgo zstd package to pure Go implementation (1477).
Improvements:
- Allow creating ClusterAdmin from client (1415).
- Set KafkaVersion in ListAcls method (1452).
- Set request version in CreateACL ClusterAdmin method (1458).
- Set request version in DeleteACL ClusterAdmin method (1461).
- Handle missed error codes on TopicMetaDataRequest and GroupCoordinatorRequest (1464).
- Remove direct usage of gofork (1465).
- Add support for Go 1.13 (1478).
- Improve behavior of NewMockListAclsResponse (1481).
Bug Fixes:
- Fix race condition in consumergroup example (1434).
- Fix brokerProducer goroutine leak (1442).
- Use released version of lz4 library (1469).
- Set correct version in MockDeleteTopicsResponse (1484).
- Fix CLI help message typo (1494).
Known Issues:
- Please don't use Zstd, as it doesn't work right now. See #1252
Bug Fixes:
New Features:
- Add support for Kafka 2.3.0 (1418).
- Add support for ListConsumerGroupOffsets v2 (1374).
- Add support for DeleteConsumerGroup (1417).
- Add support for SASLVersion configuration (1410).
- Add kerberos support (1366).
Improvements:
- Improve sasl_scram_client example (1406).
- Fix shutdown and race-condition in consumer-group example (1404).
- Add support for error codes 77—81 (1397).
- Pool internal objects allocated per message (1385).
- Reduce packet decoder allocations (1373).
- Support timeout when fetching metadata (1359).
Bug Fixes:
- Fix fetch size integer overflow (1376).
- Handle and log throttled FetchResponses (1383).
- Refactor misspelled word Resouce to Resource (1368).
Improvements:
Bug Fixes:
- Fix V5 MetadataRequest nullable topics array (1353).
- Use a different SCRAM client for each broker connection (1349).
- Fix AllowAutoTopicCreation for MetadataRequest greater than v3 (1344).
New Features:
- Add Offline Replicas Operation to Client (1318).
- Allow using proxy when connecting to broker (1326).
- Implement ReadCommitted (1307).
- Add support for Kafka 2.2.0 (1331).
- Add SASL SCRAM-SHA-512 and SCRAM-SHA-256 mechanismes (1331).
Improvements:
- Unregister all broker metrics on broker stop (1232).
- Add SCRAM authentication example (1303).
- Add consumergroup examples (1304).
- Expose consumer batch size metric (1296).
- Add TLS options to console producer and consumer (1300).
- Reduce client close bookkeeping (1297).
- Satisfy error interface in create responses (1154).
- Please lint gods (1346).
Bug Fixes:
- Fix multi consumer group instance crash (1338).
- Update lz4 to latest version (1347).
- Retry ErrNotCoordinatorForConsumer in new consumergroup session (1231).
- Fix cleanup error handler (1332).
- Fix rate condition in PartitionConsumer (1156).
New Features:
- Add CreateAclRequest, DescribeAclRequest, DeleteAclRequest (1236).
- Add DescribeTopic, DescribeConsumerGroup, ListConsumerGroups, ListConsumerGroupOffsets admin requests (1178).
- Implement SASL/OAUTHBEARER (1240).
Improvements:
- Add Go mod support (1282).
- Add error codes 73—76 (1239).
- Add retry backoff function (1160).
- Maintain metadata in the producer even when retries are disabled (1189).
- Include ReplicaAssignment in ListTopics (1274).
- Add producer performance tool (1222).
- Add support LogAppend timestamps (1258).
Bug Fixes:
- Fix potential deadlock when a heartbeat request fails (1286).
- Fix consuming compacted topic (1227).
- Set correct Kafka version for DescribeConfigsRequest v1 (1277).
- Update kafka test version (1273).
New Features:
- Add optional replica id in offset request (1100).
Improvements:
- Implement DescribeConfigs Request + Response v1 & v2 (1230).
- Reuse compression objects (1185).
- Switch from png to svg for GoDoc link in README (1243).
- Fix typo in deprecation notice for FetchResponseBlock.Records (1242).
- Fix typos in consumer metadata response file (1244).
Bug Fixes:
- Revert to individual msg retries for non-idempotent (1203).
- Respect MaxMessageBytes limit for uncompressed messages (1141).
New Features:
- Add support for zstd compression (#1170).
- Add support for Idempotent Producer (#1152).
- Add support support for Kafka 2.1.0 (#1229).
- Add support support for OffsetCommit request/response pairs versions v1 to v5 (#1201).
- Add support support for OffsetFetch request/response pair up to version v5 (#1198).
Improvements:
- Export broker's Rack setting (#1173).
- Always use latest patch version of Go on CI (#1202).
- Add error codes 61 to 72 (#1195).
Bug Fixes:
- Fix build without cgo (#1182).
- Fix go vet suggestion in consumer group file (#1209).
- Fix typos in code and comments (#1228).
New Features:
- Implement a higher-level consumer group (#1099).
Improvements:
- Add support for Go 1.11 (#1176).
Bug Fixes:
- Fix encoding of
MetadataResponsewith version 2 and higher (#1174). - Fix race condition in mock async producer (#1174).
New Features:
- Make
Partitioner.RequiresConsistencyvary per-message (#1112). - Add customizable partitioner (#1118).
- Add
ClusterAdminsupport forCreateTopic,DeleteTopic,CreatePartitions,DeleteRecords,DescribeConfig,AlterConfig,CreateACL,ListAcls,DeleteACL(#1055).
Improvements:
- Add support for Kafka 2.0.0 (#1149).
- Allow setting
LocalAddrwhen dialing an address to support multi-homed hosts (#1123). - Simpler offset management (#1127).
Bug Fixes:
- Fix mutation of
ProducerMessage.MetaDatawhen producing to Kafka (#1110). - Fix consumer block when response did not contain all the expected topic/partition blocks (#1086).
- Fix consumer block when response contains only constrol messages (#1115).
- Add timeout config for ClusterAdmin requests (#1142).
- Add version check when producing message with headers (#1117).
- Fix
MetadataRequestfor empty list of topics (#1132). - Fix producer topic metadata on-demand fetch when topic error happens in metadata response (#1125).
New Features:
- Add support for gzip compression levels (#1044).
- Add support for Metadata request/response pairs versions v1 to v5 (#1047, #1069).
- Add versioning to JoinGroup request/response pairs (#1098)
- Add support for CreatePartitions, DeleteGroups, DeleteRecords request/response pairs (#1065, #1096, #1027).
- Add
Controller()method to Client interface (#1063).
Improvements:
- ConsumerMetadataReq/Resp has been migrated to FindCoordinatorReq/Resp (#1010).
- Expose missing protocol parts:
msgSetandrecordBatch(#1049). - Add support for v1 DeleteTopics Request (#1052).
- Add support for Go 1.10 (#1064).
- Claim support for Kafka 1.1.0 (#1073).
Bug Fixes:
- Fix FindCoordinatorResponse.encode to allow nil Coordinator (#1050, #1051).
- Clear all metadata when we have the latest topic info (#1033).
- Make
PartitionConsumer.Closeidempotent (#1092).
New Features:
- Add support for the Create/Delete Topics request/response pairs (#1007, #1008).
- Add support for the Describe/Create/Delete ACL request/response pairs (#1009).
- Add support for the five transaction-related request/response pairs (#1016).
Improvements:
- Permit setting version on mock producer responses (#999).
- Add
NewMockBrokerListenerhelper for testing TLS connections (#1019). - Changed the default value for
Consumer.Fetch.Defaultfrom 32KiB to 1MiB which results in much higher throughput in most cases (#1024). - Reuse the
time.Tickeracross fetch requests in the PartitionConsumer to reduce CPU and memory usage when processing many partitions (#1028). - Assign relative offsets to messages in the producer to save the brokers a recompression pass (#1002, #1015).
Bug Fixes:
- Fix producing uncompressed batches with the new protocol format (#1032).
- Fix consuming compacted topics with the new protocol format (#1005).
- Fix consuming topics with a mix of protocol formats (#1021).
- Fix consuming when the broker includes multiple batches in a single response (#1022).
- Fix detection of
PartialTrailingMessagewhen the partial message was truncated before the magic value indicating its version (#1030). - Fix expectation-checking in the mock of
SyncProducer.SendMessages(#1035).
New Features:
- Claim official support for Kafka 1.0, though it did already work (#984).
- Helper methods for Kafka version numbers to/from strings (#989).
- Implement CreatePartitions request/response (#985).
Improvements:
- Add error codes 45-60 (#986).
Bug Fixes:
- Fix slow consuming for certain Kafka 0.11/1.0 configurations (#982).
- Correctly determine when a FetchResponse contains the new message format (#990).
- Fix producing with multiple headers (#996).
- Fix handling of truncated record batches (#998).
- Fix leaking metrics when closing brokers (#991).
New Features:
- Add support for the new Kafka 0.11 record-batch format, including the wire protocol and the necessary behavioural changes in the producer and consumer. Transactions and idempotency are not yet supported, but producing and consuming should work with all the existing bells and whistles (batching, compression, etc) as well as the new custom headers. Thanks to Vlad Hanciuta of Arista Networks for this work. Part of (#901).
Bug Fixes:
- Fix encoding of ProduceResponse versions in test (#970).
- Return partial replicas list when we have it (#975).
New Features:
- Support for FetchRequest version 3 (#905).
- Permit setting version on mock FetchResponses (#939).
- Add a configuration option to support storing only minimal metadata for extremely large clusters (#937).
- Add
PartitionOffsetManager.ResetOffsetfor backtracking tracked offsets (#932).
Improvements:
- Provide the block-level timestamp when consuming compressed messages (#885).
Client.ReplicasandClient.InSyncReplicasnow respect the order returned by the broker, which can be meaningful (#930).- Use a
Tickerto reduce consumer timer overhead at the cost of higher variance in the actual timeout (#933).
Bug Fixes:
- Gracefully handle messages with negative timestamps (#907).
- Raise a proper error when encountering an unknown message version (#940).
New Features:
- Added support for the
ApiVersionsrequest and response pair, and Kafka version 0.10.2 (#867). Note that you still need to specify the Kafka version in the Sarama configuration for the time being. - Added a
Brokersmethod to the Client which returns the complete set of active brokers (#813). - Added an
InSyncReplicasmethod to the Client which returns the set of all in-sync broker IDs for the given partition, now that the Kafka versions for which this was misleading are no longer in our supported set (#872). - Added a
NewCustomHashPartitionermethod which allows constructing a hash partitioner with a custom hash method in case the default (FNV-1a) is not suitable (#837, #841).
Improvements:
- Recognize more Kafka error codes (#859).
Bug Fixes:
- Fix an issue where decoding a malformed FetchRequest would not return the correct error (#818).
- Respect ordering of group protocols in JoinGroupRequests. This fix is
transparent if you're using the
AddGroupProtocolorAddGroupProtocolMetadatahelpers; otherwise you will need to switch from theGroupProtocolsfield (now deprecated) to useOrderedGroupProtocols(#812). - Fix an alignment-related issue with atomics on 32-bit architectures (#859).
Important: As of Sarama 1.11 it is necessary to set the config value of
Producer.Return.Successes to true in order to use the SyncProducer. Previous
versions would silently override this value when instantiating a SyncProducer
which led to unexpected values and data races.
New Features:
- Metrics! Thanks to Sébastien Launay for all his work on this feature (#701, #746, #766).
- Add support for LZ4 compression (#786).
- Add support for ListOffsetRequest v1 and Kafka 0.10.1 (#775).
- Added a
HighWaterMarksmethod to the Consumer which aggregates theHighWaterMarkOffsetvalues of its child topic/partitions (#769).
Bug Fixes:
- Fixed producing when using timestamps, compression and Kafka 0.10 (#759).
- Added missing decoder methods to DescribeGroups response (#756).
- Fix producer shutdown when
Return.Errorsis disabled (#787). - Don't mutate configuration in SyncProducer (#790).
- Fix crash on SASL initialization failure (#795).
Bug Fixes:
- Fix the documentation for
HashPartitionerwhich was incorrect (#717). - Permit client creation even when it is limited by ACLs (#722).
- Several fixes to the consumer timer optimization code, regressions introduced in v1.10.0. Go's timers are finicky (#730, #733, #734).
- Handle consuming compressed relative offsets with Kafka 0.10 (#735).
Important: As of Sarama 1.10 it is necessary to tell Sarama the version of
Kafka you are running against (via the config.Version value) in order to use
features that may not be compatible with old Kafka versions. If you don't
specify this value it will default to 0.8.2 (the minimum supported), and trying
to use more recent features (like the offset manager) will fail with an error.
Also: The offset-manager's behaviour has been changed to match the upstream java consumer (see #705 and #713). If you use the offset-manager, please ensure that you are committing one greater than the last consumed message offset or else you may end up consuming duplicate messages.
New Features:
- Support for Kafka 0.10 (#672, #678, #681, and others).
- Support for configuring the target Kafka version (#676).
- Batch producing support in the SyncProducer (#677).
- Extend producer mock to allow setting expectations on message contents (#667).
Improvements:
- Support
nilcompressed messages for deleting in compacted topics (#634). - Pre-allocate decoding errors, greatly reducing heap usage and GC time against misbehaving brokers (#690).
- Re-use consumer expiry timers, removing one allocation per consumed message (#707).
Bug Fixes:
- Actually default the client ID to "sarama" like we say we do (#664).
- Fix a rare issue where
Client.Leadercould return the wrong error (#685). - Fix a possible tight loop in the consumer (#693).
- Match upstream's offset-tracking behaviour (#705).
- Report UnknownTopicOrPartition errors from the offset manager (#706).
- Fix possible negative partition value from the HashPartitioner (#709).
New Features:
- Add support for custom offset manager retention durations (#602).
- Publish low-level mocks to enable testing of third-party producer/consumer implementations (#570).
- Declare support for Golang 1.6 (#611).
- Support for SASL plain-text auth (#648).
Improvements:
Bug Fixes:
- Fix race condition shutting down the OffsetManager (#658).
New Features:
- Full support for Kafka 0.9:
Improvements:
- Optimize for fewer system calls when reading from the network (#584).
- Automatically retry
InvalidMessageerrors to match upstream behaviour (#589).
New Features:
- Preliminary support for Kafka 0.9 (#572). This comes with several caveats:
Improvements:
- Don't wait for request timeouts on dead brokers, greatly speeding recovery when the TCP connection is left hanging (#548).
- Refactored part of the producer. The new version provides a much more elegant solution to #449. It is also slightly more efficient, and much more precise in calculating batch sizes when compression is used (#549, #550, #551).
Bug Fixes:
- Fix race condition in consumer test mock (#553).
Bug Fixes:
- Fix panic that could occur if a user-supplied message value failed to encode (#449).
New Features:
- Implementation of a consumer offset manager using the APIs introduced in Kafka 0.8.2. The API is designed mainly for integration into a future high-level consumer, not for direct use, although it is possible to use it directly. (#461).
Improvements:
- CRC32 calculation is much faster on machines with SSE4.2 instructions, removing a major hotspot from most profiles (#255).
Bug Fixes:
- Make protocol decoding more robust against some malformed packets generated by go-fuzz (#523, #525) or found in other ways (#528).
- Fix a potential race condition panic in the consumer on shutdown (#529).
New Features:
- TLS-encrypted network connections are now supported. This feature is subject to change when Kafka releases built-in TLS support, but for now this is enough to work with TLS-terminating proxies (#154).
Improvements:
- The consumer will not block if a single partition is not drained by the user; all other partitions will continue to consume normally (#485).
- Formatting of error strings has been much improved (#495).
- Internal refactoring of the producer for code cleanliness and to enable future work (#300).
Bug Fixes:
- Fix a potential deadlock in the consumer on shutdown (#475).
Bug Fixes:
- Don't include the partitioner in the producer's "fetch partitions" circuit-breaker (#466).
- Don't retry messages until the broker is closed when abandoning a broker in the producer (#468).
- Update the import path for snappy-go, it has moved again and the API has changed slightly (#486).
Bug Fixes:
- Update the import path for snappy-go, it has moved from google code to github (#456).
Improvements:
- Optimizations when decoding snappy messages, thanks to John Potocny (#446).
Bug Fixes:
New Features:
- The consumer now implements
Topics()andPartitions()methods to enable users to dynamically choose what topics/partitions to consume without instantiating a full client (#431). - The partition-consumer now exposes the high water mark offset value returned
by the broker via the
HighWaterMarkOffset()method (#339). - Added a
kafka-console-consumertool capable of handling multiple partitions, and deprecated the now-obsoletekafka-console-partitionConsumer(#439, #442).
Improvements:
- The producer's logging during retry scenarios is more consistent, more useful, and slightly less verbose (#429).
- The client now shuffles its initial list of seed brokers in order to prevent thundering herd on the first broker in the list (#441).
Bug Fixes:
- The producer now correctly manages its state if retries occur when it is shutting down, fixing several instances of confusing behaviour and at least one potential deadlock (#419).
- The consumer now handles messages for different partitions asynchronously, making it much more resilient to specific user code ordering (#325).
New Features:
- The client now tracks consumer group coordinators using
ConsumerMetadataRequests similar to how it tracks partition leadership using
regular MetadataRequests (#411).
This adds two methods to the client API:
Coordinator(consumerGroup string) (*Broker, error)RefreshCoordinator(consumerGroup string) error
Improvements:
- ConsumerMetadataResponses now automatically create a Broker object out of the ID/address/port combination for the Coordinator; accessing the fields individually has been deprecated (#413).
- Much improved handling of
OffsetOutOfRangeerrors in the consumer. Consumers will fail to start if the provided offset is out of range (#418) and they will automatically shut down if the offset falls out of range (#424). - Small performance improvement in encoding and decoding protocol messages (#427).
Bug Fixes:
- Fix a rare race condition in the client's background metadata refresher if it happens to be activated while the client is being closed (#422).
Improvements:
- The producer's behaviour when
Flush.Frequencyis set is now more intuitive (#389). - The producer is now somewhat more memory-efficient during and after retrying messages due to an improved queue implementation (#396).
- The consumer produces much more useful logging output when leadership changes (#385).
- The client's
GetOffsetmethod will now automatically refresh metadata and retry once in the event of stale information or similar (#394). - Broker connections now have support for using TCP keepalives (#407).
Bug Fixes:
- The OffsetCommitRequest message now correctly implements all three possible API versions (#390, #400).
Improvements:
- Wrap the producer's partitioner call in a circuit-breaker so that repeatedly broken topics don't choke throughput (#373).
Bug Fixes:
- Fix the producer's internal reference counting in certain unusual scenarios (#367).
- Fix the consumer's internal reference counting in certain unusual scenarios (#369).
- Fix a condition where the producer's internal control messages could have gotten stuck (#368).
- Fix an issue where invalid partition lists would be cached when asking for metadata for a non-existant topic (#372).
Version 1.0.0 is the first tagged version, and is almost a complete rewrite. The primary differences with previous untagged versions are:
- The producer has been rewritten; there is now a
SyncProducerwith a blocking API, and anAsyncProducerthat is non-blocking. - The consumer has been rewritten to only open one connection per broker instead of one connection per partition.
- The main types of Sarama are now interfaces to make depedency injection easy; mock implementations for
Consumer,SyncProducerandAsyncProducerare provided in thegit.luolix.top/IBM/sarama/mockspackage. - For most uses cases, it is no longer necessary to open a
Client; this will be done for you. - All the configuration values have been unified in the
Configstruct. - Much improved test suite.