Release latest

Release latest

Release 0.8.0.Final

Release 0.8.0.Final

Release 0.7.0.Final

• Debezium: 2.7.2.Final
• Apache Iceberg: 1.6.1
• Quarkus: 3.13.3

Breaking Changes

Equality deletes:

• Now Equality delete files are containing key fields only and they are written with record projection. Now delete files only includes key files, previously delete files were including full delete record.

Notable changes

• Various code improvement added. Now code base is more closely aligned with upstream iceberg-kafka-connect
• Now size of the files generated is defined write.target-file-size-bytes config, previously Long.MAX_VALUE was used
• Iceberg upgraded from 1.6.0 to 1.6.1
• Debezium upgraded from 2.7.0.Final to 2.7.2.Final
• Quarkus upgraded from 3.13.0 to 3.13.3

What's Changed

• Bump version.quarkus from 3.13.0 to 3.13.2 by @dependabot in #400
• Bump io.smallrye:jandex-maven-plugin from 3.2.1 to 3.2.2 by @dependabot in #399
• Bump version.debezium from 2.7.0.Final to 2.7.1.Final by @dependabot in #398
• Bump version.testcontainers from 1.20.0 to 1.20.1 by @dependabot in #396
• Add code improvements by @ismailsimsek in #409
• Improve code, Improve class and method namings by @ismailsimsek in #410
• Improve deduplication code by @ismailsimsek in #411
• Use write.target-file-size-bytes config, previously Long.MAX_VALUE was used by @ismailsimsek in #412
• Fix, Create GenericAppenderFactory respecting to identifier fields. And create it with table properties by @ismailsimsek in #413
• Explicitly set operationId as UUID when creating OutputFileFactory by @ismailsimsek in #414
• Use Key fields and Record Projection for deletes by @ismailsimsek in #408
• Rename variable equalityFieldIds to identifierFieldIds by @ismailsimsek in #415
• Bump org.awaitility:awaitility from 4.2.1 to 4.2.2 by @dependabot in #406
• Bump org.postgresql:postgresql from 42.7.3 to 42.7.4 by @dependabot in #407
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.3.1 to 3.5.0 by @dependabot in #405
• In case of exception, call writer.abort() to cleanup by @ismailsimsek in #417
• Upgrade debezium and Quarkus to latest minor version by @ismailsimsek in #418

Full Changelog: 0.6.0.Final...0.7.0.Final

Release 0.6.0.Final

• Debezium: 2.7.0.Final
• Apache Iceberg: 1.6.0
• Quarkus: 3.13.0

Notable changes

• new config added to control casing of iceberg table names. To change tables names uppercase or lowercase folloing configuration properties could be used, see documentation. debezium.sink.iceberg.destination-uppercase-table-names=true or debezium.sink.iceberg.destination-lowercase-table-names=true
• Quarkus upgraded from 3.12.x to 3.13.x, See Quarkus release notes here

What's Changed

• Add option to change case of destination/iceberg table names by @ismailsimsek in #389
• Bump version.quarkus from 3.12.3 to 3.13.0 by @dependabot in #392
• Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.2.5 to 3.3.1 by @dependabot in #395
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.3.0 to 3.3.1 by @dependabot in #394
• Bump io.smallrye:jandex-maven-plugin from 3.2.0 to 3.2.1 by @dependabot in #393

Full Changelog: 0.5.0.Final...0.6.0.Final

Release 0.5.0.Final

• Debezium: 2.7.0.Final
• Apache Iceberg: 1.6.0
• Quarkus: 3.12.3

What's Changed

• Bump io.minio:minio from 8.5.7 to 8.5.11 by @dependabot in #372
• Bump org.codehaus.groovy:groovy-bom from 3.0.21 to 3.0.22 by @dependabot in #374
• Bump com.squareup.okio:okio from 3.4.0 to 3.9.0 by @dependabot in #376
• Upgrade quarkus from 3.10.0 to 3.10.2 by @ismailsimsek in #382
• Bump version.testcontainers from 1.19.8 to 1.20.0 by @dependabot in #387
• Bump org.apache.maven.plugins:maven-enforcer-plugin from 3.4.1 to 3.5.0 by @dependabot in #385

Full Changelog: 0.5.0.Beta...0.5.0.Final

Release 0.5.0.Beta

• Debezium: 2.7.0.Final
• Apache Iceberg: 1.5.2
• Quarkus: 3.10.0

Breaking Changes

Debezium version upgraded from 2.5.x to 2.7.x:

• For Debezium 2.6 Breaking Changes please see release notes
• For Debezium 2.7 Breaking Changes please see release notes

IcebergSchemaHistory wrong table location corrected:

IcebergSchemaHistory class was using wrong table namespace {catalog-name}.{table-name} now this is corrected to use {table-namespace}.{table-name}
If you were using different catalog-name please follow following steps for migration

1. stop debezium server
2. copy the IcebergSchemaHistory table to new location: {table-namespace}.{table-name}
3. start debezium server

Configuration change:

Configuration option debezium.sink.iceberg.upsert-op-column, defaultValue = __op is removed.

Notable changes

• Quarkus upgraded from 3.6.x to 3.10.x, See Quarkus release notes here
• Java release target upgraded from 11 to 17
• Full support for debezium nested events added. Now its possible to run consumer without event flattening.
• Debezium event processing improved, unnecessary schema de-serialization is avoided.

What's Changed

• Upgrade java to 17 by @ismailsimsek in #318
• Upgrade debezium to 2.7.0.Alpha1 by @ismailsimsek in #317
• Bump version.quarkus from 3.6.9 to 3.10.0 by @dependabot in #313
• Simplify example config, removed hadoop catalog example by @ismailsimsek in #321
• Minor, Improve naming of state tables used in tests by @ismailsimsek in #322
• Add mongodb flattening settings to example config by @ismailsimsek in #324
• Fix reading config values in debezium storage classes by @ismailsimsek in #327
• Minor improvements to icebergevents consumer, IcebergEventsChangeCons… by @ismailsimsek in #328
• Update codeql-analysis.yml to use java 17 by @ismailsimsek in #329
• Upgrade spark to 4.0.x version and enable Mysql tests by @ismailsimsek in #331
• Remove DynamicBatchSizeWait class by @ismailsimsek in #332
• Enforce maven version is greater than 3.8.6 by @ismailsimsek in #334
• Don't partition tables on append mode. by @ismailsimsek in #340
• Bump io.smallrye:jandex-maven-plugin from 3.1.7 to 3.2.0 by @dependabot in #338
• Bump org.postgresql:postgresql from 42.7.2 to 42.7.3 by @dependabot in #337
• Bump version.testcontainers from 1.19.7 to 1.19.8 by @dependabot in #336
• Improve IcebergChangeEvent class, Optimize event and schema deserialization by @ismailsimsek in #343
• Fail deduplication if event key is null by @ismailsimsek in #344
• Improve naming, rename JsonSchema to ChangeEventSchema by @ismailsimsek in #345
• Improve code and schema conversion tests by @ismailsimsek in #346
• Add more tests on schema conversion by @ismailsimsek in #348
• Improve conversion of debezium schema to iceberg schema by @ismailsimsek in #347
• Revert "Bump io.smallrye:jandex-maven-plugin from 3.1.7 to 3.2.0 (#338)" by @ismailsimsek in #349
• Improve setting identifier fields by @ismailsimsek in #350
• Allow nested fields to be set as identifier by @ismailsimsek in #351
• Lower minimum maven version enforcement to 3.6.3 by @ismailsimsek in #353
• Optimize test logs, decrease verbose logs by @ismailsimsek in #354
• Check if configuration contains event flattening by @ismailsimsek in #352
• Test Field Addition is working, automatic schema expansion by @ismailsimsek in #355
• Test consumer without event flattening, Add flag to control identifier-field creation by @ismailsimsek in #356
• Document new features and improve documentation by @ismailsimsek in #357
• Remove stage repo used for spark 4.0.0-preview1 dependency by @ismailsimsek in #358
• Update README.md by @ismailsimsek in #359
• Update dependabot to weekly interval by @ismailsimsek in #360
• Bump docker/build-push-action from 5 to 6 by @dependabot in #361
• Improve handling delete events and logging message by @ismailsimsek in #366
• Improve documentation, schema change behaviour by @ismailsimsek in #367
• Bump com.squareup.okhttp3:okhttp from 4.11.0 to 4.12.0 by @dependabot in #364
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.2.5 to 3.3.0 by @dependabot in #363
• Bump version.debezium from 2.7.0.Alpha1 to 2.7.0.Final by @dependabot in #371

Full Changelog: 0.4.0.Final...0.5.0.Beta

Release 0.4.1.Final

This release includes fix to debezium storage classes {IcebergSchemaHistory, IcebergOffsetBackingStore}. Now both classes can be used by icebergevents consumer too.

• Debezium: 2.7.0.Alpha1
• Apache Iceberg: 1.5.2
• Quarkus: 3.10.0

Breaking Changes

IcebergSchemaHistory class was using wrong table name {catalog-name}.{table-name} now this is corrected to {table-namespace}.{table-name}
If you were using different catalog-name previously please follow following steps for migration

1. stop debezium server
2. move the table to new location: {table-namespace}.{table-name}
3. start the debezium server

What's Changed

• Simplify example config, removed hadoop catalog example by @ismailsimsek in #321
• Fix reading config values in debezium storage classes by @ismailsimsek in #327
• Minor improvements to icebergevents consumer, IcebergEventsChangeCons… by @ismailsimsek in #328

Release 0.4.0.Final

Release 0.4.0.Final

What's Changed

• Fix multiple log4j logger bindings by @ismailsimsek in #289
• Fix __source_ts_ms values with 0 conversion and enable mongodb test by @ismailsimsek in #292
• Remove maven stage repository by @ismailsimsek in #293
• Test upsert with tables without PK by @ismailsimsek in #295
• Bump softprops/action-gh-release from 1 to 2 by @dependabot in #301
• Add example setup using docker-compose by @ismailsimsek in #302
• Add iceberg-hive-metastore jar to distribution by @ismailsimsek in #308
• Update docker-compose example by @ismailsimsek in #309
• Bump io.smallrye:jandex-maven-plugin from 3.1.6 to 3.1.7 by @dependabot in #300
• Bump version.testcontainers from 1.19.6 to 1.19.7 by @dependabot in #299
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.2.2 to 3.2.5 by @dependabot in #297
• Bump org.awaitility:awaitility from 4.2.0 to 4.2.1 by @dependabot in #296
• Bump dev-drprasad/delete-tag-and-release from 1.0 to 1.1 by @dependabot in #310
• Bump org.apache.maven.plugins:maven-assembly-plugin from 3.6.0 to 3.7.1 by @dependabot in #311
• Update debezium version to 2.5.4.Final by @ismailsimsek in #315
• Upgrade iceberg to 1.5.2 by @ismailsimsek in #316

Full Changelog: 0.3.0.Final...0.4.0.Final

Release 0.4.0.Beta

This release includes dependency upgrades and improvements with dependency versions.

Breaking Changes

Removed iceberg-spark-runtime-* dependency from the release in #114

Debezium version upgraded from 2.2.1 to 2.5.2
For Debezium 2.2 Breaking Changes please see release notes
For Debezium 2.3 Breaking Changes please see release notes
For Debezium 2.4 Breaking Changes please see release notes
For Debezium 2.5 Breaking Changes please see release notes

Iceberg upgraded from 1.3.x to 1.5.0
For Iceberg changes please see https://iceberg.apache.org/releases/

Quarkus upgraded from 3.3.2 to 3.6.9
For Quarkus changes please see https://quarkus.io/blog/tag/release/

What's Changed

• Improve exception messages and fix committing to IcebergSchemaHistory by @ismailsimsek in #234
• Upgrade iceberg to 1.4.0 by @ismailsimsek in #237
• Correct schema history parameter in application.properties.example file by @ismailsimsek in #238
• Upgrade debezium version to 2.4.x by @ismailsimsek in #235
• Bump org.apache.maven.plugins:maven-assembly-plugin from 3.4.2 to 3.6.0 by @dependabot in #225
• Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.0.0-M7 to 3.1.2 by @dependabot in #245
• Bump docker/build-push-action from 4 to 5 by @dependabot in #242
• Bump docker/login-action from 2 to 3 by @dependabot in #241
• Bump actions/checkout from 2 to 4 by @dependabot in #240
• Bump com.squareup.okhttp3:okhttp from 4.10.0 to 4.11.0 by @dependabot in #243
• Bump version.hadoop from 3.3.2 to 3.3.6 by @dependabot in #244
• Upgrade iceberg to 1.4.1 by @ismailsimsek in #247
• Bump actions/setup-java from 3 to 4 by @dependabot in #253
• Bump version.testcontainers from 1.19.0 to 1.19.1 by @dependabot in #250
• Bump com.google.cloud:google-cloud-nio from 0.124.19 to 0.127.7 by @dependabot in #254
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.0.0-M7 to 3.2.2 by @dependabot in #255
• Improve documentation by @ismailsimsek in #258
• Upgrade iceberg dependency 1.4.1 to 1.4.3 by @ismailsimsek in #260
• Upgrade quarkus 3.3.2 to 3.5.3 by @ismailsimsek in #261
• Upgrade debezium to 2.5.0.Final by @ismailsimsek in #262
• documentation typo fix by @robinsinghstudios in #272
• Bump org.postgresql:postgresql from 42.6.0 to 42.7.2 in /debezium-server-iceberg-sink by @dependabot in #270
• Bump github/codeql-action from 2 to 3 by @dependabot in #268
• Bump com.fasterxml.jackson:jackson-bom from 2.13.5 to 2.16.1 by @dependabot in #267
• Bump version.awssdk from 2.19.4 to 2.22.9 by @dependabot in #265
• Bump io.minio:minio from 8.5.1 to 8.5.7 by @dependabot in #264
• Bump version.groovy from 3.0.17 to 3.0.20 by @dependabot in #263
• Upgrade dependencies, Upgrade iceberg to 1.5.0 and debezium to 2.5.2.Final by @ismailsimsek in #273
• Use jackson version same as iceberg 2.14.2 by @ismailsimsek in #274
• Add example config for Hive metastore catalog by @ismailsimsek in #275
• Align package dependencies to iceberg and revert iceberg back to 1.4.3 by @ismailsimsek in #276
• Revert iceberg version to 1.4.3, It is changed to 1.4.2 by mistake by @ismailsimsek in #279
• Bump version.groovy from 3.0.20 to 3.0.21 by @dependabot in #280
• Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.1.2 to 3.2.5 by @dependabot in #284
• Bump version.testcontainers from 1.19.3 to 1.19.6 by @dependabot in #281
• Upgrade quarkus from 3.5.3 to 3.6.9 by @ismailsimsek in #285
• Upgrade iceberg to 1.5.0 by @ismailsimsek in #278
• Use iceberg-spark only for tests by @racevedoo in #114
• Switch to iceberg cloud bundle dependecies by @ismailsimsek in #288

New Contributors

• @robinsinghstudios made their first contribution in #272

Full Changelog: 0.3.0.Beta...0.4.0.Beta

Release 0.3.0.Final

Release 0.3.0.Final

Breaking Changes

Iceberg upgraded from 1.3.1 to 1.4.3
For Iceberg changes please see https://iceberg.apache.org/releases/

Quarkus upgraded from 3.3.2 to 3.5.3
For Quarkus changes please see https://quarkus.io/blog/tag/release/

What's Changed

• Improve exception messages and fix committing to IcebergSchemaHistory by @ismailsimsek in #234
• Upgrade iceberg to 1.4.0 by @ismailsimsek in #237
• Correct schema history parameter in application.properties.example file by @ismailsimsek in #238
• Upgrade debezium version to 2.4.x by @ismailsimsek in #235
• Bump org.apache.maven.plugins:maven-assembly-plugin from 3.4.2 to 3.6.0 by @dependabot in #225
• Bump org.apache.maven.plugins:maven-failsafe-plugin from 3.0.0-M7 to 3.1.2 by @dependabot in #245
• Bump docker/build-push-action from 4 to 5 by @dependabot in #242
• Bump docker/login-action from 2 to 3 by @dependabot in #241
• Bump actions/checkout from 2 to 4 by @dependabot in #240
• Bump com.squareup.okhttp3:okhttp from 4.10.0 to 4.11.0 by @dependabot in #243
• Bump version.hadoop from 3.3.2 to 3.3.6 by @dependabot in #244
• Upgrade iceberg to 1.4.1 by @ismailsimsek in #247
• Bump actions/setup-java from 3 to 4 by @dependabot in #253
• Bump version.testcontainers from 1.19.0 to 1.19.1 by @dependabot in #250
• Bump com.google.cloud:google-cloud-nio from 0.124.19 to 0.127.7 by @dependabot in #254
• Bump org.apache.maven.plugins:maven-surefire-plugin from 3.0.0-M7 to 3.2.2 by @dependabot in #255
• Improve documentation by @ismailsimsek in #258
• Upgrade iceberg dependency 1.4.1 to 1.4.3 by @ismailsimsek in #260
• Upgrade quarkus 3.3.2 to 3.5.3 by @ismailsimsek in #261

New Contributors

• @robinsinghstudios made their first contribution in #272

Full Changelog: 0.3.0.Beta...0.3.0.Final