v4.41.0

For installation instructions check out the getting started guide.

Added

• Field max_records_per_request added to the aws_sqs output. (@Jeffail)

Fixed

• (Benthos) Fixed an issue where running a CLI with a custom environment would cause imported templates to be rejected. (@Jeffail)

Changed

• The -cgo suffixed docker images are no longer built and pushed along with the regular images. This decision was made due to low demand, and the unacceptable cadence with which the image base (Debian) receives security updates. It is still possible to create your own CGO builds with the command CGO_ENABLED=1 make TAGS=x_benthos_extra redpanda-connect. (@Jeffail)

The full change log can be found here.

v4.40.0

For installation instructions check out the getting started guide.

Added

• New pg_stream input supporting change data capture (CDC) from PostgreSQL. (@le-vlad)
• Field metadata_max_age added to the redpanda_migrator_offsets output. (@mihaitodor)
• Field kafka_timestamp_ms added to the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator outputs. (@mihaitodor)
• (Benthos) New Bloblang method timestamp. (@mihaitodor)
• (Benthos) New benchmark processor. (@ooesili)

Fixed

• Addresses an issue where snowflake_streaming could create more channels than configured. (@rockwotj)

Changed

• The snowflake_streaming output with schema_evolution.enabled set to true can now autocreate tables. (@rockwotj)
• Fields translate_schema_ids and schema_registry_output_resource added to the redpanda_migrator output. (@mihaitodor)
• Fields backfill_dependencies and input_resource added to the schema_registry output. (@mihaitodor)
• The schema_registry input and output and the schema_registry_encode and schema_registry_decode processors now use the github.com/twmb/franz-go/pkg/sr SchemaRegistry client. (@mihaitodor)
• Metadata field kafka_timestamp_ms added to the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator inputs now contains a unix timestamp with millisecond precision. (@mihaitodor)
• Metadata field kafka_timestamp removed from the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator inputs. (@mihaitodor)

The full change log can be found here.

v4.39.0

For installation instructions check out the getting started guide.

Added

• New timeplus input. (@ye11ow)
• New snowflake_streaming output. (@rockwotj)
• Redpanda Connect will now use an optional /etc/redpanda/connector_list.yaml config to determine which connectors are available to run. (@Jeffail)
• (Benthos) Field follow_redirects added to the http processor. (@ooesili)
• New CLI flag --secrets added. (@Jeffail)
• New CLI flag --disable-telemetry added. (@Jeffail)
• New experimental spicedb watch input. (@simon0191)
• New redpanda_common input and output. (@Jeffail)
• New redpanda input and output. (@Jeffail)
• New snowflake_streaming output. (@rockwotj)

Fixed

• The kafka, kafka_franz and redpanda_migrator outputs no longer waste CPU for large batches. (@rockwotj)

Changed

• The aws_sqs output field url now supports interpolation functions. (@rockwotj)
• (Benthos) CLI --set flags can now mutate array values indexed from the end via negative integers. E.g. --set 'foo.-1=meow' would set the last index of the array foo to the value of meow. (@Jeffail)

The full change log can be found here.

v4.39.0-rc1

For installation instructions check out the getting started guide.

NOTE: This is a release candidate, you can download a binary from this page.

Added

• New timeplus input. (@ye11ow)
• New snowflake_streaming output. (@rockwotj)
• Redpanda Connect will now use an optional /etc/redpanda/connector_list.yaml config to determine which connectors are available to run. (@Jeffail)
• (Benthos) Field follow_redirects added to the http processor. (@ooesili)
• New CLI flag --secrets added. (@Jeffail)
• New CLI flag --disable-telemetry added. (@Jeffail)

Changed

• The aws_sqs output field url now supports interpolation functions. (@rockwotj)
• (Benthos) CLI --set flags can now mutate array values indexed from the end via negative integers. E.g. --set 'foo.-1=meow' would set the last index of the array foo to the value of meow. (@Jeffail)

The full change log can be found here.

v4.38.0

For installation instructions check out the getting started guide.

Added

• Anonymous telemetry data is now sent by Connect instances after running for >5 mins. Details about which data is sent, when it is sent, and how to disable it can be found in the telemetry README. (@Jeffail)
• Field checksum_algorithm added to the aws_s3 output. (@dom-lee-naimuri)
• Field nkey added to nats, nats_jetstream, nats_kv and nats_stream components. (@ye11ow)
• Field private_key added to the snowflake_put output. (@mihaitodor)
• New azure_data_lake_gen2 output. (@ooesili)
• New timeplus output. (@ye11ow)

Fixed

• The elasticsearch output now performs retries for HTTP status code 429 (Too Many Requests). (@kahoowkh)
• The docs for the collection field of the mongodb output now specify support for interpolation functions. (@mihaitodor)

Changed

• All components with a default path field value (such as the aws_s3 output) containing the deprecated function count have now been changed to use the new function counter. This could potentially change behaviour in cases where multiple components are executing a mapping with a count function sharing the same of the old default count, and these counters need to cascade. This is an extremely unlikely scenario, but for all users of these components it is recommended that your path is defined explicitly, and in a future major version we will be removing the defaults.

The full change log can be found here.

v4.38.0-rc5

For installation instructions check out the getting started guide.

Added

• Anonymous telemetry data is now sent by Connect instances after running for >5 mins. Details about which data is sent, when it is sent, and how to disable it can be found in the telemetry README. (@Jeffail)
• Field checksum_algorithm added to the aws_s3 output. (@dom-lee-naimuri)
• Field nkey added to nats, nats_jetstream, nats_kv and nats_stream components. (@ye11ow)
• Field private_key added to the snowflake_put output. (@mihaitodor)
• New azure_data_lake_gen2 output. (@ooesili)
• New timeplus output. (@ye11ow)

Fixed

• The elasticsearch output now performs retries for HTTP status code 429 (Too Many Requests). (@kahoowkh)
• The docs for the collection field of the mongodb output now specify support for interpolation functions. (@mihaitodor)

Changed

• All components with a default path field value (such as the aws_s3 output) containing the deprecated function count have now been changed to use the new function counter. This could potentially change behaviour in cases where multiple components are executing a mapping with a count function sharing the same of the old default count, and these counters need to cascade. This is an extremely unlikely scenario, but for all users of these components it is recommended that your path is defined explicitly, and in a future major version we will be removing the defaults.

The full change log can be found here.

v4.38.0-rc4

For installation instructions check out the getting started guide.

Added

• Anonymous telemetry data is now sent by Connect instances after running for >5 mins. Details about which data is sent, when it is sent, and how to disable it can be found in the telemetry README. (@Jeffail)
• Field checksum_algorithm added to the aws_s3 output. (@dom-lee-naimuri)
• Field nkey added to nats, nats_jetstream, nats_kv and nats_stream components. (@ye11ow)
• Field private_key added to the snowflake_put output. (@mihaitodor)
• New azure_data_lake_gen2 output. (@ooesili)
• New timeplus output. (@ye11ow)

Fixed

• The elasticsearch output now performs retries for HTTP status code 429 (Too Many Requests). (@kahoowkh)
• The docs for the collection field of the mongodb output now specify support for interpolation functions. (@mihaitodor)

Changed

• All components with a default path field value (such as the aws_s3 output) containing the deprecated function count have now been changed to use the new function counter. This could potentially change behaviour in cases where multiple components are executing a mapping with a count function sharing the same of the old default count, and these counters need to cascade. This is an extremely unlikely scenario, but for all users of these components it is recommended that your path is defined explicitly, and in a future major version we will be removing the defaults.

The full change log can be found here.

v4.38.0-rc3

For installation instructions check out the getting started guide.

NOTE: This is a release candidate, you can download a binary from this page.

Added

• Field checksum_algorithm added to the aws_s3 output. (@dom-lee-naimuri)
• Field nkey added to nats, nats_jetstream, nats_kv and nats_stream components. (@ye11ow)
• Field private_key added to the snowflake_put output. (@mihaitodor)

Fixed

• The elasticsearch output now performs retries for HTTP status code 429 (Too Many Requests). (@kahoowkh)
• The docs for the collection field of the mongodb output now specify support for interpolation functions. (@mihaitodor)

The full change log can be found here.

v4.37.0

For installation instructions check out the getting started guide.

Added

• New experimental gcp_vertex_ai_embeddings processor. (@rockwotj)
• New experimental aws_bedrock_embeddings processor. (@rockwotj)
• New experimental cohere_chat and cohere_embeddings processors. (@rockwotj)
• New experimental questdb output. (@sklarsa)
• Field metadata_max_age added to the kafka_franz input. (@Scarjit)
• Field metadata_max_age added to the kafka_migrator input. (@mihaitodor)
• New experimental cypher output. (@rockwotj)
• New experimental couchbase output. (@rockwotj)
• Field fetch_in_order added to the schema_registry input. (@mihaitodor)

Fixed

• Fixed a bug with the input_resource field for the kafka_migrator output where new topics weren't created as expected. (@mihaitodor)
• Fixed a bug in the kafka_migrator input which could lead to extra duplicate messages during a consumer group rebalance. (@mihaitodor)
• kafka_migrator, kafka_migrator_offsets and kafka_migrator_bundle components renamed to redpanda_migrator, redpanda_migrator_offsets and redpanda_migrator_bundle (@mihaitodor)

Fixed

• Fixes a panic in the parquet_encode processor (@mihaitodor)

The full change log can be found here.

v4.37.0-rc2

For installation instructions check out the getting started guide.

Added

• New experimental gcp_vertex_ai_embeddings processor. (@rockwotj)
• New experimental aws_bedrock_embeddings processor. (@rockwotj)
• New experimental cohere_chat and cohere_embeddings processors. (@rockwotj)
• New experimental questdb output. (@sklarsa)
• Field metadata_max_age added to the kafka_franz input. (@Scarjit)
• Field metadata_max_age added to the kafka_migrator input. (@mihaitodor)
• New experimental cypher output. (@rockwotj)
• New experimental couchbase output. (@rockwotj)
• Field fetch_in_order added to the schema_registry input. (@mihaitodor)

Fixed

• Fixed a bug with the input_resource field for the kafka_migrator output where new topics weren't created as expected. (@mihaitodor)
• Fixed a bug in the kafka_migrator input which could lead to extra duplicate messages during a consumer group rebalance. (@mihaitodor)
• kafka_migrator, kafka_migrator_offsets and kafka_migrator_bundle components renamed to redpanda_migrator, redpanda_migrator_offsets and redpanda_migrator_bundle (@mihaitodor)

Fixed

• Fixes a panic in the parquet_encode processor (@mihaitodor)

The full change log can be found here.