Skip to content

Commit

Permalink
Rename kafka_migrator components to redpanda_migrator
Browse files Browse the repository at this point in the history
Signed-off-by: Mihai Todor <todormihai@gmail.com>
  • Loading branch information
mihaitodor committed Sep 23, 2024
1 parent 94f9751 commit 076878c
Show file tree
Hide file tree
Showing 14 changed files with 454 additions and 441 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ All notable changes to this project will be documented in this file.

- Fixed a bug with the `input_resource` field for the `kafka_migrator` output where new topics weren't created as expected. (@mihaitodor)
- Fixed a bug in the `kafka_migrator` input which could lead to extra duplicate messages during a consumer group rebalance. (@mihaitodor)
- `kafka_migrator`, `kafka_migrator_offsets` and `kafka_migrator_bundle` components renamed to `redpanda_migrator`, `redpanda_migrator_offsets` and `redpanda_migrator_bundle` (@mihaitodor)

## 4.36.0 - 2024-09-11

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
= kafka_migrator
= redpanda_migrator
:type: input
:status: beta
:categories: ["Services"]
Expand All @@ -23,7 +23,7 @@
component_type_dropdown::[]
A Kafka Migrator input using the https://github.com/twmb/franz-go[Franz Kafka client library^].
A Redpanda Migrator input using the https://github.com/twmb/franz-go[Franz Kafka client library^].
Introduced in version 4.35.0.
Expand All @@ -38,7 +38,7 @@ Common::
# Common config fields, showing default values
input:
label: ""
kafka_migrator:
redpanda_migrator:
seed_brokers: [] # No default (required)
topics: [] # No default (required)
regexp_topics: false
Expand All @@ -55,7 +55,7 @@ Advanced::
# All config fields, showing default values
input:
label: ""
kafka_migrator:
redpanda_migrator:
seed_brokers: [] # No default (required)
topics: [] # No default (required)
regexp_topics: false
Expand All @@ -82,7 +82,7 @@ input:
check: ""
processors: [] # No default (optional)
topic_lag_refresh_period: 5s
output_resource: kafka_migrator_output
output_resource: redpanda_migrator_output
replication_factor_override: true
replication_factor: 3
metadata_max_age: 5m
Expand All @@ -93,15 +93,15 @@ input:
Reads a batch of messages from a Kafka broker and waits for the output to acknowledge the writes before updating the Kafka consumer group offset.
This input should be used in combination with a `kafka_migrator` output which it can query for existing topics.
This input should be used in combination with a `redpanda_migrator` output which it can query for existing topics.
When a consumer group is specified this input consumes one or more topics where partitions will automatically balance across any other connected clients with the same consumer group. When a consumer group is not specified topics can either be consumed in their entirety or with explicit partitions.
It attempts to create all selected topics it along with their associated ACLs in the broker that the `kafka_migrator` output points to identified by the label specified in `output_resource`.
It attempts to create all selected topics it along with their associated ACLs in the broker that the `redpanda_migrator` output points to identified by the label specified in `output_resource`.
== Metrics
Emits a `input_kafka_migrator_lag` metric with `topic` and `partition` labels for each consumed topic.
Emits a `input_redpanda_migrator_lag` metric with `topic` and `partition` labels for each consumed topic.
== Metadata
Expand Down Expand Up @@ -719,12 +719,12 @@ The period of time between each topic lag refresh cycle.
=== `output_resource`
The label of the kafka_migrator output in which the currently selected topics need to be created before attempting to read messages.
The label of the redpanda_migrator output in which the currently selected topics need to be created before attempting to read messages.
*Type*: `string`
*Default*: `"kafka_migrator_output"`
*Default*: `"redpanda_migrator_output"`
=== `replication_factor_override`
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
= kafka_migrator_bundle
= redpanda_migrator_bundle
:type: input
:status: experimental
:categories: ["Services"]
Expand All @@ -23,27 +23,27 @@
component_type_dropdown::[]
Kafka Migrator bundle input
Redpanda Migrator bundle input
```yml
# Config fields, showing default values
input:
label: ""
kafka_migrator_bundle:
kafka_migrator: {} # No default (required)
redpanda_migrator_bundle:
redpanda_migrator: {} # No default (required)
schema_registry: {} # No default (required)
migrate_schemas_before_data: true
```
All-in-one input which reads messages and schemas from a Kafka or Redpanda cluster. This input is meant to be used
together with the `kafka_migrator_bundle` output.
together with the `redpanda_migrator_bundle` output.
== Fields
=== `kafka_migrator`
=== `redpanda_migrator`
The `kafka_migrator` input configuration.
The `redpanda_migrator` input configuration.
*Type*: `object`
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
= kafka_migrator
= redpanda_migrator
:type: output
:status: beta
:categories: ["Services"]
Expand All @@ -23,7 +23,7 @@
component_type_dropdown::[]
A Kafka Migrator output using the https://github.com/twmb/franz-go[Franz Kafka client library^].
A Redpanda Migrator output using the https://github.com/twmb/franz-go[Franz Kafka client library^].
Introduced in version 4.35.0.
Expand All @@ -38,7 +38,7 @@ Common::
# Common config fields, showing default values
output:
label: ""
kafka_migrator:
redpanda_migrator:
seed_brokers: [] # No default (required)
topic: "" # No default (required)
key: "" # No default (optional)
Expand All @@ -63,7 +63,7 @@ Advanced::
# All config fields, showing default values
output:
label: ""
kafka_migrator:
redpanda_migrator:
seed_brokers: [] # No default (required)
topic: "" # No default (required)
key: "" # No default (optional)
Expand Down Expand Up @@ -95,7 +95,7 @@ output:
client_certs: []
sasl: [] # No default (optional)
timestamp: ${! timestamp_unix() } # No default (optional)
input_resource: kafka_migrator_input
input_resource: redpanda_migrator_input
replication_factor_override: true
replication_factor: 3
```
Expand All @@ -105,12 +105,18 @@ output:
Writes a batch of messages to a Kafka broker and waits for acknowledgement before propagating it back to the input.
This output should be used in combination with a `kafka_migrator` input which it can query for topic and ACL configurations.
This output should be used in combination with a `redpanda_migrator` input which it can query for topic and ACL configurations.
If the configured broker does not contain the current message topic, it attempts to create it along with the topic
ACLs which are read automatically from the `kafka_migrator` input identified by the label specified in
ACLs which are read automatically from the `redpanda_migrator` input identified by the label specified in
`input_resource`.
ACL migration adheres to the following principles:
- `ALLOW WRITE` ACLs for topics are not migrated
- `ALLOW ALL` ACLs for topics are downgraded to `ALLOW READ`
- Only topic ACLs are migrated, group ACLs are not migrated
== Examples
Expand All @@ -124,14 +130,14 @@ Writes messages to the configured broker and creates topics and topic ACLs if th
```yaml
output:
kafka_migrator:
redpanda_migrator:
seed_brokers: [ "127.0.0.1:9093" ]
topic: ${! metadata("kafka_topic").or(throw("missing kafka_topic metadata")) }
key: ${! metadata("kafka_key") }
partitioner: manual
partition: ${! metadata("kafka_partition").or(throw("missing kafka_partition metadata")) }
timestamp: ${! metadata("kafka_timestamp_unix").or(timestamp_unix()) }
input_resource: kafka_migrator_input
input_resource: redpanda_migrator_input
max_in_flight: 1
```
Expand Down Expand Up @@ -829,12 +835,12 @@ timestamp: ${! metadata("kafka_timestamp_unix") }
=== `input_resource`
The label of the kafka_migrator input from which to read the configurations for topics and ACLs which need to be created.
The label of the redpanda_migrator input from which to read the configurations for topics and ACLs which need to be created.
*Type*: `string`
*Default*: `"kafka_migrator_input"`
*Default*: `"redpanda_migrator_input"`
=== `replication_factor_override`
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
= kafka_migrator_bundle
= redpanda_migrator_bundle
:type: output
:status: experimental
:categories: ["Services"]
Expand All @@ -23,26 +23,26 @@
component_type_dropdown::[]
Kafka Migrator bundle output
Redpanda Migrator bundle output
```yml
# Config fields, showing default values
output:
label: ""
kafka_migrator_bundle:
kafka_migrator: {} # No default (required)
redpanda_migrator_bundle:
redpanda_migrator: {} # No default (required)
schema_registry: {} # No default (required)
```
All-in-one output which writes messages and schemas to a Kafka or Redpanda cluster. This output is meant to be used
together with the `kafka_migrator_bundle` input.
together with the `redpanda_migrator_bundle` input.
== Fields
=== `kafka_migrator`
=== `redpanda_migrator`
The `kafka_migrator` output configuration.
The `redpanda_migrator` output configuration.
*Type*: `object`
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
= kafka_migrator_offsets
= redpanda_migrator_offsets
:type: output
:status: beta
:categories: ["Services"]
Expand All @@ -23,7 +23,7 @@
component_type_dropdown::[]
Kafka Migrator consumer group offsets output using the https://github.com/twmb/franz-go[Franz Kafka client library^].
Redpanda Migrator consumer group offsets output using the https://github.com/twmb/franz-go[Franz Kafka client library^].
Introduced in version 4.35.0.
Expand All @@ -38,7 +38,7 @@ Common::
# Common config fields, showing default values
output:
label: ""
kafka_migrator_offsets:
redpanda_migrator_offsets:
seed_brokers: [] # No default (required)
kafka_key: ${! @kafka_key }
max_in_flight: 1
Expand All @@ -53,7 +53,7 @@ Advanced::
# All config fields, showing default values
output:
label: ""
kafka_migrator_offsets:
redpanda_migrator_offsets:
seed_brokers: [] # No default (required)
kafka_key: ${! @kafka_key }
client_id: benthos
Expand Down
12 changes: 6 additions & 6 deletions internal/impl/kafka/enterprise/init.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,18 @@ import (
_ "github.com/redpanda-data/benthos/v4/public/components/pure"
)

//go:embed kafka_migrator_bundle_input.tmpl.yaml
var kafkaMigratorInputTemplate []byte
//go:embed redpanda_migrator_bundle_input.tmpl.yaml
var redpandaMigratorInputTemplate []byte

//go:embed kafka_migrator_bundle_output.tmpl.yaml
var kafkaMigratorOutputTemplate []byte
//go:embed redpanda_migrator_bundle_output.tmpl.yaml
var redpandaMigratorOutputTemplate []byte

func init() {
if err := service.RegisterTemplateYAML(string(kafkaMigratorInputTemplate)); err != nil {
if err := service.RegisterTemplateYAML(string(redpandaMigratorInputTemplate)); err != nil {
panic(err)
}

if err := service.RegisterTemplateYAML(string(kafkaMigratorOutputTemplate)); err != nil {
if err := service.RegisterTemplateYAML(string(redpandaMigratorOutputTemplate)); err != nil {
panic(err)
}
}
Loading

0 comments on commit 076878c

Please sign in to comment.