From 56f93faeba0c9c4da0d50462bf13b3f072686d3c Mon Sep 17 00:00:00 2001 From: fengyubiao Date: Mon, 5 Aug 2024 19:40:44 +0800 Subject: [PATCH 01/18] [improve] [pip] PIP-370: An optional flag: disable the replicators to automatically trigger topic creation --- pip/pip-370.md | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 pip/pip-370.md diff --git a/pip/pip-370.md b/pip/pip-370.md new file mode 100644 index 0000000000000..6e24ad6020c0f --- /dev/null +++ b/pip/pip-370.md @@ -0,0 +1,48 @@ +# PIP-370: An optional flag: disable the replicators to automatically trigger topic creation + +# Background knowledge + +Users using Geo-Replication backup data across multi clusters, and Admin APIs related to Geo-Replication and internal replicators of brokers will trigger topics auto-creation between clusters. +- For partitioned topics. + - After enabled namespace-level Geo-Replication: broker will create topics on the remote cluster automatically when calling `pulsar-admin topics create-partitioned-topic`. It does not depend on enabling `allowAutoTopicCreation`. + - When enabling topic-level Geo-Replication on a partitioned topic: broker will create topics on the remote cluster automatically. It does not depend on enabling `allowAutoTopicCreation`. +- For non-partitioned topics and partitions of partitioned topics. + - The internal Geo-Replicator will trigger topics auto-creation for remote clusters. **(Highlight)** It depends on enabling `allowAutoTopicCreation`. In fact, this behavior is not related to Geo-Replication, it is the behavior of the internal producer of Geo Replicator, + +# Motivation + +But if users want to maintain pulsar resources manually, there is no choice that prevent pulsar resources creation affects each other between clusters. + +**Passed solution**: disable `allowAutoTopicCreation`, the APIs `pulsar-admin topics create-partitioned-topic` still create topics on the remote cluster when enabled namespace level replication, the API `enable topic-level replication` still create topics, And the internal replicator will keep printing error logs due to a not found error. + +# Goals + +Introduce a flag that used to disable the replicators to automatically trigger topic creation. + +# Detailed Design + +## Configuration + +**broker.conf** +```properties +# It is not a dynamic config, default value is "true". +# See details below. +replicationTriggerRemoteTopicCreation=true +``` + +## Design & Implementation Details + +After disabled `replicationTriggerRemoteTopicCreation`. +- After enabled namespace-level Geo-Replication: broker will not create topics on the remote cluster automatically when calling `pulsar-admin topics create-partitioned-topic`. +- When enabling topic-level Geo-Replication on a partitioned topic: broker will not create topics on the remote cluster automatically. +- The internal Geo-Replicator will not trigger topics auto-creation for remote clusters, it just keeps retrying to check if the topic exists on the remote cluster, once the topic was created, the replicator start. +- It does not change the behavior that creating subscriptions after enable `enableReplicatedSubscriptions`, the subscription will also be created on the remote cluster after users enabled `enableReplicatedSubscriptions`. + +# Backward & Forward Compatibility + +The feature that disable `replicationTriggerRemoteTopicCreation` depends on the API `PulsarClient.getPartitionsForTopic(String topic, boolean metadataAutoCreationEnabled)`, which was introduced by [PIP-344](https://github.com/apache/pulsar/blob/master/pip/pip-344.md). +- Only when user's all clusters support PIP-344, users can disable `replicationTriggerRemoteTopicCreation`( `>=3.0.6` or `>=3.3.1` ). + +# Links +* Mailing List discussion thread: +* Mailing List voting thread: From 101ed51e7753774f72003d76916c3bee50196262 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 8 Aug 2024 11:47:32 +0800 Subject: [PATCH 02/18] Update pip-370.md --- pip/pip-370.md | 36 ++++++++++++++++++++++-------------- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index 6e24ad6020c0f..3a7b3225ead68 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -2,22 +2,27 @@ # Background knowledge -Users using Geo-Replication backup data across multi clusters, and Admin APIs related to Geo-Replication and internal replicators of brokers will trigger topics auto-creation between clusters. +Users using Geo-Replication backup data across multiple clusters, as well as Admin APIs related to Geo-Replication and internal replicators of brokers, will trigger topics of auto-creation between clusters. - For partitioned topics. - - After enabled namespace-level Geo-Replication: broker will create topics on the remote cluster automatically when calling `pulsar-admin topics create-partitioned-topic`. It does not depend on enabling `allowAutoTopicCreation`. - - When enabling topic-level Geo-Replication on a partitioned topic: broker will create topics on the remote cluster automatically. It does not depend on enabling `allowAutoTopicCreation`. + - After enabling namespace-level Geo-Replication: the broker will create topics on the remote cluster automatically when calling `pulsar-admin topics create-partitioned-topic`. It does not depend on enabling `allowAutoTopicCreation`. + - When enabling topic-level Geo-Replication on a partitioned topic: the broker will create topics on the remote cluster automatically. It does not depend on enabling `allowAutoTopicCreation`. - For non-partitioned topics and partitions of partitioned topics. - - The internal Geo-Replicator will trigger topics auto-creation for remote clusters. **(Highlight)** It depends on enabling `allowAutoTopicCreation`. In fact, this behavior is not related to Geo-Replication, it is the behavior of the internal producer of Geo Replicator, + - The internal Geo-Replicator will trigger topics auto-creation for remote clusters. **(Highlight)** It depends on enabling `allowAutoTopicCreation`. In fact, this behavior is not related to Geo-Replication, it is the behavior of the internal producer of Geo-Replicator, # Motivation -But if users want to maintain pulsar resources manually, there is no choice that prevent pulsar resources creation affects each other between clusters. +In the following scenarios, automatic topic creation across clusters is problematic due to race conditions during deployments, and there is no choice that prevents pulsar resource creation affects each other between clusters. +1. Users want to maintain pulsar resources manually. +2. Users pulsar resources using `GitOps CD` automated deployment, for which + a. Clusters are deployed simultaneously without user intervention. + b. Each cluster is precisely configured from git repo config variables - including the list of all tenants/namespaces/topics to be created in each cluster. + c. Clusters are configured to be exact clones of each other in terms of pulsar resources. **Passed solution**: disable `allowAutoTopicCreation`, the APIs `pulsar-admin topics create-partitioned-topic` still create topics on the remote cluster when enabled namespace level replication, the API `enable topic-level replication` still create topics, And the internal replicator will keep printing error logs due to a not found error. # Goals -Introduce a flag that used to disable the replicators to automatically trigger topic creation. +Introduce a flag to disable the replicators to automatically trigger topic creation. # Detailed Design @@ -25,24 +30,27 @@ Introduce a flag that used to disable the replicators to automatically trigger t **broker.conf** ```properties -# It is not a dynamic config, default value is "true". +# It is not a dynamic config, the default value is "true" to preserve backward-compatible behavior. # See details below. replicationTriggerRemoteTopicCreation=true ``` ## Design & Implementation Details -After disabled `replicationTriggerRemoteTopicCreation`. -- After enabled namespace-level Geo-Replication: broker will not create topics on the remote cluster automatically when calling `pulsar-admin topics create-partitioned-topic`. -- When enabling topic-level Geo-Replication on a partitioned topic: broker will not create topics on the remote cluster automatically. -- The internal Geo-Replicator will not trigger topics auto-creation for remote clusters, it just keeps retrying to check if the topic exists on the remote cluster, once the topic was created, the replicator start. -- It does not change the behavior that creating subscriptions after enable `enableReplicatedSubscriptions`, the subscription will also be created on the remote cluster after users enabled `enableReplicatedSubscriptions`. +- If `replicationTriggerRemoteTopicCreation` is set to `false`. + 1. After enabling namespace-level Geo-Replication: the broker will not create topics on the remote cluster automatically when calling `pulsar-admin topics create-partitioned-topic`. + 2. When enabling topic-level Geo-Replication on a partitioned topic: broker will not create topics on the remote cluster automatically. + 3. The internal Geo-Replicator will not trigger topic auto-creation for remote clusters, it just keeps retrying to check if the topic exists on the remote cluster, once the topic is created, the replicator starts. + 4. It does not change the behavior that creating subscriptions after enabling `enableReplicatedSubscriptions`, the subscription will also be created on the remote cluster after users enable. `enableReplicatedSubscriptions`. + 5. The config `allowAutoTopicCreation` still works for the local cluster as before, it will not be affected by the new config `replicationTriggerRemoteTopicCreation`. +- If `replicationTriggerRemoteTopicCreation` is set to `true`. + a. All components work as before. # Backward & Forward Compatibility -The feature that disable `replicationTriggerRemoteTopicCreation` depends on the API `PulsarClient.getPartitionsForTopic(String topic, boolean metadataAutoCreationEnabled)`, which was introduced by [PIP-344](https://github.com/apache/pulsar/blob/master/pip/pip-344.md). +The feature that disables `replicationTriggerRemoteTopicCreation` depends on the API `PulsarClient.getPartitionsForTopic(String topic, boolean metadataAutoCreationEnabled)`, which was introduced by [PIP-344](https://github.com/apache/pulsar/blob/master/pip/pip-344.md). - Only when user's all clusters support PIP-344, users can disable `replicationTriggerRemoteTopicCreation`( `>=3.0.6` or `>=3.3.1` ). # Links -* Mailing List discussion thread: +* Mailing List discussion thread: https://lists.apache.org/thread/9fx354cqcy3412w1nx8kwdf9h141omdg * Mailing List voting thread: From d2d7da571e22920f67557404b4a6666d5f10a3b4 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 8 Aug 2024 14:43:18 +0800 Subject: [PATCH 03/18] Update pip-370.md --- pip/pip-370.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index 3a7b3225ead68..b58a03c7baecb 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -49,7 +49,8 @@ replicationTriggerRemoteTopicCreation=true # Backward & Forward Compatibility The feature that disables `replicationTriggerRemoteTopicCreation` depends on the API `PulsarClient.getPartitionsForTopic(String topic, boolean metadataAutoCreationEnabled)`, which was introduced by [PIP-344](https://github.com/apache/pulsar/blob/master/pip/pip-344.md). -- Only when user's all clusters support PIP-344, users can disable `replicationTriggerRemoteTopicCreation`( `>=3.0.6` or `>=3.3.1` ). + +Since PIP-370 will be released after PIP-344, it is fine. This is just to illustrate that if someone wants to cherry-pick PIP-370 for another branch in the future, they need to cherry-pick PIP-344 as well. # Links * Mailing List discussion thread: https://lists.apache.org/thread/9fx354cqcy3412w1nx8kwdf9h141omdg From ebee688bbf742d40e78ae7b68a9ae118202527f4 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 8 Aug 2024 14:46:21 +0800 Subject: [PATCH 04/18] Update pip-370.md --- pip/pip-370.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index b58a03c7baecb..6b5cb7a67a86e 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -12,11 +12,12 @@ Users using Geo-Replication backup data across multiple clusters, as well as Adm # Motivation In the following scenarios, automatic topic creation across clusters is problematic due to race conditions during deployments, and there is no choice that prevents pulsar resource creation affects each other between clusters. -1. Users want to maintain pulsar resources manually. -2. Users pulsar resources using `GitOps CD` automated deployment, for which - a. Clusters are deployed simultaneously without user intervention. - b. Each cluster is precisely configured from git repo config variables - including the list of all tenants/namespaces/topics to be created in each cluster. - c. Clusters are configured to be exact clones of each other in terms of pulsar resources. + +- Users want to maintain pulsar resources manually. +- Users pulsar resources using `GitOps CD` automated deployment, for which + - Clusters are deployed simultaneously without user intervention. + - Each cluster is precisely configured from git repo config variables - including the list of all tenants/namespaces/topics to be created in each cluster. + - Clusters are configured to be exact clones of each other in terms of pulsar resources. **Passed solution**: disable `allowAutoTopicCreation`, the APIs `pulsar-admin topics create-partitioned-topic` still create topics on the remote cluster when enabled namespace level replication, the API `enable topic-level replication` still create topics, And the internal replicator will keep printing error logs due to a not found error. From f73bbaa60697480c2ba3bc47c6eca2f65bd56ee2 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Mon, 12 Aug 2024 11:34:35 +0800 Subject: [PATCH 05/18] Update pip-370.md --- pip/pip-370.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index 6b5cb7a67a86e..ffc99910820e8 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -55,4 +55,4 @@ Since PIP-370 will be released after PIP-344, it is fine. This is just to illust # Links * Mailing List discussion thread: https://lists.apache.org/thread/9fx354cqcy3412w1nx8kwdf9h141omdg -* Mailing List voting thread: +* Mailing List voting thread: https://lists.apache.org/thread/vph22st5td1rdh1gd68gkrnp9doo6ct2 From 26717bcdeb401be619a588519277cd44900c117b Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Mon, 12 Aug 2024 14:06:24 +0800 Subject: [PATCH 06/18] Update pip-370.md --- pip/pip-370.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index ffc99910820e8..65a2703951d19 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -1,4 +1,4 @@ -# PIP-370: An optional flag: disable the replicators to automatically trigger topic creation +# PIP-370: configurable remote topic creation in geo-replication # Background knowledge From bc2512134fc8e548185d9ec9dba5b4a05715308a Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Tue, 13 Aug 2024 10:23:17 +0800 Subject: [PATCH 07/18] Update pip-370.md --- pip/pip-370.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index 65a2703951d19..b5150b1f94e86 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -33,23 +33,23 @@ Introduce a flag to disable the replicators to automatically trigger topic creat ```properties # It is not a dynamic config, the default value is "true" to preserve backward-compatible behavior. # See details below. -replicationTriggerRemoteTopicCreation=true +createTopicToRemoteClusterForReplication=true ``` ## Design & Implementation Details -- If `replicationTriggerRemoteTopicCreation` is set to `false`. +- If `createTopicToRemoteClusterForReplication` is set to `false`. 1. After enabling namespace-level Geo-Replication: the broker will not create topics on the remote cluster automatically when calling `pulsar-admin topics create-partitioned-topic`. 2. When enabling topic-level Geo-Replication on a partitioned topic: broker will not create topics on the remote cluster automatically. 3. The internal Geo-Replicator will not trigger topic auto-creation for remote clusters, it just keeps retrying to check if the topic exists on the remote cluster, once the topic is created, the replicator starts. 4. It does not change the behavior that creating subscriptions after enabling `enableReplicatedSubscriptions`, the subscription will also be created on the remote cluster after users enable. `enableReplicatedSubscriptions`. - 5. The config `allowAutoTopicCreation` still works for the local cluster as before, it will not be affected by the new config `replicationTriggerRemoteTopicCreation`. -- If `replicationTriggerRemoteTopicCreation` is set to `true`. + 5. The config `allowAutoTopicCreation` still works for the local cluster as before, it will not be affected by the new config `createTopicToRemoteClusterForReplication`. +- If `createTopicToRemoteClusterForReplication` is set to `true`. a. All components work as before. # Backward & Forward Compatibility -The feature that disables `replicationTriggerRemoteTopicCreation` depends on the API `PulsarClient.getPartitionsForTopic(String topic, boolean metadataAutoCreationEnabled)`, which was introduced by [PIP-344](https://github.com/apache/pulsar/blob/master/pip/pip-344.md). +The feature that disables `createTopicToRemoteClusterForReplication` depends on the API `PulsarClient.getPartitionsForTopic(String topic, boolean metadataAutoCreationEnabled)`, which was introduced by [PIP-344](https://github.com/apache/pulsar/blob/master/pip/pip-344.md). Since PIP-370 will be released after PIP-344, it is fine. This is just to illustrate that if someone wants to cherry-pick PIP-370 for another branch in the future, they need to cherry-pick PIP-344 as well. From 3026a4daf1ef04a628d57d4ce49af76f99111188 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Tue, 13 Aug 2024 10:25:06 +0800 Subject: [PATCH 08/18] Update pip-370.md --- pip/pip-370.md | 1 + 1 file changed, 1 insertion(+) diff --git a/pip/pip-370.md b/pip/pip-370.md index b5150b1f94e86..fa7e50506bbda 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -2,6 +2,7 @@ # Background knowledge +## The current behavior of enabling Geo-Replication Users using Geo-Replication backup data across multiple clusters, as well as Admin APIs related to Geo-Replication and internal replicators of brokers, will trigger topics of auto-creation between clusters. - For partitioned topics. - After enabling namespace-level Geo-Replication: the broker will create topics on the remote cluster automatically when calling `pulsar-admin topics create-partitioned-topic`. It does not depend on enabling `allowAutoTopicCreation`. From 25d6de5fa3f45374f51ab5738e878775a2249860 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Tue, 13 Aug 2024 10:31:40 +0800 Subject: [PATCH 09/18] Update pip-370.md --- pip/pip-370.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index fa7e50506bbda..59f402a61abb1 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -2,7 +2,7 @@ # Background knowledge -## The current behavior of enabling Geo-Replication +**The current topic creation behavior when enabling Geo-Replication** Users using Geo-Replication backup data across multiple clusters, as well as Admin APIs related to Geo-Replication and internal replicators of brokers, will trigger topics of auto-creation between clusters. - For partitioned topics. - After enabling namespace-level Geo-Replication: the broker will create topics on the remote cluster automatically when calling `pulsar-admin topics create-partitioned-topic`. It does not depend on enabling `allowAutoTopicCreation`. @@ -46,7 +46,7 @@ createTopicToRemoteClusterForReplication=true 4. It does not change the behavior that creating subscriptions after enabling `enableReplicatedSubscriptions`, the subscription will also be created on the remote cluster after users enable. `enableReplicatedSubscriptions`. 5. The config `allowAutoTopicCreation` still works for the local cluster as before, it will not be affected by the new config `createTopicToRemoteClusterForReplication`. - If `createTopicToRemoteClusterForReplication` is set to `true`. - a. All components work as before. + a. All components work as before, see details: `Motivation -> The current topic creation behavior when enabling Geo-Replication` # Backward & Forward Compatibility From 76e41a45a8d7ee46820459310ed4d1f14e009f73 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Tue, 13 Aug 2024 10:41:28 +0800 Subject: [PATCH 10/18] Update pip-370.md --- pip/pip-370.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index 59f402a61abb1..99fa02b1cb70a 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -50,9 +50,7 @@ createTopicToRemoteClusterForReplication=true # Backward & Forward Compatibility -The feature that disables `createTopicToRemoteClusterForReplication` depends on the API `PulsarClient.getPartitionsForTopic(String topic, boolean metadataAutoCreationEnabled)`, which was introduced by [PIP-344](https://github.com/apache/pulsar/blob/master/pip/pip-344.md). - -Since PIP-370 will be released after PIP-344, it is fine. This is just to illustrate that if someone wants to cherry-pick PIP-370 for another branch in the future, they need to cherry-pick PIP-344 as well. +This PIP guarantees full compatibility with default settings(the default value of `createTopicToRemoteClusterForReplication` is `true`). If you want to cherry-pick PIP-370 for another branch in the future, you need to cherry-pick PIP-344 as well. Because the behavior of disables `createTopicToRemoteClusterForReplication` depends on the API `PulsarClient.getPartitionsForTopic(String topic, boolean metadataAutoCreationEnabled)`, which was introduced by [PIP-344](https://github.com/apache/pulsar/blob/master/pip/pip-344.md). # Links * Mailing List discussion thread: https://lists.apache.org/thread/9fx354cqcy3412w1nx8kwdf9h141omdg From 31b0519f07d5cc5c4debff3f6dfc352777702b00 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Tue, 13 Aug 2024 22:01:32 +0800 Subject: [PATCH 11/18] Update pip-370.md --- pip/pip-370.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index 99fa02b1cb70a..7dd6ed96208ee 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -32,8 +32,11 @@ Introduce a flag to disable the replicators to automatically trigger topic creat **broker.conf** ```properties -# It is not a dynamic config, the default value is "true" to preserve backward-compatible behavior. -# See details below. +# Whether the internal replicator will trigger topic auto-creation on the remote cluster. +# 1. After enabling namespace-level Geo-Replication: whether the local broker will create topics on the remote cluster automatically when calling `pulsar-admin topics create-partitioned-topic`. +# 2. When enabling topic-level Geo-Replication on a partitioned topic: whether the local broker will create topics on the remote cluster. +# 3. Whether the internal Geo-Replicator in the local cluster will trigger non-persistent topic auto-creation for remote clusters. +# It is not a dynamic config, the default value is "true" to preserve backward-compatible behavior. createTopicToRemoteClusterForReplication=true ``` From 6b5499c22bcc8b91a0cdadfaaa6e50f5c2291441 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Tue, 13 Aug 2024 22:04:58 +0800 Subject: [PATCH 12/18] Update pip-370.md --- pip/pip-370.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index 7dd6ed96208ee..a8cde23ca0347 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -32,7 +32,7 @@ Introduce a flag to disable the replicators to automatically trigger topic creat **broker.conf** ```properties -# Whether the internal replicator will trigger topic auto-creation on the remote cluster. +# Whether the internal replication of the local cluster will trigger topic auto-creation on the remote cluster. # 1. After enabling namespace-level Geo-Replication: whether the local broker will create topics on the remote cluster automatically when calling `pulsar-admin topics create-partitioned-topic`. # 2. When enabling topic-level Geo-Replication on a partitioned topic: whether the local broker will create topics on the remote cluster. # 3. Whether the internal Geo-Replicator in the local cluster will trigger non-persistent topic auto-creation for remote clusters. From 5b6acf56b9a9abea80e103d9d327b989a9780165 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Wed, 14 Aug 2024 20:24:47 +0800 Subject: [PATCH 13/18] Update pip-370.md --- pip/pip-370.md | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index a8cde23ca0347..dc559852438b2 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -7,6 +7,7 @@ Users using Geo-Replication backup data across multiple clusters, as well as Adm - For partitioned topics. - After enabling namespace-level Geo-Replication: the broker will create topics on the remote cluster automatically when calling `pulsar-admin topics create-partitioned-topic`. It does not depend on enabling `allowAutoTopicCreation`. - When enabling topic-level Geo-Replication on a partitioned topic: the broker will create topics on the remote cluster automatically. It does not depend on enabling `allowAutoTopicCreation`. + - When calling `pulsar-admin topics update-partitioned-topic -p {partitions}`, the broker will also update partitions on the remote cluster automatically. - For non-partitioned topics and partitions of partitioned topics. - The internal Geo-Replicator will trigger topics auto-creation for remote clusters. **(Highlight)** It depends on enabling `allowAutoTopicCreation`. In fact, this behavior is not related to Geo-Replication, it is the behavior of the internal producer of Geo-Replicator, @@ -24,7 +25,9 @@ In the following scenarios, automatic topic creation across clusters is problema # Goals -Introduce a flag to disable the replicators to automatically trigger topic creation. +- Introduce a flag to disable the replicators to automatically trigger topic creation. +- Move all topic creation/expand-partitions behaviors related to Replication to the internal Replicator, pulsar admin API that relates to pulsar topics management does not care about replication anymore. + - Move the topic creation operations from `pulsar-admin topics create-partitioned-topic`, `pulsar-admin topics update-partitioned-topic -p {partitions}` and `pulsar-admin topics set-replication-clusters` to the component Replicator in the broker internal. # Detailed Design @@ -42,19 +45,47 @@ createTopicToRemoteClusterForReplication=true ## Design & Implementation Details +### Phase 1: Introduce a flag to disable the replicators to automatically trigger topic creation. - If `createTopicToRemoteClusterForReplication` is set to `false`. 1. After enabling namespace-level Geo-Replication: the broker will not create topics on the remote cluster automatically when calling `pulsar-admin topics create-partitioned-topic`. 2. When enabling topic-level Geo-Replication on a partitioned topic: broker will not create topics on the remote cluster automatically. 3. The internal Geo-Replicator will not trigger topic auto-creation for remote clusters, it just keeps retrying to check if the topic exists on the remote cluster, once the topic is created, the replicator starts. - 4. It does not change the behavior that creating subscriptions after enabling `enableReplicatedSubscriptions`, the subscription will also be created on the remote cluster after users enable. `enableReplicatedSubscriptions`. + 4. It does not change the behavior of creating subscriptions after enabling `enableReplicatedSubscriptions`, the subscription will also be created on the remote cluster after users enable. `enableReplicatedSubscriptions`. 5. The config `allowAutoTopicCreation` still works for the local cluster as before, it will not be affected by the new config `createTopicToRemoteClusterForReplication`. - If `createTopicToRemoteClusterForReplication` is set to `true`. a. All components work as before, see details: `Motivation -> The current topic creation behavior when enabling Geo-Replication` +### Phase 2: The replicator will check remote topics' partitioned metadata and update partitions in the remote cluster to the same as the current cluster if needed. +- If `createTopicToRemoteClusterForReplication` is set to `false`. + - The behavior is the same as Phase 1. +- If `createTopicToRemoteClusterForReplication` is set to `true`. + - When a replicator for a topic partition starts, it checks the partitioned metadata in the remote cluster first and updates partitions in the remote cluster to the same as the current cluster if needed. Seem the example as follows: + - `local_cluster.topic.partitions = 2` and the topic does not exist in the remote cluster: create a partitioned topic with `2` partitions in the remote cluster. + - Before `PIP-370 Phase 2`: the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), and will not care about partitioned metadata. + - `local_cluster.topic.partitions = 2` and `remote_cluster.topic.partitions = 1`: expand `remote_cluster.topic.partitions` to `2`. + - Before `PIP-370 Phase 2`: the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), and the partitioned metadata in the remote cluster is still `1`. + - `local_cluster.topic.partitions = 2` and `remote_cluster.topic.partitions >=2 `: modifies nothing. + - Before `PIP-370 Phase 2`: It is the same as before `PIP-370`. + - `local_cluster.topic.partitions = 2` and `remote_cluster.topic.partitions = 0(non-partitioned topic)`: the replicator prints error logs in the background. + - Before `PIP-370 Phase 2`: the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), then users will get `3` non-partitioned topics: `[{tp}, {topic}-partition-0, {topic}-partition-1`. + - `local_cluster.topic.partitions = 0(non-partitioned topic)` and `remote_cluster.topic.partitions = 0(non-partitioned topic)`: modifies nothing. + - Before `PIP-370 Phase 2`: It is the same as before `PIP-370`. + - `local_cluster.topic.partitions = 0(non-partitioned topic)` and the topic does not exist in the remote cluster: create a non-partitioned topic in the remote cluster. + - Before `PIP-370 Phase 2`: It is the same as before `PIP-370`. + - `local_cluster.topic.partitions = 0(non-partitioned topic)` and and `remote_cluster.topic.partitions >= 1`: the replicator prints error logs in the background. + - Before `PIP-370 Phase 2`: the replicator will only trigger a non-partitioned topic's creation, then users will get `1` non-partitioned topic and `1` partitioned topic. + - Pulsar admin API that relates to pulsar topics management does not care about replication anymore. + # Backward & Forward Compatibility +## Regarding to Phase-1 This PIP guarantees full compatibility with default settings(the default value of `createTopicToRemoteClusterForReplication` is `true`). If you want to cherry-pick PIP-370 for another branch in the future, you need to cherry-pick PIP-344 as well. Because the behavior of disables `createTopicToRemoteClusterForReplication` depends on the API `PulsarClient.getPartitionsForTopic(String topic, boolean metadataAutoCreationEnabled)`, which was introduced by [PIP-344](https://github.com/apache/pulsar/blob/master/pip/pip-344.md). +## Regarding to Phase-2 +The two scenarios are as follows, the replication will not work as before, which will lead backlog increase, please take care of checking your clusters before upgrading. +- `local_cluster.topic.partitions = 2` and `remote_cluster.topic.partitions = 0(non-partitioned topic)`: see detail in the section `Design & Implementation Details -> Phase-2`. +- `local_cluster.topic.partitions = 0(non-partitioned topic)` and and `remote_cluster.topic.partitions >= 1`: see detail in the section `Design & Implementation Details -> Phase-2`. + # Links * Mailing List discussion thread: https://lists.apache.org/thread/9fx354cqcy3412w1nx8kwdf9h141omdg * Mailing List voting thread: https://lists.apache.org/thread/vph22st5td1rdh1gd68gkrnp9doo6ct2 From ffe32e8ac7ce4a7221fb51173b97664ee5cdd18d Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 15 Aug 2024 12:14:35 +0800 Subject: [PATCH 14/18] Update pip-370.md --- pip/pip-370.md | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index dc559852438b2..291caf666a718 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -59,22 +59,19 @@ createTopicToRemoteClusterForReplication=true - If `createTopicToRemoteClusterForReplication` is set to `false`. - The behavior is the same as Phase 1. - If `createTopicToRemoteClusterForReplication` is set to `true`. + - Pulsar admin API that relates to pulsar topics management does not care about replication anymore. - When a replicator for a topic partition starts, it checks the partitioned metadata in the remote cluster first and updates partitions in the remote cluster to the same as the current cluster if needed. Seem the example as follows: - - `local_cluster.topic.partitions = 2` and the topic does not exist in the remote cluster: create a partitioned topic with `2` partitions in the remote cluster. - - Before `PIP-370 Phase 2`: the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), and will not care about partitioned metadata. - - `local_cluster.topic.partitions = 2` and `remote_cluster.topic.partitions = 1`: expand `remote_cluster.topic.partitions` to `2`. - - Before `PIP-370 Phase 2`: the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), and the partitioned metadata in the remote cluster is still `1`. - - `local_cluster.topic.partitions = 2` and `remote_cluster.topic.partitions >=2 `: modifies nothing. - - Before `PIP-370 Phase 2`: It is the same as before `PIP-370`. - - `local_cluster.topic.partitions = 2` and `remote_cluster.topic.partitions = 0(non-partitioned topic)`: the replicator prints error logs in the background. - - Before `PIP-370 Phase 2`: the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), then users will get `3` non-partitioned topics: `[{tp}, {topic}-partition-0, {topic}-partition-1`. - - `local_cluster.topic.partitions = 0(non-partitioned topic)` and `remote_cluster.topic.partitions = 0(non-partitioned topic)`: modifies nothing. - - Before `PIP-370 Phase 2`: It is the same as before `PIP-370`. - - `local_cluster.topic.partitions = 0(non-partitioned topic)` and the topic does not exist in the remote cluster: create a non-partitioned topic in the remote cluster. - - Before `PIP-370 Phase 2`: It is the same as before `PIP-370`. - - `local_cluster.topic.partitions = 0(non-partitioned topic)` and and `remote_cluster.topic.partitions >= 1`: the replicator prints error logs in the background. - - Before `PIP-370 Phase 2`: the replicator will only trigger a non-partitioned topic's creation, then users will get `1` non-partitioned topic and `1` partitioned topic. - - Pulsar admin API that relates to pulsar topics management does not care about replication anymore. + +| `partitions` of local cluster | `partitions` of remote cluster | After `PIP-370 Phase 2` | Before `PIP-370 Phase 2` | +| --- | --- | --- | --- | +| `2` | no topic exists | create a partitioned topic with `2` partitions in the remote cluster | the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), and will not care about partitioned metadata. | +| `2` | `1`| expand remote's `partitions` to `2` | the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), and the partitioned metadata in the remote cluster is still `1` | +| `2` | `2` | modifies nothing. | The same as "After `PIP-370 Phase 2`" | +| `2` | `>2` | modifies nothing, the messages will be copied to the same partition in the remote cluster, and no message will be copied to the partition who is larger than `2` in the remote cluster | The same as "After `PIP-370 Phase 2`" | +| `2` | `0`(non-partitioned topic) | The replicator does not copy any data and just prints error logs in the background. | the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), then users will get `3` non-partitioned topics: `[{tp}, {topic}-partition-0, {topic}-partition-1`. | +| `0`(non-partitioned topic) | `0`(non-partitioned topic) | Copy data normally | It is the same as before `PIP-370`. | +| `0`(non-partitioned topic) | no topic exists | create a non-partitioned topic in the remote cluster. | It is the same as before `PIP-370`. | +| `0`(non-partitioned topic) | `>=1` | The replicator does not copy any data and just prints error logs in the background. | The replicator will only trigger a non-partitioned topic's creation, then users will get `1` non-partitioned topic and `1` partitioned topic. | # Backward & Forward Compatibility From 534d1cb7b3842767587be65a8d9c2e5183e1a358 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 15 Aug 2024 14:58:57 +0800 Subject: [PATCH 15/18] Update pip-370.md --- pip/pip-370.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/pip/pip-370.md b/pip/pip-370.md index 291caf666a718..6ce9644484d9a 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -73,6 +73,24 @@ createTopicToRemoteClusterForReplication=true | `0`(non-partitioned topic) | no topic exists | create a non-partitioned topic in the remote cluster. | It is the same as before `PIP-370`. | | `0`(non-partitioned topic) | `>=1` | The replicator does not copy any data and just prints error logs in the background. | The replicator will only trigger a non-partitioned topic's creation, then users will get `1` non-partitioned topic and `1` partitioned topic. | +## Metrics + + +| Name | Description | Attributes | Units| +| --- | --- | --- | --- | +| `pulsar_broker_replication_count` | Counter. The numbers of topics enabled replication. | cluster | - | +| `pulsar_broker_replication_disconnected_count` | Counter. The numbers of topics who enabled replication and it's replicator failed to connected | cluster | - | + + +# Monitoring +- If `pulsar_broker_replication_disconnected_count` keeps lager than `0` for a period time, it means some replicators do not work, we should push an alert out. + # Backward & Forward Compatibility ## Regarding to Phase-1 From 11fd8f2a229b8d478ca099dfb757055c208c3b86 Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 15 Aug 2024 15:01:58 +0800 Subject: [PATCH 16/18] Update pip-370.md --- pip/pip-370.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index 6ce9644484d9a..38387bee98b0a 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -25,9 +25,11 @@ In the following scenarios, automatic topic creation across clusters is problema # Goals -- Introduce a flag to disable the replicators to automatically trigger topic creation. -- Move all topic creation/expand-partitions behaviors related to Replication to the internal Replicator, pulsar admin API that relates to pulsar topics management does not care about replication anymore. +- **Phase 1**: Introduce a flag to disable the replicators to automatically trigger topic creation. +- **Phase 2**: Move all topic creation/expand-partitions behaviors related to Replication to the internal Replicator, pulsar admin API that relates to pulsar topics management does not care about replication anymore. - Move the topic creation operations from `pulsar-admin topics create-partitioned-topic`, `pulsar-admin topics update-partitioned-topic -p {partitions}` and `pulsar-admin topics set-replication-clusters` to the component Replicator in the broker internal. + +Note: the proposal will only focus on phase one, and the detailed design for phase two with come up with another proposal. # Detailed Design @@ -84,12 +86,12 @@ For each metric provide: --> | Name | Description | Attributes | Units| | --- | --- | --- | --- | -| `pulsar_broker_replication_count` | Counter. The numbers of topics enabled replication. | cluster | - | -| `pulsar_broker_replication_disconnected_count` | Counter. The numbers of topics who enabled replication and it's replicator failed to connected | cluster | - | +| `pulsar_broker_replication_count` | Counter. The number of topics enabled replication. | cluster | - | +| `pulsar_broker_replication_disconnected_count` | Counter. The number of topics that enabled replication and its replicator failed to connect | cluster | - | # Monitoring -- If `pulsar_broker_replication_disconnected_count` keeps lager than `0` for a period time, it means some replicators do not work, we should push an alert out. +- If `pulsar_broker_replication_disconnected_count` keeps larger than `0` for a period of time, it means some replicators do not work, we should push an alert out. # Backward & Forward Compatibility From 5b4056c539dbeec09356831b324b02534c7e752e Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Thu, 15 Aug 2024 16:51:29 +0800 Subject: [PATCH 17/18] Update pip-370.md --- pip/pip-370.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index 38387bee98b0a..ea8af6114ce22 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -67,7 +67,7 @@ createTopicToRemoteClusterForReplication=true | `partitions` of local cluster | `partitions` of remote cluster | After `PIP-370 Phase 2` | Before `PIP-370 Phase 2` | | --- | --- | --- | --- | | `2` | no topic exists | create a partitioned topic with `2` partitions in the remote cluster | the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), and will not care about partitioned metadata. | -| `2` | `1`| expand remote's `partitions` to `2` | the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), and the partitioned metadata in the remote cluster is still `1` | +| `2` | `1`| The replicator copies messages from `partition-0` to the remote cluster, does not copy any data for `partition-1` and just prints error logs in the background. | the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), and the partitioned metadata in the remote cluster is still `1` | | `2` | `2` | modifies nothing. | The same as "After `PIP-370 Phase 2`" | | `2` | `>2` | modifies nothing, the messages will be copied to the same partition in the remote cluster, and no message will be copied to the partition who is larger than `2` in the remote cluster | The same as "After `PIP-370 Phase 2`" | | `2` | `0`(non-partitioned topic) | The replicator does not copy any data and just prints error logs in the background. | the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), then users will get `3` non-partitioned topics: `[{tp}, {topic}-partition-0, {topic}-partition-1`. | From 2e8d255c78873772c9c50e2cbcbf48aac66fa66f Mon Sep 17 00:00:00 2001 From: fengyubiao <9947090@qq.com> Date: Fri, 16 Aug 2024 09:29:39 +0800 Subject: [PATCH 18/18] Update pip-370.md --- pip/pip-370.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/pip/pip-370.md b/pip/pip-370.md index ea8af6114ce22..6699846cee105 100644 --- a/pip/pip-370.md +++ b/pip/pip-370.md @@ -27,7 +27,8 @@ In the following scenarios, automatic topic creation across clusters is problema - **Phase 1**: Introduce a flag to disable the replicators to automatically trigger topic creation. - **Phase 2**: Move all topic creation/expand-partitions behaviors related to Replication to the internal Replicator, pulsar admin API that relates to pulsar topics management does not care about replication anymore. - - Move the topic creation operations from `pulsar-admin topics create-partitioned-topic`, `pulsar-admin topics update-partitioned-topic -p {partitions}` and `pulsar-admin topics set-replication-clusters` to the component Replicator in the broker internal. + - Move the topic creation operations from `pulsar-admin topics create-partitioned-topic` and `pulsar-admin topics set-replication-clusters` to the component Replicator in the broker internal. + - (The same as before)When calling `pulsar-admin topics update-partitioned-topic -p {partitions}`, the broker will also update partitions on the remote cluster automatically. Note: the proposal will only focus on phase one, and the detailed design for phase two with come up with another proposal. @@ -67,13 +68,13 @@ createTopicToRemoteClusterForReplication=true | `partitions` of local cluster | `partitions` of remote cluster | After `PIP-370 Phase 2` | Before `PIP-370 Phase 2` | | --- | --- | --- | --- | | `2` | no topic exists | create a partitioned topic with `2` partitions in the remote cluster | the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), and will not care about partitioned metadata. | -| `2` | `1`| The replicator copies messages from `partition-0` to the remote cluster, does not copy any data for `partition-1` and just prints error logs in the background. | the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), and the partitioned metadata in the remote cluster is still `1` | +| `2` | `1`| **In dispute:** The replicator copies messages from `partition-0` to the remote cluster, does not copy any data for `partition-1` and just prints error logs in the background. | the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), and the partitioned metadata in the remote cluster is still `1` | | `2` | `2` | modifies nothing. | The same as "After `PIP-370 Phase 2`" | -| `2` | `>2` | modifies nothing, the messages will be copied to the same partition in the remote cluster, and no message will be copied to the partition who is larger than `2` in the remote cluster | The same as "After `PIP-370 Phase 2`" | -| `2` | `0`(non-partitioned topic) | The replicator does not copy any data and just prints error logs in the background. | the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), then users will get `3` non-partitioned topics: `[{tp}, {topic}-partition-0, {topic}-partition-1`. | +| `2` | `>2` | **In dispute:** modifies nothing, the messages will be copied to the same partition in the remote cluster, and no message will be copied to the partition who is larger than `2` in the remote cluster | The same as "After `PIP-370 Phase 2`" | +| `2` | `0`(non-partitioned topic) | **In dispute:** The replicator does not copy any data and just prints error logs in the background. | the replicator will only trigger partition creation (`{topic}-partition-0` and `{topic}-partition-1`), then users will get `3` non-partitioned topics: `[{tp}, {topic}-partition-0, {topic}-partition-1`. | | `0`(non-partitioned topic) | `0`(non-partitioned topic) | Copy data normally | It is the same as before `PIP-370`. | | `0`(non-partitioned topic) | no topic exists | create a non-partitioned topic in the remote cluster. | It is the same as before `PIP-370`. | -| `0`(non-partitioned topic) | `>=1` | The replicator does not copy any data and just prints error logs in the background. | The replicator will only trigger a non-partitioned topic's creation, then users will get `1` non-partitioned topic and `1` partitioned topic. | +| `0`(non-partitioned topic) | `>=1` | **In dispute:** The replicator does not copy any data and just prints error logs in the background. | The replicator will only trigger a non-partitioned topic's creation, then users will get `1` non-partitioned topic and `1` partitioned topic. | ## Metrics