Regex patten based topic consumers prevent topic deletion

[demo-client]

Describe the bug
I am using Java client libraries. I created a consumer using topicsPattern for a regex pattern. As new topics matching the pattern get created, the consumer starts consuming from it. This is expected. However, I delete a topic that the consumer is consuming (even with an -f option); the topic does not get deleted (looks like it gets re-created). This is NOT EXPECTED, and is a DEFECT.

To Reproduce
Steps to reproduce the behavior:

1. Create a consumer using topicsPattern for a regex pattern say persistent://public/default/test.*
2. Create producers to start sending messages to topics test01, test02, test03 etc.
3. Once the consumer consumes the messages, stop producing to these topics
4. Execute ./pulsar-admin topics list public/default and verify that these topics do exist.
5. Execute ./pulsar-admin topics delete -f persistent://public/default/test02 to delete the topic.
6. No errors are shown.
7. However, execute ./pulsar-admin topics list public/default again and notice that test02 continues to exist.

Expected behavior
The system should allow deletion of topics even if regex pattern-based consumers have been attached to a topic.

Screenshots
NA

Desktop (please complete the following information):

• Java

Additional context
This problem gains importance as we want some listeners to ALL topics matching a certain pattern. And we have strong business reasons for the topics to be ephemeral in nature (create on demand, and delete when the work for that named topic is over)

[sijie]

@anandsinha

By default Pulsar creates topic when a consumer or a producer connects to the broker. That means even you don't use the regex subscription directly, you have a consumer connect to the topic. The topic will be recreated if a topic is deleted.

Pulsar provides a flag to disable topic auto creation. Does disabling the flag work for you?

[demo-client]

Hello @sijie , while this is a perfect technical explanation of why the issue happens; this is not how one would expect a regex pattern based subscription to work.
A regex pattern based subscription should allow the topic-list to grow when new topics are created.
Similarly,
the regex pattern based subscription should allow the topic-list to SHRINK when new topics are DELETED.

Please do let me know if you believe that this is not a valid business requirement.

[sijie]

@anandsinha agreed.

the solution I can think of is to add a flag in the consumer to tell the brokers that the consumer is a regex subscription. hence the broker will not auto-create the topic when a regex consumer connects.

[jiazhai]

In Pattern topics consumer's super class MultiTopicsConsumerImpl, there is a method to unsubscribe a given topic in the list:

CompletableFuture<Void> unsubscribeAsync(String topicName)

You may need first unsub a topic, then delete it.

[demo-client]

@sijie , thank you. Looking forward to a fix.

@jiazhai - Not sure if you are suggesting that the consumers do this; or are you suggesting that this should be done internally in the Pulsar Platform. Please do note that it would be infeasible for the consumers to do this, because the consumers listening to a regex-patten based topic do not event KNOW that a particular topic is GOING TO be deleted. This would be done by some other process based on various conditions.

[sijie]

@jiazhai I agreed with @anandsinha . In a regex subscription mode, the subscribers (consumers) usually don't know which topic to delete. So I think Pulsar should provide a mechanism to prevent a topic being re-created by a regex consumer.

[jiazhai]

@anandsinha，Yes, from my view, it is better to unsub the topic first, then do a clean topic delete, instead of delete topic by force. user could get all the subscriptions information from topics stats/stats-internal.

As @sijie mentioned, by default, topics will be auto-created if consumer/producer get reconnect. This is not in the scope of "regex subscription", even a consumer that subscribed on a dedicated topic will also re-create topic once it is deleted. If user don't want to see the topic again, he need first un-sub all consumers, and close all producers.

[demo-client]

@jiazhai - In a regex subscription mode, the subscribers (consumers) usually don't know which topic to delete. So, they will not be able to unsubscribe.

Pulsar should provide a mechanism to prevent a topic being re-created by a regex consumer.

[codelipenghui]

I think #5230 is fixed this issue, @anandsinha You can test it on the master branch since #5230 have not merged to 2.4.2 because wire protocol changes.
BTW, I checked your reproduce steps, is the producer closed or just stop publish messages? If just stop publish message, the producer may cause topic auto creation.

[sijie]

Closed this issue since it is already implemented in #6077. Feel free to reopen it if there is a problem.