This repository has been archived by the owner on Apr 1, 2024. It is now read-only.
Policy/config to limit the message size at topic level #1723
Assignees
Labels
Comments
|
I will work on it |
codelipenghui
referenced
this issue
in apache/pulsar
Nov 30, 2020
fix https://github.com/streamnative/pulsar/issues/1723 ### Motivation The current policy to control the size of the message is at the broker level(maxMessageSize). It becomes easier to plan resource quotas for client allocation if the max message size pushed can be given at the topic level. ### Modifications Now the broker-level `maxMessageSize` is returned by the broker to the client, when the broker handles `handleConnected`. The client will cached `maxMessageSize` locally. An exception will be thrown if it exceeds the limit. Topic-level cannot be implemented like this, because: 1) When `handleConnected`, the command received by the broker does not contain specific topic information, so it is not known which topic policy to return to the client. 2) The client cannot cache topic-level policy. Unlike the broker-level policy, which will not change, the topic-level policy will change dynamically, which will involve cache consistency issues. I think the best way to handle this is to let the broker determine whether it exceeds the limit, and return an exception if it exceeds the limit, and handle the exception by the client's `handleSendError`. ### Verifying this change TopicPoliciesTest.java
github-actions bot
referenced
this issue
in apache/pulsar
Nov 30, 2020
fix https://github.com/streamnative/pulsar/issues/1723 ### Motivation The current policy to control the size of the message is at the broker level(maxMessageSize). It becomes easier to plan resource quotas for client allocation if the max message size pushed can be given at the topic level. ### Modifications Now the broker-level `maxMessageSize` is returned by the broker to the client, when the broker handles `handleConnected`. The client will cached `maxMessageSize` locally. An exception will be thrown if it exceeds the limit. Topic-level cannot be implemented like this, because: 1) When `handleConnected`, the command received by the broker does not contain specific topic information, so it is not known which topic policy to return to the client. 2) The client cannot cache topic-level policy. Unlike the broker-level policy, which will not change, the topic-level policy will change dynamically, which will involve cache consistency issues. I think the best way to handle this is to let the broker determine whether it exceeds the limit, and return an exception if it exceeds the limit, and handle the exception by the client's `handleSendError`. ### Verifying this change TopicPoliciesTest.java (cherry picked from commit 9c28378)
codelipenghui
referenced
this issue
in apache/pulsar
Dec 2, 2020
fix https://github.com/streamnative/pulsar/issues/1723 ### Motivation The current policy to control the size of the message is at the broker level(maxMessageSize). It becomes easier to plan resource quotas for client allocation if the max message size pushed can be given at the topic level. ### Modifications Now the broker-level `maxMessageSize` is returned by the broker to the client, when the broker handles `handleConnected`. The client will cached `maxMessageSize` locally. An exception will be thrown if it exceeds the limit. Topic-level cannot be implemented like this, because: 1) When `handleConnected`, the command received by the broker does not contain specific topic information, so it is not known which topic policy to return to the client. 2) The client cannot cache topic-level policy. Unlike the broker-level policy, which will not change, the topic-level policy will change dynamically, which will involve cache consistency issues. I think the best way to handle this is to let the broker determine whether it exceeds the limit, and return an exception if it exceeds the limit, and handle the exception by the client's `handleSendError`. ### Verifying this change TopicPoliciesTest.java (cherry picked from commit 9c28378) Co-authored-by: feynmanlin <feynmanlin@tencent.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Is your feature request related to a problem? Please describe.
The current policy to control the size of the message is at the broker level(maxMessageSize). It becomes easier to plan resource quotas for client allocation if the max message size pushed can be given at the topic level.
Describe the solution you'd like
A config variable/policy to control the size of the message at the topic level.
Describe alternatives you've considered
N/A
Additional context
N/A
The text was updated successfully, but these errors were encountered: