[improve][broker] Unblock stuck Key_Shared subscription after consumer reconnect

[nborisov]

Fixes #21199 and #15705

Motivation

Current delivery mechanism blocks recently joined consumers from getting messages until all the previously delivered but not acknowledged messages are either acknowledged by the consumer which received the messages or the consumer disconnected

This is implementation trade off which allows not to track all the sent messages. This approach allows to reduce memory consumption and increase performance

The same time such implementation leads to a situation when a single failed message could lead to all the consumers blockage (described at apache#21199)

There is a setting allowOutOfOrderDelivery which allows to mitigate the described issue but leads to ordering guarantees loss

For some scenarios both consumption stuck and out of order delivery is not an option. The same time it could be ok to have more memory consumption and some minor performance degradation

Modifications

1. Sent but not acknowledged messages tracking added
2. Messages tracking setting added

Tracking of delivered but not acknowledged messages allows not to send them to recently joined consumers and the same time preserve ordering

Making the new functionality switchable allows to preserve the current optimized memory consumption and performance for all the users with default settings. The same time users who is ready to memory and performance trade off getting opportunity to solve both consumers stuck problem and out of order delivery

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

• Existing KeySharedSubscriptionTest extended with scenarios with new setting enabled
• Added integration test for verification of preserving ordering with the new setting enabled
• Added integration test for verification that recently joined consumers are not blocked when there are some unacknowledged messages delivered to existing consumers

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete