Stream server side filtering

[kjnilsson]

This PR implements an approach to server side consumer filtering of streams to allow consumers that are only interested in a subset of the data in a given stream to reduce the amount of unwanted data that is sent to the client application.

The filtering is done at the chunk level and is probabilistic. False positives are possible but false negatives are not. I.e. a consumer would still need to do client side filtering in addition to the broker side filtering. This can be a client feature or done per application.

The broker will never interpret or filter on individual messages. It can only be done at the chunk level.

To do the filtering the stream will for each chunk (batch of messages) written calculate a bloom filter over all the messages that include a filter value. This bloom filter data will be written immediately after the header and if present and if the consumer includes a filter value it will calculate a bloom filter hash for the consumer filter and match this against the chunk filter and only deliver the chunk to the client if it is a match.

The per chunk bloom filter size can be varied between 16 and 255 bytes depending on the expected number of overlapping filter values a stream will contain. The osiris default is 16 bytes (the smallest size).

A 16 byte bloom filter can filter around 50 different filter values with a false positive probability of ~30%.
A 255 byte bloom filter can filter around 800 values at the same false positive probability.

It is worth noting that a chunk without any messages that included a filter value will not be delivered to a consumer that includes a filter. Hence a consumer that is interested in any messages that may not contain a filter value needs to also not include a filter value. This is an important consideration for existing stream after an upgrade. There is no way to retro fit a filter value onto existing data.

• Osiris API and implementation draft
• Osiris finalisation
• Stream bloom filter size configuration
• Stream protocol consumer API
• Stream protocol publisher API
• AMQP Publisher API
  - Perhaps using a custom header x-stream-filter-value
• (AMQP consumer API)
  - Perhaps using a custom consumer arg x-stream-filter
• Feature flag