Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add filter rules for keys #898

Merged
merged 1 commit into from
Dec 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions docs/src/en/filter/filter.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,14 @@ outline: deep
RedisShake provides various built-in filter rules that users can choose from according to their needs.

## Filtering Keys
RedisShake supports filtering by key name prefixes and suffixes. You can set the following options in the configuration file, for example:
RedisShake supports filtering by key name, key name prefixes, and suffixes. You can set the following options in the configuration file, for example:
```toml
allow_key_prefix = ["user:", "product:"]
allow_key_suffix = [":active", ":valid"]
block_key_prefix = ["temp:", "cache:"]
block_key_suffix = [":tmp", ":old"]
allow_keys = ["user:1001", "product:2001"] # allowed key names
allow_key_prefix = ["user:", "product:"] # allowed key name prefixes
allow_key_suffix = [":active", ":valid"] # allowed key name suffixes
block_keys = ["temp:1001", "cache:2001"] # blocked key names
block_key_prefix = ["temp:", "cache:"] # blocked key name prefixes
block_key_suffix = [":tmp", ":old"] # blocked key name suffixes
```
If these options are not set, all keys are allowed by default.

Expand Down
12 changes: 7 additions & 5 deletions docs/src/zh/filter/filter.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,14 @@ outline: deep
RedisShake 提供了多种内置的过滤规则,用户可以根据需要选择合适的规则。

## 过滤 Key
RedisShake 支持通过键名前缀和后缀进行过滤。您可以在配置文件中设置以下选项,例如:
RedisShake 支持通过键名、键名前缀和后缀进行过滤。您可以在配置文件中设置以下选项,例如:
```toml
allow_key_prefix = ["user:", "product:"]
allow_key_suffix = [":active", ":valid"]
block_key_prefix = ["temp:", "cache:"]
block_key_suffix = [":tmp", ":old"]
allow_keys = ["user:1001", "product:2001"] # 允许的键名
allow_key_prefix = ["user:", "product:"] # 允许的键名前缀
allow_key_suffix = [":active", ":valid"] # 允许的键名后缀
block_keys = ["temp:1001", "cache:2001"] # 阻止的键名
block_key_prefix = ["temp:", "cache:"] # 阻止的键名前缀
block_key_suffix = [":tmp", ":old"] # 阻止的键名后缀
```
如果不设置这些选项,默认允许所有键。

Expand Down
2 changes: 2 additions & 0 deletions internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,10 @@ import (
)

type FilterOptions struct {
AllowKeys []string `mapstructure:"allow_keys" default:"[]"`
AllowKeyPrefix []string `mapstructure:"allow_key_prefix" default:"[]"`
AllowKeySuffix []string `mapstructure:"allow_key_suffix" default:"[]"`
BlockKeys []string `mapstructure:"block_keys" default:"[]"`
BlockKeyPrefix []string `mapstructure:"block_key_prefix" default:"[]"`
BlockKeySuffix []string `mapstructure:"block_key_suffix" default:"[]"`
AllowKeyRegex []string `mapstructure:"allow_key_regex" default:"[]"`
Expand Down
28 changes: 19 additions & 9 deletions internal/filter/filter.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ package filter
import (
"RedisShake/internal/config"
"RedisShake/internal/entry"
"log"
"RedisShake/internal/log"
"slices"
"strings"
"sync"
Expand Down Expand Up @@ -47,9 +47,9 @@ func Filter(e *entry.Entry) bool {
return false
} else {
// If we reach here, it means some keys are true and some are false
log.Printf("Error: Inconsistent filter results for entry with %d keys", len(e.Keys))
log.Printf("Passed keys: %v", passedKeys)
log.Printf("Filtered keys: %v", filteredKeys)
log.Infof("Error: Inconsistent filter results for entry with %d keys", len(e.Keys))
log.Infof("Passed keys: %v", passedKeys)
log.Infof("Filtered keys: %v", filteredKeys)
return false
}

Expand Down Expand Up @@ -97,10 +97,15 @@ func Filter(e *entry.Entry) bool {

// blockKeyFilter is block key? default false
func blockKeyFilter(key string) bool {
if len(config.Opt.Filter.BlockKeyRegex) == 0 && len(config.Opt.Filter.BlockKeyPrefix) == 0 &&
len(config.Opt.Filter.BlockKeySuffix) == 0 {
if len(config.Opt.Filter.BlockKeyRegex) == 0 &&
len(config.Opt.Filter.BlockKeyPrefix) == 0 &&
len(config.Opt.Filter.BlockKeySuffix) == 0 &&
len(config.Opt.Filter.BlockKeys) == 0 {
return false
}
if slices.Contains(config.Opt.Filter.BlockKeys, key) {
return true
}
if blockKeyMatch(config.Opt.Filter.BlockKeyRegex, key) {
return true
}
Expand All @@ -120,9 +125,14 @@ func blockKeyFilter(key string) bool {

// allowKeyFilter is allow key? default true
func allowKeyFilter(key string) bool {
// if all allow filter is empty. default is true
if len(config.Opt.Filter.AllowKeyRegex) == 0 && len(config.Opt.Filter.AllowKeyPrefix) == 0 &&
len(config.Opt.Filter.AllowKeySuffix) == 0 {
if len(config.Opt.Filter.AllowKeyRegex) == 0 &&
len(config.Opt.Filter.AllowKeyPrefix) == 0 &&
len(config.Opt.Filter.AllowKeySuffix) == 0 &&
len(config.Opt.Filter.AllowKeys) == 0 {
return true
}

if slices.Contains(config.Opt.Filter.AllowKeys, key) {
return true
}
// If the RE matches, there is no need to iterate over the others
Expand Down
6 changes: 5 additions & 1 deletion shake.toml
Original file line number Diff line number Diff line change
Expand Up @@ -43,22 +43,26 @@ buff_send = false # buffer send, default false. may be a sync delay whe
[filter]
# Allow keys with specific prefixes or suffixes
# Examples:
# allow_keys = ["user:1001", "product:2001"]
# allow_key_prefix = ["user:", "product:"]
# allow_key_suffix = [":active", ":valid"]
# allow A collection of keys containing 11-digit mobile phone numbers
# allow_key_regex = [":\\d{11}:"]
# Leave empty to allow all keys
allow_keys = []
allow_key_prefix = []
allow_key_suffix = []
allow_key_regex = []

# Block keys with specific prefixes or suffixes
# Examples:
# block_keys = ["temp:1001", "cache:2001"]
# block_key_prefix = ["temp:", "cache:"]
# block_key_suffix = [":tmp", ":old"]
# block test 11-digit mobile phone numbers keys
# block_key_regex = [":test:\\d{11}:"]
# Leave empty to block nothing
block_keys = []
block_key_prefix = []
block_key_suffix = []
block_key_regex = []
Expand Down Expand Up @@ -90,7 +94,7 @@ block_command_group = []

# Function for custom data processing
# For best practices and examples, visit:
# https://tair-opensource.github.io/RedisShake/zh/function/best_practices.html
# https://tair-opensource.github.io/RedisShake/zh/filter/function.html
function = ""

[advanced]
Expand Down
Loading