Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using cuckoo filter for new metric detection instead of cache #590

Merged
merged 5 commits into from
Jun 24, 2024

Conversation

deniszh
Copy link
Member

@deniszh deniszh commented Jun 24, 2024

Context:: Go-Carbon supports real-time indexing in trie index in Carbonserver. If the realtime-index parameter in config is > 0, it creates a special channel for a new metric with that size. Then cache.go populates that channel if the metric is missed in the cache. Then Carbonserver consumes this channel and populates the trie index from it during a file scan, creating index entries for the metric even if the whisper file doesn't exist.

Problem: Looks like the cache is not a good predictor for new metrics. When the cache is empty and there is a lot of incoming traffic, the file scan thread is blocked for a long time and the scan never finishes.

Solution: We can use a simpler structure than the cache (a map) to detect previously seen metrics. We can use bloom filters, which are good for this and have limited space. I use cuckoo filters, which are faster and support deletion.

I added cuckoo filter support to cache.go with tests. I also added support for the bloom-size parameter in the cache config. If > 0, the cache will use a bloom filter of a specified size to detect new metrics. I'm also doing deletion from the filter if a metric leaves the cache. I'm not sure if we need this, but it might help in case of long uptime.

@deniszh deniszh merged commit c9ef183 into go-graphite:master Jun 24, 2024
9 checks passed
@deniszh deniszh deleted the DZ-cuckoo-newchan branch June 24, 2024 11:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants