Add support for Redis caching #354

dgzlopes · 2020-11-19T12:11:53Z

What this PR does:
This PR adds support for Redis caching on Tempodb.

TODO:

Add basic client/support/config
Add sane defaults and more config params
Add tests
(probably) Cleanup

Which issue(s) this PR fixes:
Fixes #

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

dgzlopes · 2020-11-19T12:19:55Z

Created a WIP PR with a working version... but I've a lot of things todo for this to be usable and reviewable :)

- Add Redis client - Add Redis support - Add Redis file Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

annanay25

I've left some comments @dgzlopes, thanks for your work on the redis support. Also, sorry for a late review to go a different route, but feel free to reach out on DM to discuss further :)

tempodb/backend/redis/cache.go

docs/tempo/website/configuration/_index.md

annanay25 · 2020-11-24T08:22:28Z

tempodb/backend/redis/cache.go

@@ -0,0 +1,147 @@
+package redis


There seems to be a ton of code that's pretty much the same as memcached/cache.go .. This can get tough to maintain. I would suggest we split out the cache section into a separate module and pass it a Cache interface which looks something like this

type Cache interface { func get(ctx context.Context, key string) []byte func set(ctx context.Context, key string, val []byte) }

The different cache backends can then implement this small interface instead of the entire backend.Reader / backend.Writer interfaces. It will also help to keep all the tests in one place.

diskcache should live inside this separate module too? Probably not, I suppose 😕

diskcache should also live there, but if that turns out to be tough we can always address it in a different PR :)

annanay25 · 2020-11-24T08:22:56Z

tempodb/tempodb.go

+	if cfg.Redis != nil {
+		r, w, err = redis.New(r, w, cfg.Redis, logger)
+
+		if err != nil {
+			return nil, nil, nil, err
+		}
+	}
+


This can also be addressed by the other comment ^

Also, the config section for cache is getting a little untidy, can we add a new field like we have for backend (Line 128) which indicates which cache type we are trying to use? And then we can add these in a switch case like above.

tempodb/backend/redis/cache.go

- Add Redis client - Add Redis support - Add Redis file Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

annanay25 · 2020-11-25T13:09:24Z

tempodb/backend/redis/cache_test.go

+				object:  tt.readerObject,
+			}
+			mockW := &mockWriter{}
+			mr, _ := miniredis.Run()


A simple map cache can be implemented like in memcached tests but I saw Cortex is using this too, so OK.

annanay25

LGTM, thanks for your work @dgzlopes! 🎉

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

achatterjee-grafana · 2020-11-25T15:01:41Z

One comment for the docs, even though this PR is merged.
In the redis.md file, a title is missing. Also, can we use sentence case instead of the lower case? We can work on this together on a new PR. Thanks!

dgzlopes · 2020-11-25T20:28:45Z

Hi @achatterjee-grafana!

Maybe I'm missing something... But I see a title here: https://github.com/grafana/tempo/blob/master/docs/tempo/website/configuration/redis.md. I've to add it to some other place?

I'm happy to move everything to sentence case. I used lower case... Because I followed what we have for S3 -> https://github.com/grafana/tempo/blob/master/docs/tempo/website/configuration/s3.md 😄

dgzlopes force-pushed the add-redis-support-tempodb branch from 9515d79 to 52f28b9 Compare November 23, 2020 16:50

dgzlopes marked this pull request as ready for review November 23, 2020 17:07

dgzlopes requested review from annanay25 and joe-elliott as code owners November 23, 2020 17:07

dgzlopes changed the title ~~[WIP] Add support for Redis caching~~ Add support for Redis caching Nov 23, 2020

dgzlopes added 5 commits November 24, 2020 10:12

Add Redis caching

089396c

- Add Redis client - Add Redis support - Add Redis file Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Add Redis to docs

59d921a

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Add to changelog

69b1f59

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Add ttl

3cdc181

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Add Redis tests

3858ed2

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

dgzlopes force-pushed the add-redis-support-tempodb branch from 52f28b9 to 3858ed2 Compare November 24, 2020 09:13

annanay25 reviewed Nov 24, 2020

View reviewed changes

dgzlopes changed the title ~~Add support for Redis caching~~ [WIP] Add support for Redis caching Nov 24, 2020

dgzlopes added 9 commits November 24, 2020 22:08

Add Redis caching

ab372e1

- Add Redis client - Add Redis support - Add Redis file Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Add Redis to docs

9732281

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Add to changelog

a144a9c

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Add ttl

3e7e651

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Add Redis tests

a3b88c1

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Remove typeBloom and typeIndex

4e682db

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Remove strs

8be7e14

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Add some more redis commands

81ea1e9

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Add Redis section

212ce13

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

dgzlopes force-pushed the add-redis-support-tempodb branch from 3858ed2 to 212ce13 Compare November 24, 2020 22:21

dgzlopes requested a review from achatterjee-grafana as a code owner November 24, 2020 22:21

dgzlopes added 2 commits November 25, 2020 09:41

Change docs and intex/bloom keys

467de3a

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

Fix missing :

6086822

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

This was referenced Nov 25, 2020

[docs] Enhance storage section in docs to describe all storage backends & options #372

Closed

Refactor cache section in tempodb/backend to use Cache interface #373

Closed

annanay25 reviewed Nov 25, 2020

View reviewed changes

annanay25 changed the title ~~[WIP] Add support for Redis caching~~ Add support for Redis caching Nov 25, 2020

annanay25 approved these changes Nov 25, 2020

View reviewed changes

Add note about redis support

98b7362

Signed-off-by: Daniel González Lopes <danielgonzalezlopes@gmail.com>

annanay25 merged commit c20839c into grafana:master Nov 25, 2020

dgzlopes deleted the add-redis-support-tempodb branch November 25, 2020 14:00

dgzlopes mentioned this pull request Nov 30, 2020

Add Redis caching on Tempodb #337

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Redis caching #354

Add support for Redis caching #354

dgzlopes commented Nov 19, 2020 •

edited

Loading

dgzlopes commented Nov 19, 2020 •

edited

Loading

annanay25 left a comment

annanay25 Nov 24, 2020

dgzlopes Nov 24, 2020 •

edited

Loading

annanay25 Nov 24, 2020

annanay25 Nov 24, 2020

annanay25 Nov 25, 2020 •

edited

Loading

annanay25 left a comment

achatterjee-grafana commented Nov 25, 2020

dgzlopes commented Nov 25, 2020 •

edited

Loading

Add support for Redis caching #354

Add support for Redis caching #354

Conversation

dgzlopes commented Nov 19, 2020 • edited Loading

dgzlopes commented Nov 19, 2020 • edited Loading

annanay25 left a comment

Choose a reason for hiding this comment

annanay25 Nov 24, 2020

Choose a reason for hiding this comment

dgzlopes Nov 24, 2020 • edited Loading

Choose a reason for hiding this comment

annanay25 Nov 24, 2020

Choose a reason for hiding this comment

annanay25 Nov 24, 2020

Choose a reason for hiding this comment

annanay25 Nov 25, 2020 • edited Loading

Choose a reason for hiding this comment

annanay25 left a comment

Choose a reason for hiding this comment

achatterjee-grafana commented Nov 25, 2020

dgzlopes commented Nov 25, 2020 • edited Loading

dgzlopes commented Nov 19, 2020 •

edited

Loading

dgzlopes commented Nov 19, 2020 •

edited

Loading

dgzlopes Nov 24, 2020 •

edited

Loading

annanay25 Nov 25, 2020 •

edited

Loading

dgzlopes commented Nov 25, 2020 •

edited

Loading