-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Spring Data Redis indexes not cleaned up when setting TTLs [DATAREDIS-748] #1299
Comments
Mark Paluch commented Can you provide more details on your setup, how you tested? It's essential that your application is running so it receives expired events for secondary index cleanup |
Mark Paluch commented Ping! Any update? |
Rajesh A commented Apologies for the absence, I have added some detail but might be missing something. Is it expected that an application is responsible to perform cleanup by subscribing to keyspace notifications and remove indices and references event by event? If that is the case, this would not be a bug. |
Rajesh A commented Updated bug with some additional information and a followup question |
Mark Paluch commented It's expected that Spring Data Redis subscribes to keyspace notifications (which means that you need to have your application running) but you need to enable it:
See also: https://docs.spring.io/spring-data/redis/docs/current/reference/html/#redis.repositories.expirations |
Rajesh A commented From the link: Are you saying that by enabling keyspace-notifications on the redis server and enabling keyspaceEvent listener within the spring application, Spring will self-handle the expiration events and delete the respective indexes? If this is true, I certainly do not see this happen on Spring Boot - 1.5.9.RELEASE version. FWIW I did follow step 1 and enabled keyspace notification on Amazon Elasticache. Then in my application, I have a listener subscribed to a pattern which is capturing events as it receives, and then act upon the keys that we know have indexes and delete certain keys from sets as no longer necessary |
Mark Paluch commented Yes, you need to enable keyspace events. Please pay attention to exceptions during startup. ElastiCache has some quirks (e.g. I need to investigate about Spring Boot and the cleanup issue you mentioned |
Mark Paluch commented I wasn't able to reproduce the issue. See https://gist.github.com/mp911de/581151f6394607998371d9f0910a7ac8. I think it makes only sense to follow up this issue if you can provide an example that reproduces the issue |
Rajesh A commented In the gist, you perform a save() with a TTL set. I see events arrive when we provide a TTL value. After 10 seconds, did you check if the :idx key(s) were removed too? |
Mark Paluch commented Indeed, everything works as it's expected to work |
Rajesh A opened DATAREDIS-748 and commented
When using the Spring-data-redis library to manage repositories, Spring creates new keys and sets under the hood to facilitate various type of key lookups. I noticed that when we set TTLs on the same objects (as opposed to a straight delete), these additional keys (indexes) and set members don't get deleted.
This has caused us to run out of disk space on the redis server as the associated keys never get removed.
More details:
In my setup, lets say I have a PersonRepository storing a Person class. The Person class (hash is "person") has say a bunch of fields, of note - name (identifier), email (indexed) & ttl (default value 0). Now lets say the first time the person object gets saved, the TTL is not set as expected as its 0. A bunch of other objects get created and updated, like person:johnsmith:idx etc.
Now when I go ahead and update the object by reading it (say using repo.findOne()), updated the ttl value to something and saved - the TTL gets set as expected. At this point, once the key expires, the related indices and key references do not get removed. If instead of updating (the TTL) on the object, we went ahead and straight away deleted, I believe the references get removed
Issue Links:
("is duplicated by")
The text was updated successfully, but these errors were encountered: