Avoid clearing entire cache on transient errors

[remko]

It seems that google-cloud-ndb now clears the entire global cache (even keys not related to NDB) when any transient error occurs. Since we get transient errors regularly, and since we store other data in our cache, this is problematic for us. I realize a way to avoid this is to use a second Redis instance, but I'd like to avoid this.

Should the entire cache always be cleared on any transient error? I'm guessing limiting it to NDB keys only is too inefficient, but do transient read errors also need to clear a cache? (since I don't think that this can leave the cache in an inconsistent state?)

[chrisrossi]

Hi @remko ,

I can see how that would be annoying. Can you use the db parameter when getting your Redis connection in order to have NDB use a different db from your other applications? If using RedisCache.from_environment you can just tack the db, an integer number, to the end of your Redis url:

export REDIS_CACHE_URL=redis://localhost:6379/1

Where 1 is the db. (The default is 0.)

I think that might give you some segmentation where even if NDB has to nuke all of its own data, it doesn't nuke your other applications' data.

[remko]

@chrisrossi How did I not know about this? That indeed sounds like it would solve the most annoying part of my problem, thanks a lot!

It's still a fact that Google-managed Redis instances have a scheduled weekly time where they have many failing connections (at least from GAE), so limiting the number of nukes (e.g. only on write errors, as those would potentially leave the cache in an inconsistent state) would be handy (if this would be correct, of course)

[chrisrossi]

I think the idea is that if there is a known issue with the cache, then it calls into question the validity of the entire contents of the cache. Clearing the whole thing, whether the problem is detected during a read or a write, eliminates any risk of stale data. It's likely that's overkill, but can be hard to guarantee integrity otherwise. We're looking at it now.

[chrisrossi]

Hi @remko ,

Although we haven't specifically addressed this issue, we have with the latest release, shipped a fix that adds retry to cache operations, so it may be that you will be less likely to run into this issue now. If you could use the latest version and let us know how it goes for you, that would really helpful.

Thank you!

[remko]

@chrisrossi Thanks! I will still need to separate our databases, so I will give it a shot once this is done too.