Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Caching #246

Merged
merged 52 commits into from
Oct 21, 2021
Merged

Caching #246

merged 52 commits into from
Oct 21, 2021

Conversation

rathorevaibhav
Copy link
Member

@rathorevaibhav rathorevaibhav commented Sep 16, 2021

Fixes #235, #224

Summary

  1. Using Django cache with Django-Redis
  2. A central cache file that helps with calculating cache keys and invalidating cache
  3. Documentation update for caching

Caching Outcome

Screenshot from local docker
Right: Plio with 12 items and querying Postgres DB (no cache)
Left: Same Plio with 12 items and querying Redis (cached)
image

Test Plan

  • Wrote tests
  • Tested locally
  • Tested on staging
  • Tested on production
  • If changes in DB, update DB schema and BigQuery (staging and prod)

@codecov
Copy link

codecov bot commented Sep 16, 2021

Codecov Report

Merging #246 (75a6247) into master (97515f0) will increase coverage by 0.79%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #246      +/-   ##
==========================================
+ Coverage   96.07%   96.86%   +0.79%     
==========================================
  Files         133      143      +10     
  Lines        2496     2680     +184     
==========================================
+ Hits         2398     2596     +198     
+ Misses         98       84      -14     
Impacted Files Coverage Δ
users/views.py 95.31% <ø> (+6.49%) ⬆️
organizations/__init__.py 100.00% <100.00%> (ø)
organizations/apps.py 100.00% <100.00%> (ø)
organizations/signals.py 100.00% <100.00%> (ø)
organizations/tests.py 100.00% <100.00%> (ø)
plio/__init__.py 100.00% <100.00%> (ø)
plio/apps.py 100.00% <100.00%> (ø)
plio/cache.py 100.00% <100.00%> (ø)
plio/serializers.py 100.00% <100.00%> (+2.22%) ⬆️
plio/settings.py 88.23% <100.00%> (+0.14%) ⬆️
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 97515f0...75a6247. Read the comment docs.

Copy link
Contributor

@pritamps pritamps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some minor comments.

Will the cache automatically be updated when a plio is updated? because if not, a GET after an update will return the old value, no?

Same for deletion. When a plio is deleted, will cache automatically delete it as well? Maybe Django handles all this automatically, just curious :)

plio/cache.py Outdated Show resolved Hide resolved
plio/cache.py Show resolved Hide resolved
requirements.txt Outdated Show resolved Hide resolved
docs/CACHING.md Outdated Show resolved Hide resolved
docs/CACHING.md Outdated Show resolved Hide resolved
docs/CACHING.md Outdated Show resolved Hide resolved
@dalmia dalmia temporarily deployed to Staging October 19, 2021 23:30 Inactive
organizations/tests.py Outdated Show resolved Hide resolved

@receiver([post_save, post_delete], sender=User)
def user_update_cache(sender, instance, created, raw, **kwargs):
invalidate_cache_for_instance(instance)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since a session also returns the user data completely, ideally, this should invalidate the cache for session instances too. however, we are not caching sessions right now as we are not making a request for sessions. but if we end up doing that in the future, is there a better way to remember that we need to invalidate its cache (when we add support for session caches) rather than just relying on our ability to recollect this then?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But caching sessions doesn't make sense right?
I don't think we ever make a GET for session right?
Every time someone comes to a plio, we create a new session

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we end up adding more details to the dashboard, we'll be fetching the sessions then na?


@receiver([post_save, post_delete], sender=OrganizationUser)
def organization_user_update_cache(sender, instance, **kwargs):
invalidate_cache_for_instance(instance.user)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar comment as above - is there a better way to be sure that we won't forget updating the relevant cache instances for whom we've not added caching support right now (and all the other objects connected to them)? Ideally, there could be like a graph of objects, each connected to the ones which it relies on (based on a FOREIGN_KEY relation) and the one which have been cached are marked in, say, green, with the remaining ones marked in, say, red. Is there a tool that can do this mapping for us?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think probably that's why caching is one of the most difficult things to implement :)

Yes, I agree, we need some sort of UI representation to easy understand connected tables. An ER diagram will help definitely. I'll share it.

docs/CACHING.md Outdated Show resolved Hide resolved
docs/CACHING.md Outdated Show resolved Hide resolved
Co-authored-by: Aman Dalmia <amandalmia18@gmail.com>
organizations/tests.py Show resolved Hide resolved
organizations/tests.py Outdated Show resolved Hide resolved
plio/__init__.py Show resolved Hide resolved
plio/tests.py Show resolved Hide resolved

@receiver([post_save, post_delete], sender=User)
def user_update_cache(sender, instance, created, raw, **kwargs):
invalidate_cache_for_instance(instance)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But caching sessions doesn't make sense right?
I don't think we ever make a GET for session right?
Every time someone comes to a plio, we create a new session

users/signals.py Show resolved Hide resolved
docs/CACHING.md Outdated Show resolved Hide resolved
Co-authored-by: Aman Dalmia <amandalmia18@gmail.com>
@rathorevaibhav rathorevaibhav temporarily deployed to Staging October 21, 2021 05:42 Inactive
@dalmia dalmia merged commit fb12485 into master Oct 21, 2021
@dalmia dalmia deleted the feature/235-caching branch October 21, 2021 07:17
@dalmia dalmia modified the milestones: v0.4.1, 0.4 Oct 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable Django cache
5 participants