feat: add database cleanup logic, runs after each request #875
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kangmingtay
requested changes
Dec 29, 2022
hf
force-pushed
the
hf/add-database-cleanup
branch
from
1e98ed5
to
72be296
Compare
J0
reviewed
Mar 1, 2023
J0
reviewed
Mar 1, 2023
J0
reviewed
Apr 18, 2023
hf
force-pushed
the
hf/add-database-cleanup
branch
from
7bb4877
to
0729bf6
Compare
kangmingtay
approved these changes
May 9, 2023
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
Just finished testing, it seems to be working quite well though I needed to add one more index. But GitHub is broken again so I'll push when it comes back. |
hf
force-pushed
the
hf/add-database-cleanup
branch
from
0729bf6
to
0c939dc
Compare
Just finished testing, it seems to be working quite well though I needed to add one more index. |
|
🎉 This PR is included in version 2.66.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Merged
J0
added a commit
that referenced
this pull request
May 23, 2023
## What kind of change does this PR introduce? This PR extends #875 to clean up MFA challenges as well so that they don't clog the database. ## How this was tested set `GOTRUE_DB_CLEANUP_ENABLED = true` 1. Sign up locally 2. Enroll a factor 3. `ab -p testfileforab -T application/json -H 'Authorization: Bearer <token>' -c 10 -n 100 http://localhost:9999/factors/0bca5d9c-157a-4a15-890c-2ad33415b4f3/challenge` 4. `update auth.mfa_challenges set created_at = created_at - interval '48 hours';` 5. Make about 7 requests to ensure there's a cleanup performed --------- Co-authored-by: joel@joellee.org <joel@joellee.org>
2 tasks
uxodb
pushed a commit
to uxodb/auth
that referenced
this pull request
Nov 13, 2024
Certain database entities such as refresh tokens and sessions pile up though normal operation without being cleaned up. This PR attempts to solve the problem by using a `models.Cleanup` function which takes care of these entities. The cleanup runs after each request on non-idempotent HTTP methods (POST, PUT, DELETE, PATCH). It uses fast deletes and updates using [`FOR UPDATE SKIP LOCKED`](https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE) so that deletes don't wait for other transactions to complete. It runs after each request as this model scales better than a background job that runs periodically as it is using resources only when the API is being used externally, making database use proportional to work performed. Rows are deleted about 24-72 hours after they have expired to aid in debugging if ever necessary.
uxodb
pushed a commit
to uxodb/auth
that referenced
this pull request
Nov 13, 2024
## What kind of change does this PR introduce? This PR extends supabase#875 to clean up MFA challenges as well so that they don't clog the database. ## How this was tested set `GOTRUE_DB_CLEANUP_ENABLED = true` 1. Sign up locally 2. Enroll a factor 3. `ab -p testfileforab -T application/json -H 'Authorization: Bearer <token>' -c 10 -n 100 http://localhost:9999/factors/0bca5d9c-157a-4a15-890c-2ad33415b4f3/challenge` 4. `update auth.mfa_challenges set created_at = created_at - interval '48 hours';` 5. Make about 7 requests to ensure there's a cleanup performed --------- Co-authored-by: joel@joellee.org <joel@joellee.org>
LashaJini
pushed a commit
to LashaJini/auth
that referenced
this pull request
Nov 13, 2024
Certain database entities such as refresh tokens and sessions pile up though normal operation without being cleaned up. This PR attempts to solve the problem by using a `models.Cleanup` function which takes care of these entities. The cleanup runs after each request on non-idempotent HTTP methods (POST, PUT, DELETE, PATCH). It uses fast deletes and updates using [`FOR UPDATE SKIP LOCKED`](https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE) so that deletes don't wait for other transactions to complete. It runs after each request as this model scales better than a background job that runs periodically as it is using resources only when the API is being used externally, making database use proportional to work performed. Rows are deleted about 24-72 hours after they have expired to aid in debugging if ever necessary.
LashaJini
pushed a commit
to LashaJini/auth
that referenced
this pull request
Nov 13, 2024
## What kind of change does this PR introduce? This PR extends supabase#875 to clean up MFA challenges as well so that they don't clog the database. ## How this was tested set `GOTRUE_DB_CLEANUP_ENABLED = true` 1. Sign up locally 2. Enroll a factor 3. `ab -p testfileforab -T application/json -H 'Authorization: Bearer <token>' -c 10 -n 100 http://localhost:9999/factors/0bca5d9c-157a-4a15-890c-2ad33415b4f3/challenge` 4. `update auth.mfa_challenges set created_at = created_at - interval '48 hours';` 5. Make about 7 requests to ensure there's a cleanup performed --------- Co-authored-by: joel@joellee.org <joel@joellee.org>
LashaJini
pushed a commit
to LashaJini/auth
that referenced
this pull request
Nov 15, 2024
Certain database entities such as refresh tokens and sessions pile up though normal operation without being cleaned up. This PR attempts to solve the problem by using a `models.Cleanup` function which takes care of these entities. The cleanup runs after each request on non-idempotent HTTP methods (POST, PUT, DELETE, PATCH). It uses fast deletes and updates using [`FOR UPDATE SKIP LOCKED`](https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE) so that deletes don't wait for other transactions to complete. It runs after each request as this model scales better than a background job that runs periodically as it is using resources only when the API is being used externally, making database use proportional to work performed. Rows are deleted about 24-72 hours after they have expired to aid in debugging if ever necessary.
LashaJini
pushed a commit
to LashaJini/auth
that referenced
this pull request
Nov 15, 2024
## What kind of change does this PR introduce? This PR extends supabase#875 to clean up MFA challenges as well so that they don't clog the database. ## How this was tested set `GOTRUE_DB_CLEANUP_ENABLED = true` 1. Sign up locally 2. Enroll a factor 3. `ab -p testfileforab -T application/json -H 'Authorization: Bearer <token>' -c 10 -n 100 http://localhost:9999/factors/0bca5d9c-157a-4a15-890c-2ad33415b4f3/challenge` 4. `update auth.mfa_challenges set created_at = created_at - interval '48 hours';` 5. Make about 7 requests to ensure there's a cleanup performed --------- Co-authored-by: joel@joellee.org <joel@joellee.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Certain database entities such as refresh tokens and sessions pile up though normal operation without being cleaned up. This PR attempts to solve the problem by using a
models.Cleanupfunction which takes care of these entities.The cleanup runs after each request on non-idempotent HTTP methods (POST, PUT, DELETE, PATCH). It uses fast deletes and updates using
FOR UPDATE SKIP LOCKEDso that deletes don't wait for other transactions to complete.It runs after each request as this model scales better than a background job that runs periodically as it is using resources only when the API is being used externally, making database use proportional to work performed.
Rows are deleted about 24-72 hours after they have expired to aid in debugging if ever necessary.