Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: no full vacuum for various reports tables #5120

Merged
merged 9 commits into from
Sep 27, 2024

Conversation

Sidddddarth
Copy link
Member

@Sidddddarth Sidddddarth commented Sep 20, 2024

Description

  1. Using vacuum instead of vacuum full for various reports tables. This only takes lenient locking and doesn't disrupt other reads/writes to the tables. And while it doesn't free up disk space, it makes the space available for newer tuples added to the table.
  2. Full vacuum on startup
  3. vacuum threshold based on number of deleted rows.

Having done all this, the table size might still grow, but not at the rate when there was no vacuum at all. Each server startup would help free up disk space.

Linear Ticket

Resolves PIPE-1517

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@Sidddddarth
Copy link
Member Author

Sidddddarth commented Sep 20, 2024

went with 100k(threshold number of deleted rows) as a placeholder. But the peak rate is around 5k/s
I think 1.5M should be a good figure(5 minutes at 5k/s) wdyt? @rudderlabs/server_team
image

@Sidddddarth
Copy link
Member Author

error details reports at a higher rate
image

Copy link
Collaborator

@fracasula fracasula left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we potentially have this as a feature flag so that we can specify the type of vacuum we want?

Copy link

codecov bot commented Sep 23, 2024

Codecov Report

Attention: Patch coverage is 27.84091% with 127 lines in your changes missing coverage. Please review.

Project coverage is 73.03%. Comparing base (0ada31f) to head (042e88a).
Report is 14 commits behind head on master.

Files with missing lines Patch % Lines
enterprise/reporting/error_reporting.go 1.56% 63 Missing ⚠️
enterprise/reporting/reporting.go 29.78% 28 Missing and 5 partials ⚠️
enterprise/reporting/flusher/flusher.go 54.23% 22 Missing and 5 partials ⚠️
app/apphandlers/embeddedAppHandler.go 33.33% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5120      +/-   ##
==========================================
- Coverage   74.80%   73.03%   -1.77%     
==========================================
  Files         435      437       +2     
  Lines       50436    50898     +462     
==========================================
- Hits        37728    37173     -555     
- Misses      10241    11299    +1058     
+ Partials     2467     2426      -41     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

enterprise/reporting/error_reporting.go Outdated Show resolved Hide resolved
enterprise/reporting/error_reporting.go Outdated Show resolved Hide resolved
enterprise/reporting/reporting.go Show resolved Hide resolved
enterprise/reporting/error_reporting.go Outdated Show resolved Hide resolved
enterprise/reporting/flusher/flusher.go Outdated Show resolved Hide resolved
enterprise/reporting/reporting.go Outdated Show resolved Hide resolved
@Sidddddarth Sidddddarth merged commit 6e8b03e into master Sep 27, 2024
56 checks passed
@Sidddddarth Sidddddarth deleted the chore.reportsVacuum_noFull branch September 27, 2024 10:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants