This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
When purging, don't de-delta state groups we're about to delete #2214
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
richvdh
force-pushed
the
rav/hurry_up_purge
branch
from
bcba88a
to
8e345ce
Compare
synapse/storage/events.py
Outdated
| for i in xrange(0, len(state_groups_to_delete), 100) | ||
| ] | ||
| for chunk in chunks: | ||
| for i in xrange(0, len(state_rows), 100): |
There was a problem hiding this comment.
It seems odd to dedupe state_groups_to_delete and then use the non-deduped list here
There was a problem hiding this comment.
the set is for easy lookup at line 2101 rather than deduping. We can't easily divide a set into chunks, hence the use of the list here. There shouldn't be any dups because the query has a GROUP BY.
synapse/storage/events.py
Outdated
| new_state_edges.extend(row["state_group"] for row in rows) | ||
| new_state_edges.extend( | ||
| row["state_group"] for row in rows | ||
| if row["state_group"] not in state_groups_to_delete |
There was a problem hiding this comment.
Can you comment this please? It took me a few moments to realise what was going on
There was a problem hiding this comment.
yup will do more comments
Try to make this clearer with more comments and some variable renames
|
@erikjohnston ptal |
psaavedra
added a commit
to psaavedra/synapse
that referenced
this pull request
May 19, 2017
Changes in synapse v0.21.0 (2017-05-18) ======================================= No changes since v0.21.0-rc3 Changes in synapse v0.21.0-rc3 (2017-05-17) =========================================== Features: * Add per user rate-limiting overrides (PR matrix-org#2208) * Add config option to limit maximum number of events requested by ``/sync`` and ``/messages`` (PR matrix-org#2221) Thanks to @psaavedra! Changes: * Various small performance fixes (PR matrix-org#2201, matrix-org#2202, matrix-org#2224, matrix-org#2226, matrix-org#2227, matrix-org#2228, matrix-org#2229) * Update username availability checker API (PR matrix-org#2209, matrix-org#2213) * When purging, don't de-delta state groups we're about to delete (PR matrix-org#2214) * Documentation to check synapse version (PR matrix-org#2215) Thanks to @hamber-dick! * Add an index to event_search to speed up purge history API (PR matrix-org#2218) Bug fixes: * Fix API to allow clients to upload one-time-keys with new sigs (PR matrix-org#2206) Changes in synapse v0.21.0-rc2 (2017-05-08) =========================================== Changes: * Always mark remotes as up if we receive a signed request from them (PR matrix-org#2190) Bug fixes: * Fix bug where users got pushed for rooms they had muted (PR matrix-org#2200) Changes in synapse v0.21.0-rc1 (2017-05-08) =========================================== Features: * Add username availability checker API (PR matrix-org#2183) * Add read marker API (PR matrix-org#2120) Changes: * Enable guest access for the 3pl/3pid APIs (PR matrix-org#1986) * Add setting to support TURN for guests (PR matrix-org#2011) * Various performance improvements (PR matrix-org#2075, matrix-org#2076, matrix-org#2080, matrix-org#2083, matrix-org#2108, matrix-org#2158, matrix-org#2176, matrix-org#2185) * Make synctl a bit more user friendly (PR matrix-org#2078, matrix-org#2127) Thanks @APwhitehat! * Replace HTTP replication with TCP replication (PR matrix-org#2082, matrix-org#2097, matrix-org#2098, matrix-org#2099, matrix-org#2103, matrix-org#2014, matrix-org#2016, matrix-org#2115, matrix-org#2116, matrix-org#2117) * Support authenticated SMTP (PR matrix-org#2102) Thanks @DanielDent! * Add a counter metric for successfully-sent transactions (PR matrix-org#2121) * Propagate errors sensibly from proxied IS requests (PR matrix-org#2147) * Add more granular event send metrics (PR matrix-org#2178) Bug fixes: * Fix nuke-room script to work with current schema (PR matrix-org#1927) Thanks @zuckschwerdt! * Fix db port script to not assume postgres tables are in the public schema (PR matrix-org#2024) Thanks @jerrykan! * Fix getting latest device IP for user with no devices (PR matrix-org#2118) * Fix rejection of invites to unreachable servers (PR matrix-org#2145) * Fix code for reporting old verify keys in synapse (PR matrix-org#2156) * Fix invite state to always include all events (PR matrix-org#2163) * Fix bug where synapse would always fetch state for any missing event (PR matrix-org#2170) * Fix a leak with timed out HTTP connections (PR matrix-org#2180) * Fix bug where we didn't time out HTTP requests to ASes (PR matrix-org#2192) Docs: * Clarify doc for SQLite to PostgreSQL port (PR matrix-org#1961) Thanks @benhylau! * Fix typo in synctl help (PR matrix-org#2107) Thanks @HarHarLinks! * ``web_client_location`` documentation fix (PR matrix-org#2131) Thanks @matthewjwolff! * Update README.rst with FreeBSD changes (PR matrix-org#2132) Thanks @feld! * Clarify setting up metrics (PR matrix-org#2149) Thanks @encks!
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
... because that's (a) pointless (b) slow and (c) eats all my disk space just when I'm trying to free disk space