Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Sliding Sync /sync endpoint (initial implementation) #17187

Merged
merged 123 commits into from
Jun 6, 2024

Commits on May 6, 2024

  1. Configuration menu
    Copy the full SHA
    f9e6e53 View commit details
    Browse the repository at this point in the history

Commits on May 7, 2024

  1. Add Sliding Sync /sync/e2ee endpoint for To-Device messages

    Based on:
    
     - MSC3575: Sliding Sync (aka Sync v3): matrix-org/matrix-spec-proposals#3575
     - MSC3885: Sliding Sync Extension: To-Device messages: matrix-org/matrix-spec-proposals#3885
     - MSC3884: Sliding Sync Extension: E2EE: matrix-org/matrix-spec-proposals#3884
    MadLittleMods committed May 7, 2024
    Configuration menu
    Copy the full SHA
    1e05a05 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. Configuration menu
    Copy the full SHA
    5e925f6 View commit details
    Browse the repository at this point in the history
  2. Comment on tests

    MadLittleMods committed May 8, 2024
    Configuration menu
    Copy the full SHA
    69f9143 View commit details
    Browse the repository at this point in the history
  3. Prefer Sync v2 vs Sliding Sync distinction

    We were using the enum just to distinguish /sync v2
    vs Sliding Sync /sync/e2ee so we should just make an
    enum for that instead of trying to glom onto the
    existing `sync_type` (overloading it).
    MadLittleMods committed May 8, 2024
    Configuration menu
    Copy the full SHA
    d4ff933 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    371ec57 View commit details
    Browse the repository at this point in the history
  5. Ugly overloads

    MadLittleMods committed May 8, 2024
    Configuration menu
    Copy the full SHA
    06d12e5 View commit details
    Browse the repository at this point in the history
  6. Fix lint

    MadLittleMods committed May 8, 2024
    Configuration menu
    Copy the full SHA
    b8b70ba View commit details
    Browse the repository at this point in the history
  7. Add changelog

    MadLittleMods committed May 8, 2024
    Configuration menu
    Copy the full SHA
    c60a4f8 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    10ffae6 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. Try calculate more

    MadLittleMods committed May 9, 2024
    Configuration menu
    Copy the full SHA
    6bf4896 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8871dac View commit details
    Browse the repository at this point in the history
  3. Add comments docs

    MadLittleMods committed May 9, 2024
    Configuration menu
    Copy the full SHA
    0892283 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    adb7e20 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    f098355 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    6b7cfd7 View commit details
    Browse the repository at this point in the history
  7. Describe test

    MadLittleMods committed May 9, 2024
    Configuration menu
    Copy the full SHA
    b9e5379 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. Configuration menu
    Copy the full SHA
    f9c9d44 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    654e8f6 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. Configuration menu
    Copy the full SHA
    aee594a View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. More optional

    MadLittleMods committed May 15, 2024
    Configuration menu
    Copy the full SHA
    2863fba View commit details
    Browse the repository at this point in the history
  2. Fill out more options

    MadLittleMods committed May 15, 2024
    Configuration menu
    Copy the full SHA
    2dd0cde View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c8256b6 View commit details
    Browse the repository at this point in the history
  4. Iterating

    MadLittleMods committed May 15, 2024
    Configuration menu
    Copy the full SHA
    ee6baba View commit details
    Browse the repository at this point in the history
  5. Copy body to config

    MadLittleMods committed May 15, 2024
    Configuration menu
    Copy the full SHA
    f3db068 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. Merge branch 'develop' into madlittlemods/msc3575-sliding-sync-e2ee

    Conflicts:
    	synapse/handlers/sync.py
    	tests/handlers/test_sync.py
    MadLittleMods committed May 16, 2024
    Configuration menu
    Copy the full SHA
    9bdfa16 View commit details
    Browse the repository at this point in the history
  2. Lint

    MadLittleMods committed May 16, 2024
    Configuration menu
    Copy the full SHA
    7331401 View commit details
    Browse the repository at this point in the history
  3. Fix test inheritance

    MadLittleMods committed May 16, 2024
    Configuration menu
    Copy the full SHA
    b23abca View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    821a1b3 View commit details
    Browse the repository at this point in the history
  5. Format docstring

    MadLittleMods committed May 16, 2024
    Configuration menu
    Copy the full SHA
    35ca937 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    4ad7a8b View commit details
    Browse the repository at this point in the history

Commits on May 20, 2024

  1. Configuration menu
    Copy the full SHA
    3092ab5 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3539abe View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5f194f9 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    02cecfa View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    bfa8c63 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    07d84ab View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    6dadfe9 View commit details
    Browse the repository at this point in the history

Commits on May 21, 2024

  1. Configuration menu
    Copy the full SHA
    9ffafe7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f6122ff View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2f112e7 View commit details
    Browse the repository at this point in the history
  4. Lint

    MadLittleMods committed May 21, 2024
    Configuration menu
    Copy the full SHA
    c2221bb View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    717b160 View commit details
    Browse the repository at this point in the history
  6. Add some tests

    MadLittleMods committed May 21, 2024
    Configuration menu
    Copy the full SHA
    c826550 View commit details
    Browse the repository at this point in the history
  7. Handle more edge cases

    MadLittleMods committed May 21, 2024
    Configuration menu
    Copy the full SHA
    fe48188 View commit details
    Browse the repository at this point in the history
  8. WIP

    MadLittleMods committed May 21, 2024
    Configuration menu
    Copy the full SHA
    fd355f6 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    dd9356a View commit details
    Browse the repository at this point in the history

Commits on May 22, 2024

  1. Log why no unsigned

    MadLittleMods committed May 22, 2024
    Configuration menu
    Copy the full SHA
    17783c3 View commit details
    Browse the repository at this point in the history
  2. Remove debug logs

    MadLittleMods committed May 22, 2024
    Configuration menu
    Copy the full SHA
    343de8f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1b3a5bf View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c82a084 View commit details
    Browse the repository at this point in the history
  5. Merge branch 'develop' into madlittlemods/msc3575-sliding-sync-e2ee

    Conflicts:
    	synapse/handlers/sync.py
    MadLittleMods committed May 22, 2024
    Configuration menu
    Copy the full SHA
    514aba5 View commit details
    Browse the repository at this point in the history
  6. Update filter to be more precise and avoid more work

     - Added `room.account_data` and `room.presence` to avoid extra work in `_generate_sync_entry_for_rooms()`
     - Added a comment to the top-level `account_data` and `presence` filters that `(This is just here for good measure)`
    
    See #17167 (comment)
    MadLittleMods committed May 22, 2024
    Configuration menu
    Copy the full SHA
    9749795 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    06ac1da View commit details
    Browse the repository at this point in the history
  8. Use @parameterized_class

    As suggested in #17167 (comment)
    MadLittleMods committed May 22, 2024
    Configuration menu
    Copy the full SHA
    3da6bc1 View commit details
    Browse the repository at this point in the history
  9. Fix lints

    MadLittleMods committed May 22, 2024
    Configuration menu
    Copy the full SHA
    d4b41aa View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    89db566 View commit details
    Browse the repository at this point in the history
  11. Add changelog

    MadLittleMods committed May 22, 2024
    Configuration menu
    Copy the full SHA
    c7b8743 View commit details
    Browse the repository at this point in the history
  12. Use client_patterns()

    MadLittleMods committed May 22, 2024
    Configuration menu
    Copy the full SHA
    a7c6476 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    13d6146 View commit details
    Browse the repository at this point in the history
  14. Start assembling lists

    MadLittleMods committed May 22, 2024
    Configuration menu
    Copy the full SHA
    c7f7ae4 View commit details
    Browse the repository at this point in the history
  15. Encode response

    MadLittleMods committed May 22, 2024
    Configuration menu
    Copy the full SHA
    4c7d7e6 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. Configuration menu
    Copy the full SHA
    6606ac1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ab0b844 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a482545 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    04eeee6 View commit details
    Browse the repository at this point in the history
  5. Merge branch 'develop' into madlittlemods/msc3575-sliding-sync-0.0.1

    Conflicts:
    	synapse/rest/client/sync.py
    MadLittleMods committed May 23, 2024
    Configuration menu
    Copy the full SHA
    8c3de84 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    37af87a View commit details
    Browse the repository at this point in the history
  7. Revert as TODO says

    MadLittleMods committed May 23, 2024
    Configuration menu
    Copy the full SHA
    a822a05 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    f9fa683 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    d1bd02d View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    b5b3e77 View commit details
    Browse the repository at this point in the history
  11. Fix lints

    MadLittleMods committed May 23, 2024
    Configuration menu
    Copy the full SHA
    65d9b79 View commit details
    Browse the repository at this point in the history
  12. Fix unserialize type

    MadLittleMods committed May 23, 2024
    Configuration menu
    Copy the full SHA
    adc0e2f View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. Configuration menu
    Copy the full SHA
    b12fee5 View commit details
    Browse the repository at this point in the history
  2. Fill in rest docstring

    MadLittleMods committed May 28, 2024
    Configuration menu
    Copy the full SHA
    44e9a92 View commit details
    Browse the repository at this point in the history
  3. Add better comments

    MadLittleMods committed May 28, 2024
    Configuration menu
    Copy the full SHA
    b632cbb View commit details
    Browse the repository at this point in the history
  4. Fill out docstring todo

    MadLittleMods committed May 28, 2024
    Configuration menu
    Copy the full SHA
    abf139a View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    a28569f View commit details
    Browse the repository at this point in the history
  6. Tweak comments

    MadLittleMods committed May 28, 2024
    Configuration menu
    Copy the full SHA
    950fd70 View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. Add rest test

    MadLittleMods committed May 29, 2024
    Configuration menu
    Copy the full SHA
    8bf5a62 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    34d67fd View commit details
    Browse the repository at this point in the history

Commits on May 30, 2024

  1. Configuration menu
    Copy the full SHA
    49998e0 View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. Configuration menu
    Copy the full SHA
    09609cb View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8f09313 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    803fbbe View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    a0c042e View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    271a196 View commit details
    Browse the repository at this point in the history
  6. Fix circular imports when running specific tests

    Before:
    ```
    $ SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.replication.storage.test_events
    [...]
    Traceback (most recent call last):
      File "pypoetry/virtualenvs/matrix-synapse-xCtC9ulO-py3.12/lib/python3.12/site-packages/twisted/trial/runner.py", line 711, in loadByName
        return self.suiteFactory([self.findByName(name, recurse=recurse)])
      File "pypoetry/virtualenvs/matrix-synapse-xCtC9ulO-py3.12/lib/python3.12/site-packages/twisted/trial/runner.py", line 474, in findByName
        obj = reflect.namedModule(searchName)
      File "pypoetry/virtualenvs/matrix-synapse-xCtC9ulO-py3.12/lib/python3.12/site-packages/twisted/python/reflect.py", line 156, in namedModule
        topLevel = __import__(name)
      File "synapse/tests/replication/storage/test_events.py", line 33, in <module>
        from synapse.handlers.room import RoomEventSource
      File "synapse/synapse/handlers/room.py", line 74, in <module>
        from synapse.rest.admin._base import assert_user_is_admin
      File "synapse/synapse/rest/__init__.py", line 24, in <module>
        from synapse.rest import admin
      File "synapse/synapse/rest/admin/__init__.py", line 41, in <module>
        from synapse.handlers.pagination import PURGE_HISTORY_ACTION_NAME
      File "synapse/synapse/handlers/pagination.py", line 30, in <module>
        from synapse.handlers.room import ShutdownRoomParams, ShutdownRoomResponse
    builtins.ImportError: cannot import name 'ShutdownRoomParams' from partially initialized module 'synapse.handlers.room' (most likely due to a circular import) (synapse/synapse/handlers/room.py)
    ```
    MadLittleMods committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    4155e18 View commit details
    Browse the repository at this point in the history
  7. Update usage

    MadLittleMods committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    939695d View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    73c20d9 View commit details
    Browse the repository at this point in the history
  9. Fix random lints

    MadLittleMods committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    7b41f41 View commit details
    Browse the repository at this point in the history
  10. Add changelog

    MadLittleMods committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    09638ac View commit details
    Browse the repository at this point in the history
  11. Merge branch 'madlittlemods/rooms-for-user-event-pos' into madlittlem…

    …ods/msc3575-sliding-sync-0.0.1
    
    Conflicts:
    	synapse/handlers/initial_sync.py
    	tests/replication/storage/test_events.py
    MadLittleMods committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    8dca8f5 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. Configuration menu
    Copy the full SHA
    9c6ec25 View commit details
    Browse the repository at this point in the history
  2. Properly compare tokens and event positions

    Avoid flawed raw stream comparison.
    MadLittleMods committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    1268a54 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e4c66b8 View commit details
    Browse the repository at this point in the history
  4. Add support for kicks

    MadLittleMods committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    35db057 View commit details
    Browse the repository at this point in the history
  5. Add licensing headers

    MadLittleMods committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    3514aa0 View commit details
    Browse the repository at this point in the history
  6. Adjust wording

    MadLittleMods committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    970a0c6 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    64df6fb View commit details
    Browse the repository at this point in the history
  8. Note the extras

    MadLittleMods committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    8bb357a View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    03dd87a View commit details
    Browse the repository at this point in the history
  10. Fix typo

    MadLittleMods committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    9e46b2a View commit details
    Browse the repository at this point in the history
  11. Add None defaults

    MadLittleMods committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    54dbc27 View commit details
    Browse the repository at this point in the history
  12. Merge branch 'develop' into madlittlemods/msc3575-sliding-sync-0.0.1

    Conflicts:
    	synapse/storage/databases/main/roommember.py
    MadLittleMods committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    f6a5905 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    07f57a4 View commit details
    Browse the repository at this point in the history
  14. Balance parenthesis

    MadLittleMods committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    d3ce27b View commit details
    Browse the repository at this point in the history
  15. Switch fixup order to fix edge case with newly_left rooms

    Previously, we added back newly_left rooms and our second
    fixup was removing them. We can just switch the order
    of the fixups to solve this.
    MadLittleMods committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    dfee21a View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. Add test with multiple event persisters that fails the previous `get_…

    …sync_room_ids_for_user` implementation
    
    The new implementation catches the problem with an assert
    but I think it's possible to make it work as well.
    
    ```
    SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.handlers.test_sliding_sync.GetSyncRoomIdsForUserEventShardTestCase
    ```
    MadLittleMods committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    3ce0892 View commit details
    Browse the repository at this point in the history
  2. Allow new get_sync_room_ids_for_user implementation to work with mu…

    …ltiple event persisters
    
    Before, the problem scenario would get caught in one of the assertions because
    we expect the to_token <= membership_snapshot_token or vice-versa but it's
    possible the tokens are intertwined and neither is ahead of each other.
    Especially since the `instance_map` in `membership_snapshot_token` is made up
    from the `stream_ordering` of membership events at various stream positions
    and processed on different instances (not current stream positions).
    
    We get into trouble when stream positions are lagging between workers and our
    now/`to_token` doesn't cleanly compare to `membership_snapshot_token`.
    
    What we really want to assert is that the `to_token` <= the stream positions
    at the time we asked for the room membership snapshot. Since
    `get_rooms_for_local_user_where_membership_is()` doesn't return that
    information, the closest we can get is to get the stream positions before we
    ask for the room membership snapshot and consider that good enough to compare
    against.
    MadLittleMods committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    2864837 View commit details
    Browse the repository at this point in the history
  3. Remove extra for-loop

    Co-authored-by: Erik Johnston <erikj@element.io>
    MadLittleMods and erikjohnston committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    2af467d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    7bbe2ed View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    1fc1b58 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    6a6cdc6 View commit details
    Browse the repository at this point in the history
  7. No need to check from/to token relationship

    See #17187 (comment)
    
    `get_membership_changes_for_user(from_key=xxx, to_key=xxx)` will handle
    getting out what we need and filter the results based on the tokens
    (even in cases where the from_key is ahead of the to_key).
    MadLittleMods committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    278ba63 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    5678307 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    703cdc9 View commit details
    Browse the repository at this point in the history
  10. Fix separator label

    MadLittleMods committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    c7d1fc3 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. Configuration menu
    Copy the full SHA
    0f6646d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0153a6e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    6f10b97 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c89f012 View commit details
    Browse the repository at this point in the history