Add `order_by` to list users' media admin API #8978

dklimpel · 2020-12-22T11:49:44Z

Add order_by to the admin API GET /_synapse/admin/v1/users/<user_id>/media.
This is a breaking change for default sort order to #8647 which was introduced in Synapse v1.23.0.

It is possible to prevent the breaking change.
At the moment ist the sort per default ascending.
In the room API has every order field its own default direction:

synapse/synapse/storage/databases/main/room.py

Lines 391 to 438 in 4218473

    
           # Set ordering 
        
           if RoomSortOrder(order_by) == RoomSortOrder.SIZE: 
        
               # Deprecated in favour of RoomSortOrder.JOINED_MEMBERS 
        
               order_by_column = "curr.joined_members" 
        
               order_by_asc = False 
        
           elif RoomSortOrder(order_by) == RoomSortOrder.ALPHABETICAL: 
        
               # Deprecated in favour of RoomSortOrder.NAME 
        
               order_by_column = "state.name" 
        
               order_by_asc = True 
        
           elif RoomSortOrder(order_by) == RoomSortOrder.NAME: 
        
               order_by_column = "state.name" 
        
               order_by_asc = True 
        
           elif RoomSortOrder(order_by) == RoomSortOrder.CANONICAL_ALIAS: 
        
               order_by_column = "state.canonical_alias" 
        
               order_by_asc = True 
        
           elif RoomSortOrder(order_by) == RoomSortOrder.JOINED_MEMBERS: 
        
               order_by_column = "curr.joined_members" 
        
               order_by_asc = False 
        
           elif RoomSortOrder(order_by) == RoomSortOrder.JOINED_LOCAL_MEMBERS: 
        
               order_by_column = "curr.local_users_in_room" 
        
               order_by_asc = False 
        
           elif RoomSortOrder(order_by) == RoomSortOrder.VERSION: 
        
               order_by_column = "rooms.room_version" 
        
               order_by_asc = False 
        
           elif RoomSortOrder(order_by) == RoomSortOrder.CREATOR: 
        
               order_by_column = "rooms.creator" 
        
               order_by_asc = True 
        
           elif RoomSortOrder(order_by) == RoomSortOrder.ENCRYPTION: 
        
               order_by_column = "state.encryption" 
        
               order_by_asc = True 
        
           elif RoomSortOrder(order_by) == RoomSortOrder.FEDERATABLE: 
        
               order_by_column = "state.is_federatable" 
        
               order_by_asc = True 
        
           elif RoomSortOrder(order_by) == RoomSortOrder.PUBLIC: 
        
               order_by_column = "rooms.is_public" 
        
               order_by_asc = True 
        
           elif RoomSortOrder(order_by) == RoomSortOrder.JOIN_RULES: 
        
               order_by_column = "state.join_rules" 
        
               order_by_asc = True 
        
           elif RoomSortOrder(order_by) == RoomSortOrder.GUEST_ACCESS: 
        
               order_by_column = "state.guest_access" 
        
               order_by_asc = True 
        
           elif RoomSortOrder(order_by) == RoomSortOrder.HISTORY_VISIBILITY: 
        
               order_by_column = "state.history_visibility" 
        
               order_by_asc = True 
        
           elif RoomSortOrder(order_by) == RoomSortOrder.STATE_EVENTS: 
        
               order_by_column = "curr.current_state_events" 
        
               order_by_asc = False

In my opinion this makes it complicated because you don't know what the default sort order is now.

Related to: #8647 (comment)

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
Pull request includes a sign off
Code style is correct (run the linters)

Signed-off-by: Dirk Klimpel dirk@klimpel.org

clokep · 2020-12-30T14:48:55Z

It seems like it would be pretty straightforward to make the default the previous sort order, which would make it backwards compatible. Is there a good reason not to do this?

dklimpel · 2021-01-07T13:40:32Z

It seems like it would be pretty straightforward to make the default the previous sort order, which would make it backwards compatible. Is there a good reason not to do this?

I have did an update to change default sort order to prevent breaking change.

erikjohnston

I think this looks OK, my concern is the performance of this. We only have indexes on media_id, user_id and created_ts, so the other sort fields will involve walking the entire table. I don't know if we should a) only support sorting by those columns or b) put warnings in the doc?

docs/admin_api/user_admin_api.rst

clokep · 2021-02-05T13:56:24Z

@dklimpel Is this ready for another look?

dklimpel · 2021-02-05T14:19:15Z

@dklimpel Is this ready for another look?

Yes.

richvdh

looks generally great! A few suggestions here.

docs/admin_api/user_admin_api.rst

synapse/rest/admin/users.py

richvdh · 2021-02-11T15:52:05Z

synapse/rest/admin/users.py

+        if (
+            parse_string(request, "order_by") is None
+            and parse_string(request, "dir") is None
+        ):
+            order_by = MediaSortOrder.CREATED_TS.value
+            direction = "b"


I think it would be clearer to do:

Suggested change

if (

parse_string(request, "order_by") is None

and parse_string(request, "dir") is None

):

order_by = MediaSortOrder.CREATED_TS.value

direction = "b"

if (

"order_by" not in request.args and "dir" not in request.args

):

order_by = MediaSortOrder.CREATED_TS.value

direction = "b"

else:

order_by = parse_string(...) # etc

synapse/storage/databases/main/media_repository.py

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

clokep

Looks pretty good overall, but I have a couple of comments, sorry about that!

docs/admin_api/user_admin_api.rst

clokep · 2021-02-18T13:16:25Z

synapse/rest/admin/users.py

+        if b"order_by" not in request.args and b"dir" not in request.args:
+            order_by = MediaSortOrder.CREATED_TS.value
+            direction = "b"


Isn't this identical to providing the default key below?

This is for the special case of backward compatibility of this API. In case of no request parameter there is a special value of default: direction is b. In a normal case the direction is f.
It was a request: #8978 (comment)

Ah, right! I missed that the direction was opposite the default. Good catch!

clokep · 2021-02-18T13:20:50Z

synapse/storage/databases/main/media_repository.py

+            if MediaSortOrder(order_by) == MediaSortOrder.MEDIA_ID:
+                order_by_column = "media_id"
+            elif MediaSortOrder(order_by) == MediaSortOrder.UPLOAD_NAME:
+                order_by_column = "upload_name"
+            elif MediaSortOrder(order_by) == MediaSortOrder.CREATED_TS:
+                order_by_column = "created_ts"
+            elif MediaSortOrder(order_by) == MediaSortOrder.LAST_ACCESS_TS:
+                order_by_column = "last_access_ts"
+            elif MediaSortOrder(order_by) == MediaSortOrder.MEDIA_LENGTH:
+                order_by_column = "media_length"
+            elif MediaSortOrder(order_by) == MediaSortOrder.MEDIA_TYPE:
+                order_by_column = "media_type"
+            elif MediaSortOrder(order_by) == MediaSortOrder.QUARANTINED_BY:
+                order_by_column = "quarantined_by"
+            elif MediaSortOrder(order_by) == MediaSortOrder.SAFE_FROM_QUARANTINE:
+                order_by_column = "safe_from_quarantine"


I think this block isn't really necessary and can be replaced with the following:

order_by_column = MediaSortOrder(order_by).value

This will raise a ValueError if an invalid value is provided.

tests/rest/admin/test_user.py

docs/admin_api/user_admin_api.rst

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

order test parameter replace MediaSortOrder block

tests/rest/admin/test_user.py

clokep · 2021-02-18T14:11:42Z

tests/rest/admin/test_user.py

+        # Different sort order of SQlite and PostreSQL foor bool
+        # self._order_test("quarantined_by", sorted([media1, media2]) + [media3])
+        # self._order_test("quarantined_by", "f", sorted([media1, media2]) + [media3])
+        # self._order_test("quarantined_by", "b", [media3] + sorted([media1, media2]))


This seems unfortunate. Why is this not the case for safe_from_quarantine?

I will do a new test.

Different sort order of SQlite and PostreSQL:
If a media is not in quarantine quarantined_by is NULL
SQLite considers NULL to be smaller than any other value.
PostreSQL considers NULL to be larger than any other value.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

Conflicts: tests/rest/admin/test_user.py

clokep

I think this is OK. I'm not thrilled about the sort order being different from sqlite and postgresql, but I don't know if there's a solution to that. (I'll leave this for a day or two before merging in case someone else has an idea though.)

changes handled

dklimpel · 2021-02-18T19:51:42Z

I think this is OK. I'm not thrilled about the sort order being different from sqlite and postgresql, but I don't know if there's a solution to that. (I'll leave this for a day or two before merging in case someone else has an idea though.)

If someone has an idea. Some information.

synapse/synapse/storage/databases/main/search.py

Lines 569 to 579 in 1baab20

    
           # We add an arbitrary limit here to ensure we don't try to pull the 
        
           # entire table from the database. 
        
           if isinstance(self.database_engine, PostgresEngine): 
        
               sql += ( 
        
                   " ORDER BY origin_server_ts DESC NULLS LAST," 
        
                   " stream_ordering DESC NULLS LAST LIMIT ?" 
        
               ) 
        
           elif isinstance(self.database_engine, Sqlite3Engine): 
        
               sql += " ORDER BY origin_server_ts DESC, stream_ordering DESC LIMIT ?" 
        
           else: 
        
               raise Exception("Unrecognized database engine")

https://www.postgresql.org/docs/current/queries-order.html
The NULLS FIRST and NULLS LAST options can be used to determine whether nulls appear before or after non-null values in the sort ordering. By default, null values sort as if larger than any non-null value; that is, NULLS FIRST is the default for DESC order, and NULLS LAST otherwise.

https://www.sqlitetutorial.net/sqlite-order-by/
SQLite 3.30.0 added the NULLS FIRST and NULLS LAST options to the ORDER BY clause. The NULLS FIRST option specifies that the NULLs will appear at the beginning of the result set while the NULLS LAST option place NULLs at the end of the result set.

It seems that with the same syntax.
What is the best solution?
ASC NULLS LAST and DESC NULLS FIRST
or
ASC NULLS FIRST and DESC NULLS LAST

See also: https://learnsql.com/blog/how-to-order-rows-with-nulls/

clokep · 2021-02-22T19:38:03Z

From chatting about this with the team it seems there's a few other spots where the ordering differs between SQLite and PostgreSQL. So I think this is OK.

@dklimpel

Synapse 1.29.0 (2021-03-08) =========================== Note that synapse now expects an `X-Forwarded-Proto` header when used with a reverse proxy. Please see [UPGRADE.rst](UPGRADE.rst#upgrading-to-v1290) for more details on this change. No significant changes. Synapse 1.29.0rc1 (2021-03-04) ============================== Features -------- - Add rate limiters to cross-user key sharing requests. ([\#8957](matrix-org/synapse#8957)) - Add `order_by` to the admin API `GET /_synapse/admin/v1/users/<user_id>/media`. Contributed by @dklimpel. ([\#8978](matrix-org/synapse#8978)) - Add some configuration settings to make users' profile data more private. ([\#9203](matrix-org/synapse#9203)) - The `no_proxy` and `NO_PROXY` environment variables are now respected in proxied HTTP clients with the lowercase form taking precedence if both are present. Additionally, the lowercase `https_proxy` environment variable is now respected in proxied HTTP clients on top of existing support for the uppercase `HTTPS_PROXY` form and takes precedence if both are present. Contributed by Timothy Leung. ([\#9372](matrix-org/synapse#9372)) - Add a configuration option, `user_directory.prefer_local_users`, which when enabled will make it more likely for users on the same server as you to appear above other users. ([\#9383](matrix-org/synapse#9383), [\#9385](matrix-org/synapse#9385)) - Add support for regenerating thumbnails if they have been deleted but the original image is still stored. ([\#9438](matrix-org/synapse#9438)) - Add support for `X-Forwarded-Proto` header when using a reverse proxy. ([\#9472](matrix-org/synapse#9472), [\#9501](matrix-org/synapse#9501), [\#9512](matrix-org/synapse#9512), [\#9539](matrix-org/synapse#9539)) Bugfixes -------- - Fix a bug where users' pushers were not all deleted when they deactivated their account. ([\#9285](matrix-org/synapse#9285), [\#9516](matrix-org/synapse#9516)) - Fix a bug where a lot of unnecessary presence updates were sent when joining a room. ([\#9402](matrix-org/synapse#9402)) - Fix a bug that caused multiple calls to the experimental `shared_rooms` endpoint to return stale results. ([\#9416](matrix-org/synapse#9416)) - Fix a bug in single sign-on which could cause a "No session cookie found" error. ([\#9436](matrix-org/synapse#9436)) - Fix bug introduced in v1.27.0 where allowing a user to choose their own username when logging in via single sign-on did not work unless an `idp_icon` was defined. ([\#9440](matrix-org/synapse#9440)) - Fix a bug introduced in v1.26.0 where some sequences were not properly configured when running `synapse_port_db`. ([\#9449](matrix-org/synapse#9449)) - Fix deleting pushers when using sharded pushers. ([\#9465](matrix-org/synapse#9465), [\#9466](matrix-org/synapse#9466), [\#9479](matrix-org/synapse#9479), [\#9536](matrix-org/synapse#9536)) - Fix missing startup checks for the consistency of certain PostgreSQL sequences. ([\#9470](matrix-org/synapse#9470)) - Fix a long-standing bug where the media repository could leak file descriptors while previewing media. ([\#9497](matrix-org/synapse#9497)) - Properly purge the event chain cover index when purging history. ([\#9498](matrix-org/synapse#9498)) - Fix missing chain cover index due to a schema delta not being applied correctly. Only affected servers that ran development versions. ([\#9503](matrix-org/synapse#9503)) - Fix a bug introduced in v1.25.0 where `/_synapse/admin/join/` would fail when given a room alias. ([\#9506](matrix-org/synapse#9506)) - Prevent presence background jobs from running when presence is disabled. ([\#9530](matrix-org/synapse#9530)) - Fix rare edge case that caused a background update to fail if the server had rejected an event that had duplicate auth events. ([\#9537](matrix-org/synapse#9537)) Improved Documentation ---------------------- - Update the example systemd config to propagate reloads to individual units. ([\#9463](matrix-org/synapse#9463)) Internal Changes ---------------- - Add documentation and type hints to `parse_duration`. ([\#9432](matrix-org/synapse#9432)) - Remove vestiges of `uploads_path` configuration setting. ([\#9462](matrix-org/synapse#9462)) - Add a comment about systemd-python. ([\#9464](matrix-org/synapse#9464)) - Test that we require validated email for email pushers. ([\#9496](matrix-org/synapse#9496)) - Allow python to generate bytecode for synapse. ([\#9502](matrix-org/synapse#9502)) - Fix incorrect type hints. ([\#9515](matrix-org/synapse#9515), [\#9518](matrix-org/synapse#9518)) - Add type hints to device and event report admin API. ([\#9519](matrix-org/synapse#9519)) - Add type hints to user admin API. ([\#9521](matrix-org/synapse#9521)) - Bump the versions of mypy and mypy-zope used for static type checking. ([\#9529](matrix-org/synapse#9529))

dklimpel added 2 commits December 22, 2020 12:39

Add order_by to list users' media admin API

f6b4954

update test - mismatch SQlite and PostreSQL

23a731e

dklimpel marked this pull request as ready for review December 22, 2020 13:34

clokep requested a review from a team December 30, 2020 14:04

dklimpel added 2 commits January 4, 2021 11:43

Change default sort order to prevent breaking change

e0b5345

Merge remote-tracking branch 'synapse/develop' into order_media

278fe01

erikjohnston reviewed Jan 11, 2021

View reviewed changes

docs/admin_api/user_admin_api.rst Outdated Show resolved Hide resolved

dklimpel added 2 commits January 11, 2021 15:05

Add performance warning

c8f1904

Merge remote-tracking branch 'synapse/develop' into order_media

f0e8c8a

clokep requested a review from a team February 5, 2021 14:25

richvdh previously requested changes Feb 11, 2021

View reviewed changes

dklimpel and others added 3 commits February 11, 2021 20:49

Apply suggestions from code review - Part 1

84ab2b6

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Apply suggestions from code review - Part 2

ef3a96f

Merge remote-tracking branch 'synapse/develop' into order_media

5e813f5

dklimpel requested a review from richvdh February 11, 2021 21:28

Add missing store to UserMediaRestTestCase

cd85a9d

richvdh requested review from a team and removed request for richvdh February 16, 2021 14:52

clokep suggested changes Feb 18, 2021

View reviewed changes

dklimpel and others added 2 commits February 18, 2021 14:33

Apply suggestions from code review - Part 1

222af2f

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

Apply suggestions from code review - Part 2

179dc3c

order test parameter replace MediaSortOrder block

clokep reviewed Feb 18, 2021

View reviewed changes

dklimpel and others added 4 commits February 18, 2021 15:19

Update tests/rest/admin/test_user.py

50e6c67

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

re-order test

15947d6

Merge remote-tracking branch 'synapse/develop' into order_media

79a5ee5

Conflicts: tests/rest/admin/test_user.py

fix wrong merge and code style

fc6c345

dklimpel added 2 commits February 18, 2021 17:31

fix test

dd69590

Merge remote-tracking branch 'synapse/develop' into order_media

48ed7f8

clokep self-requested a review February 18, 2021 19:04

clokep approved these changes Feb 18, 2021

View reviewed changes

clokep merged commit 71c9f8d into matrix-org:develop Feb 22, 2021

dklimpel deleted the order_media branch February 22, 2021 19:44

dklimpel mentioned this pull request Mar 30, 2021

Add order_by to list user admin API #9691

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `order_by` to list users' media admin API #8978

Add `order_by` to list users' media admin API #8978

dklimpel commented Dec 22, 2020

clokep commented Dec 30, 2020

dklimpel commented Jan 7, 2021

erikjohnston left a comment

clokep commented Feb 5, 2021

dklimpel commented Feb 5, 2021

richvdh left a comment

richvdh Feb 11, 2021

clokep left a comment

clokep Feb 18, 2021

dklimpel Feb 18, 2021

clokep Feb 18, 2021

clokep Feb 18, 2021

clokep Feb 18, 2021

dklimpel Feb 18, 2021

dklimpel Feb 18, 2021

clokep left a comment

dklimpel commented Feb 18, 2021 •

edited

Loading

clokep commented Feb 22, 2021

	# Set ordering
	if RoomSortOrder(order_by) == RoomSortOrder.SIZE:
	# Deprecated in favour of RoomSortOrder.JOINED_MEMBERS
	order_by_column = "curr.joined_members"
	order_by_asc = False
	elif RoomSortOrder(order_by) == RoomSortOrder.ALPHABETICAL:
	# Deprecated in favour of RoomSortOrder.NAME
	order_by_column = "state.name"
	order_by_asc = True
	elif RoomSortOrder(order_by) == RoomSortOrder.NAME:
	order_by_column = "state.name"
	order_by_asc = True
	elif RoomSortOrder(order_by) == RoomSortOrder.CANONICAL_ALIAS:
	order_by_column = "state.canonical_alias"
	order_by_asc = True
	elif RoomSortOrder(order_by) == RoomSortOrder.JOINED_MEMBERS:
	order_by_column = "curr.joined_members"
	order_by_asc = False
	elif RoomSortOrder(order_by) == RoomSortOrder.JOINED_LOCAL_MEMBERS:
	order_by_column = "curr.local_users_in_room"
	order_by_asc = False
	elif RoomSortOrder(order_by) == RoomSortOrder.VERSION:
	order_by_column = "rooms.room_version"
	order_by_asc = False
	elif RoomSortOrder(order_by) == RoomSortOrder.CREATOR:
	order_by_column = "rooms.creator"
	order_by_asc = True
	elif RoomSortOrder(order_by) == RoomSortOrder.ENCRYPTION:
	order_by_column = "state.encryption"
	order_by_asc = True
	elif RoomSortOrder(order_by) == RoomSortOrder.FEDERATABLE:
	order_by_column = "state.is_federatable"
	order_by_asc = True
	elif RoomSortOrder(order_by) == RoomSortOrder.PUBLIC:
	order_by_column = "rooms.is_public"
	order_by_asc = True
	elif RoomSortOrder(order_by) == RoomSortOrder.JOIN_RULES:
	order_by_column = "state.join_rules"
	order_by_asc = True
	elif RoomSortOrder(order_by) == RoomSortOrder.GUEST_ACCESS:
	order_by_column = "state.guest_access"
	order_by_asc = True
	elif RoomSortOrder(order_by) == RoomSortOrder.HISTORY_VISIBILITY:
	order_by_column = "state.history_visibility"
	order_by_asc = True
	elif RoomSortOrder(order_by) == RoomSortOrder.STATE_EVENTS:
	order_by_column = "curr.current_state_events"
	order_by_asc = False

Add order_by to list users' media admin API #8978

Add order_by to list users' media admin API #8978

Conversation

dklimpel commented Dec 22, 2020

Pull Request Checklist

clokep commented Dec 30, 2020

dklimpel commented Jan 7, 2021

erikjohnston left a comment

Choose a reason for hiding this comment

clokep commented Feb 5, 2021

dklimpel commented Feb 5, 2021

richvdh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clokep left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clokep left a comment

Choose a reason for hiding this comment

dklimpel commented Feb 18, 2021 • edited Loading

clokep commented Feb 22, 2021

Add `order_by` to list users' media admin API #8978

Add `order_by` to list users' media admin API #8978

dklimpel commented Feb 18, 2021 •

edited

Loading