Use Pydantic to systematically validate a first batch of endpoints in `synapse.rest.client.account`. #13188

DMRobertson · 2022-07-05T16:55:56Z

A contribution towards #13183.

Covers:

EmailPasswordRequestTokenRestServlet
PasswordRestServlet
DeactivateAccountRestServlet
EmailThreepidRequestTokenRestServlet

it's about to get reused

synapse/rest/client/account.py

DMRobertson · 2022-07-05T18:48:42Z

If anyone wants to see the kind of error messages this produces, try running this test case on my other pydantic experiment branch.

I think the name of the model isn't user-relevant. The "error context" (stuff in brackets) also loks like noise to an end-user. But we should be able to inspect the ValidationError and format it as we please.

/home/dmr/.cache/pypoetry/virtualenvs/matrix-synapse-7yaa6cKe-py3.10/bin/python /home/dmr/.local/share/JetBrains/Toolbox/apps/PyCharm-C/ch-0/221.5787.24/plugins/python-ce/helpers/pycharm/_jb_unittest_runner.py --path /home/dmr/workspace/synapse-2/tests/config/test_oidc2.py
Testing started at 19:38 ...
Launching unittests with arguments python -m unittest /home/dmr/workspace/synapse-2/tests/config/test_oidc2.py in /home/dmr/workspace/synapse-2/tests/config



Ran 13 tests in 0.033s

OK

Process finished with exit code 0

1 validation error for OIDCProviderModel
attribute_requirements
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
attribute_requirements
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
attribute_requirements
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
attribute_requirements
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
attribute_requirements
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
attribute_requirements -> 0
  value is not a valid dict (type=type_error.dict)

1 validation error for OIDCProviderModel
attribute_requirements
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> attribute
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> value
  field required (type=value_error.missing)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> attribute
  field required (type=value_error.missing)

2 validation errors for OIDCProviderModel
attribute_requirements -> 0 -> attribute
  field required (type=value_error.missing)
attribute_requirements -> 0 -> value
  field required (type=value_error.missing)

1 validation error for OIDCProviderModel
attribute_requirements -> 0 -> answer
  extra fields not permitted (type=value_error.extra)

1 validation error for OIDCProviderModel
client_auth_method
  value is not a valid enumeration member; permitted: 'client_secret_basic', 'client_secret_post', 'none' (type=type_error.enum; enum_values=[<ClientAuthMethods.client_secret_basic: 'client_secret_basic'>, <ClientAuthMethods.client_secret_post: 'client_secret_post'>, <ClientAuthMethods.none: 'none'>])

1 validation error for OIDCProviderModel
client_auth_method
  value is not a valid enumeration member; permitted: 'client_secret_basic', 'client_secret_post', 'none' (type=type_error.enum; enum_values=[<ClientAuthMethods.client_secret_basic: 'client_secret_basic'>, <ClientAuthMethods.client_secret_post: 'client_secret_post'>, <ClientAuthMethods.none: 'none'>])

1 validation error for OIDCProviderModel
client_auth_method
  value is not a valid enumeration member; permitted: 'client_secret_basic', 'client_secret_post', 'none' (type=type_error.enum; enum_values=[<ClientAuthMethods.client_secret_basic: 'client_secret_basic'>, <ClientAuthMethods.client_secret_post: 'client_secret_post'>, <ClientAuthMethods.none: 'none'>])

1 validation error for OIDCProviderModel
client_auth_method
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
client_auth_method
  value is not a valid enumeration member; permitted: 'client_secret_basic', 'client_secret_post', 'none' (type=type_error.enum; enum_values=[<ClientAuthMethods.client_secret_basic: 'client_secret_basic'>, <ClientAuthMethods.client_secret_post: 'client_secret_post'>, <ClientAuthMethods.none: 'none'>])

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
discover
  value is not a valid boolean (type=value_error.strictbool)

1 validation error for OIDCProviderModel
authorization_endpoint
  authorization_endpoint is required if discovery is disabled (type=value_error)

1 validation error for OIDCProviderModel
authorization_endpoint
  authorization_endpoint is required if discovery is disabled (type=value_error)

1 validation error for OIDCProviderModel
token_endpoint
  token_endpoint is required if discovery is disabled (type=value_error)

1 validation error for OIDCProviderModel
token_endpoint
  token_endpoint is required if discovery is disabled (type=value_error)

1 validation error for OIDCProviderModel
idp_brand
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_brand
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_brand
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  expected string or bytes-like object (type=type_error)

1 validation error for OIDCProviderModel
idp_icon
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_icon
  mxc URI '' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_icon
  mxc URI 'notaurl' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_icon
  mxc URI 'https://example.com' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_icon
  mxc URI 'mxc://mxc://mxc://' did not match expected format (type=value_error)

1 validation error for OIDCProviderModel
idp_id
  field required (type=value_error.missing)

1 validation error for OIDCProviderModel
idp_id
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_id
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
idp_id
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_id
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
idp_id
  ensure this value has at least 1 characters (type=value_error.any_str.min_length; limit_value=1)

1 validation error for OIDCProviderModel
idp_id
  ensure this value has at most 250 characters (type=value_error.any_str.max_length; limit_value=250)

1 validation error for OIDCProviderModel
idp_id
  string does not match regex "^[A-Za-z0-9._~-]+$" (type=value_error.str.regex; pattern=^[A-Za-z0-9._~-]+$)

1 validation error for OIDCProviderModel
idp_id
  ensure this value has at most 250 characters (type=value_error.any_str.max_length; limit_value=250)

1 validation error for OIDCProviderModel
issuer
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
issuer
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
issuer
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
issuer
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
issuer
  field required (type=value_error.missing)

2 validation errors for LegacyOIDCProviderModel
idp_id
  str type expected (type=type_error.str)
idp_name
  str type expected (type=type_error.str)

2 validation errors for LegacyOIDCProviderModel
idp_id
  str type expected (type=type_error.str)
idp_name
  str type expected (type=type_error.str)

2 validation errors for LegacyOIDCProviderModel
idp_id
  str type expected (type=type_error.str)
idp_name
  str type expected (type=type_error.str)

2 validation errors for LegacyOIDCProviderModel
idp_id
  none is not an allowed value (type=type_error.none.not_allowed)
idp_name
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  value is not a valid tuple (type=type_error.tuple)

1 validation error for OIDCProviderModel
scopes
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
scopes -> 0
  none is not an allowed value (type=type_error.none.not_allowed)

1 validation error for OIDCProviderModel
scopes -> 0
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
scopes -> 0
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
scopes -> 0
  str type expected (type=type_error.str)

1 validation error for OIDCProviderModel
userinfo_endpoint
  userinfo_requirement is required if discovery is disabled andthe 'openid' scope is not requested (type=value_error)

1 validation error for OIDCProviderModel
userinfo_endpoint
  userinfo_requirement is required if discovery is disabled andthe 'openid' scope is not requested (type=value_error)

1 validation error for OIDCProviderModel
userinfo_endpoint
  userinfo_requirement is required if discovery is disabled andthe 'openid' scope is not requested (type=value_error)

reivilibre

This is exciting to see, frankly :-). Should have been done 20 years ago!

Loose-by-default types is not a design decision I'm the biggest fan of, but it beats having to maintain our own library and it's just a matter of code review, so I say I'm happy with this.

I think changing 400 → 404 is out of scope for this change.
Save for that and a couple of what I believe to be slip-ups, I'd be happy to land this change, so it's a LGTM in principle (unsure why you have it as a draft, presumably you have more in mind).

synapse/rest/client/account.py

synapse/rest/client/models.py

This reverts commit 9a47994.

synapse/rest/client/account.py

DMRobertson · 2022-07-20T13:04:42Z

(unsure why you have it as a draft, presumably you have more in mind).

I had grand plans to cover all of the endpoints in /client/account. I only got about halfway through this file before getting to this stage. Probably makes more sense to file additional PRs for each endpoint.

DMRobertson · 2022-08-08T17:04:18Z

synapse/rest/client/models.py

+    id_access_token: Optional[StrictStr]
+    id_server: Optional[StrictStr]
+    next_link: Optional[StrictStr]


(Edited: I was wrong about this, and I have corrected the comment.)

Remark. The word "optional" has two senses:

nullable: Optional[T] might hold None, or a T

not required: the field may be omitted, to be replaced by a default value. That value might (or might not) be None.

The senses are orthogonal, and supported by Pydantic. To express "nullable, but required" one uses an elipsis.

>>> class M(BaseModel): ... a: StrictInt ... b: Optional[StrictInt] = ... ... c: StrictInt = 123 ... d: Optional[StrictInt] = None ... e: Optional[StrictInt] = 123 ... >>> M.parse_obj({"a": 1}) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "pydantic/main.py", line 521, in pydantic.main.BaseModel.parse_obj File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__ pydantic.error_wrappers.ValidationError: 1 validation error for M b field required (type=value_error.missing) >>> M.parse_obj({"a": 1, "b": None}) M(a=1, b=None, c=123, d=None, e=123)

Writing x: Optional[T] without a default behaves like case 4. (Which is what I think we want 99 times out of 100.)

>>> class M(BaseModel): ... x: Optional[int] ... >>> M.parse_obj({}) M(x=None) >>> M.parse_obj({"x": None}) M(x=None) >>> M.parse_obj({"x": 234}) M(x=234)

synapse/rest/client/models.py

DMRobertson · 2022-08-08T18:15:19Z

Since last time:

relax to Pydantic 1.7.4 for better distro compatibility
more use of StrictStr rather than str, (related: 13336)
given this a proper changelog
enforced that id_access_token is required when id_server is provided
changed Pydantic's defaults: ignore extra fields, create frozen instances

richvdh

lgtm. we should ship this!

richvdh · 2022-08-09T10:57:13Z

synapse/http/servlet.py

+    try:
+        instance = model_type.parse_obj(content)
+    except ValidationError as e:
+        raise SynapseError(HTTPStatus.BAD_REQUEST, str(e), errcode=Codes.BAD_JSON)


It might be nice to define a new subclass of SynapseError for this. Not going to block this PR on it though.

Oh, so we could catch a validation failure specifically? Happy to revisit this i the future if it turns out we need it

richvdh · 2022-08-09T10:58:10Z

changelog.d/13188.misc

@@ -0,0 +1 @@
+Improve validation of some account-related REST endpoints.


I think this is a feature, and we should name the endpoints in question.

Thanks, bd917d3

richvdh · 2022-08-09T11:00:42Z

synapse/rest/client/models.py

+from synapse.util.threepids import validate_email
+
+
+class AuthenticationData(SynapseBaseModel):


probably worth giving this a docstring to say where it is used. I think it's part of the request for user-interactive auth?

I snuck this into c404f8f by mistake, but oh well

richvdh · 2022-08-09T11:02:44Z

synapse/types.py

@@ -920,3 +921,22 @@ class UserProfile(TypedDict):
 class RetentionPolicy:
    min_lifetime: Optional[int] = None
    max_lifetime: Optional[int] = None
+
+
+class SynapseBaseModel(BaseModel):


should this really be here, rather than somewhere in synapse.rest?

I'm conscious that synapse.types is imported everywhere. If we ever want to unpick our circular imports, adding more stuff to it isn't going to help.

FWIW I'd naively expect that a name like synapse.types ought to be something that you can import everywhere.

I considered putting in rest. But we might also want to use this for parsing the config. Though having said that I'm not sure if "ignore unknown fields" makes sense for config.

Happy to move it to rest now rather than anticipating the future.

synapse/http/servlet.py

Really need a lint for this, see e.g. #13336

and give it a rename to be a bit more specific

richvdh

looks great, thanks!

Synapse 1.66.0rc1 (2022-08-23) ============================== This release removes the ability for homeservers to delegate email ownership verification and password reset confirmation to identity servers. This removal was originally planned for Synapse 1.64, but was later deferred until now. See the [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details. Features -------- - Improve validation of request bodies for the following client-server API endpoints: [`/account/password`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpassword), [`/account/password/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpasswordemailrequesttoken), [`/account/deactivate`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountdeactivate) and [`/account/3pid/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidemailrequesttoken). ([\#13188](#13188), [\#13563](#13563)) - Add forgotten status to [Room Details Admin API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#room-details-api). ([\#13503](#13503)) - Add an experimental implementation for [MSC3852 (Expose user agents on `Device`)](matrix-org/matrix-spec-proposals#3852). ([\#13549](#13549)) - Add `org.matrix.msc2716v4` experimental room version with updated content fields. Part of [MSC2716 (Importing history)](matrix-org/matrix-spec-proposals#2716). ([\#13551](#13551)) - Add support for compression to federation responses. ([\#13537](#13537)) - Improve performance of sending messages in rooms with thousands of local users. ([\#13522](#13522), [\#13547](#13547)) Bugfixes -------- - Faster room joins: make `/joined_members` block whilst the room is partial stated. ([\#13514](#13514)) - Fix a bug introduced in Synapse 1.21.0 where the [`/event_reports` Admin API](https://matrix-org.github.io/synapse/develop/admin_api/event_reports.html) could return a total count which was larger than the number of results you can actually query for. ([\#13525](#13525)) - Fix a bug introduced in Synapse 1.52.0 where sending server notices fails if `max_avatar_size` or `allowed_avatar_mimetypes` is set and not `system_mxid_avatar_url`. ([\#13566](#13566)) - Fix a bug where the `opentracing.force_tracing_for_users` config option would not apply to [`/sendToDevice`](https://spec.matrix.org/v1.3/client-server-api/#put_matrixclientv3sendtodeviceeventtypetxnid) and [`/keys/upload`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3keysupload) requests. ([\#13574](#13574)) Improved Documentation ---------------------- - Add `openssl` example for generating registration HMAC digest. ([\#13472](#13472)) - Tidy up Synapse's README. ([\#13491](#13491)) - Document that event purging related to the `redaction_retention_period` config option is executed only every 5 minutes. ([\#13492](#13492)) - Add a warning to retention documentation regarding the possibility of database corruption. ([\#13497](#13497)) - Document that the `DOCKER_BUILDKIT=1` flag is needed to build the docker image. ([\#13515](#13515)) - Add missing links in `user_consent` section of configuration manual. ([\#13536](#13536)) - Fix the doc and some warnings that were referring to the nonexistent `custom_templates_directory` setting (instead of `custom_template_directory`). ([\#13538](#13538)) Deprecations and Removals ------------------------- - Remove the ability for homeservers to delegate email ownership verification and password reset confirmation to identity servers. See [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details. Internal Changes ---------------- - Update the rejected state of events during de-partial-stating. ([\#13459](#13459)) - Avoid blocking lazy-loading `/sync`s during partial joins due to remote memberships. Pull remote memberships from auth events instead of the room state. ([\#13477](#13477)) - Refuse to start when faster joins is enabled on a deployment with workers, since worker configurations are not currently supported. ([\#13531](#13531)) - Allow use of both `@trace` and `@tag_args` stacked on the same function. ([\#13453](#13453)) - Instrument the federation/backfill part of `/messages` for understandable traces in Jaeger. ([\#13489](#13489)) - Instrument `FederationStateIdsServlet` (`/state_ids`) for understandable traces in Jaeger. ([\#13499](#13499), [\#13554](#13554)) - Track HTTP response times over 10 seconds from `/messages` (`synapse_room_message_list_rest_servlet_response_time_seconds`). ([\#13533](#13533)) - Add metrics to track how the rate limiter is affecting requests (sleep/reject). ([\#13534](#13534), [\#13541](#13541)) - Add metrics to time how long it takes us to do backfill processing (`synapse_federation_backfill_processing_before_time_seconds`, `synapse_federation_backfill_processing_after_time_seconds`). ([\#13535](#13535), [\#13584](#13584)) - Add metrics to track rate limiter queue timing (`synapse_rate_limit_queue_wait_time_seconds`). ([\#13544](#13544)) - Update metrics to track `/messages` response time by room size. ([\#13545](#13545)) - Refactor methods in `synapse.api.auth.Auth` to use `Requester` objects everywhere instead of user IDs. ([\#13024](#13024)) - Clean-up tests for notifications. ([\#13471](#13471)) - Add some miscellaneous comments to document sync, especially around `compute_state_delta`. ([\#13474](#13474)) - Use literals in place of `HTTPStatus` constants in tests. ([\#13479](#13479), [\#13488](#13488)) - Add comments about how event push actions are rotated. ([\#13485](#13485)) - Modify HTML template content to better support mobile devices' screen sizes. ([\#13493](#13493)) - Add a linter script which will reject non-strict types in Pydantic models. ([\#13502](#13502)) - Reduce the number of tests using legacy TCP replication. ([\#13543](#13543)) - Allow specifying additional request fields when using the `HomeServerTestCase.login` helper method. ([\#13549](#13549)) - Make `HomeServerTestCase` load any configured homeserver modules automatically. ([\#13558](#13558))

Synapse 1.66.0 (2022-08-31) =========================== No significant changes since 1.66.0rc2. This release removes the ability for homeservers to delegate email ownership verification and password reset confirmation to identity servers. This removal was originally planned for Synapse 1.64, but was later deferred until now. See the [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details. Deployments with multiple workers should note that the direct TCP replication configuration was deprecated in Synapse v1.18.0 and will be removed in Synapse v1.67.0. In particular, the TCP `replication` [listener](https://matrix-org.github.io/synapse/v1.66/usage/configuration/config_documentation.html#listeners) type (not to be confused with the `replication` resource on the `http` listener type) and the `worker_replication_port` config option will be removed . To migrate to Redis, add the [`redis` config](https://matrix-org.github.io/synapse/v1.66/workers.html#shared-configuration), then remove the TCP `replication` listener from config of the master and `worker_replication_port` from worker config. Note that a HTTP listener with a `replication` resource is still required. See the [worker documentation](https://matrix-org.github.io/synapse/v1.66/workers.html) for more details. Synapse 1.66.0rc2 (2022-08-30) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.66.0rc1 where the new rate limit metrics were misreported (`synapse_rate_limit_sleep_affected_hosts`, `synapse_rate_limit_reject_affected_hosts`). ([\matrix-org#13649](matrix-org#13649)) Synapse 1.66.0rc1 (2022-08-23) ============================== Features -------- - Improve validation of request bodies for the following client-server API endpoints: [`/account/password`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpassword), [`/account/password/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpasswordemailrequesttoken), [`/account/deactivate`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountdeactivate) and [`/account/3pid/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidemailrequesttoken). ([\matrix-org#13188](matrix-org#13188), [\matrix-org#13563](matrix-org#13563)) - Add forgotten status to [Room Details Admin API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#room-details-api). ([\matrix-org#13503](matrix-org#13503)) - Add an experimental implementation for [MSC3852 (Expose user agents on `Device`)](matrix-org/matrix-spec-proposals#3852). ([\matrix-org#13549](matrix-org#13549)) - Add `org.matrix.msc2716v4` experimental room version with updated content fields. Part of [MSC2716 (Importing history)](matrix-org/matrix-spec-proposals#2716). ([\matrix-org#13551](matrix-org#13551)) - Add support for compression to federation responses. ([\matrix-org#13537](matrix-org#13537)) - Improve performance of sending messages in rooms with thousands of local users. ([\matrix-org#13522](matrix-org#13522), [\matrix-org#13547](matrix-org#13547)) Bugfixes -------- - Faster room joins: make `/joined_members` block whilst the room is partial stated. ([\matrix-org#13514](matrix-org#13514)) - Fix a bug introduced in Synapse 1.21.0 where the [`/event_reports` Admin API](https://matrix-org.github.io/synapse/develop/admin_api/event_reports.html) could return a total count which was larger than the number of results you can actually query for. ([\matrix-org#13525](matrix-org#13525)) - Fix a bug introduced in Synapse 1.52.0 where sending server notices fails if `max_avatar_size` or `allowed_avatar_mimetypes` is set and not `system_mxid_avatar_url`. ([\matrix-org#13566](matrix-org#13566)) - Fix a bug where the `opentracing.force_tracing_for_users` config option would not apply to [`/sendToDevice`](https://spec.matrix.org/v1.3/client-server-api/#put_matrixclientv3sendtodeviceeventtypetxnid) and [`/keys/upload`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3keysupload) requests. ([\matrix-org#13574](matrix-org#13574)) Improved Documentation ---------------------- - Add `openssl` example for generating registration HMAC digest. ([\matrix-org#13472](matrix-org#13472)) - Tidy up Synapse's README. ([\matrix-org#13491](matrix-org#13491)) - Document that event purging related to the `redaction_retention_period` config option is executed only every 5 minutes. ([\matrix-org#13492](matrix-org#13492)) - Add a warning to retention documentation regarding the possibility of database corruption. ([\matrix-org#13497](matrix-org#13497)) - Document that the `DOCKER_BUILDKIT=1` flag is needed to build the docker image. ([\matrix-org#13515](matrix-org#13515)) - Add missing links in `user_consent` section of configuration manual. ([\matrix-org#13536](matrix-org#13536)) - Fix the doc and some warnings that were referring to the nonexistent `custom_templates_directory` setting (instead of `custom_template_directory`). ([\matrix-org#13538](matrix-org#13538)) Deprecations and Removals ------------------------- - Remove the ability for homeservers to delegate email ownership verification and password reset confirmation to identity servers. See [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details. Internal Changes ---------------- - Update the rejected state of events during de-partial-stating. ([\matrix-org#13459](matrix-org#13459)) - Avoid blocking lazy-loading `/sync`s during partial joins due to remote memberships. Pull remote memberships from auth events instead of the room state. ([\matrix-org#13477](matrix-org#13477)) - Refuse to start when faster joins is enabled on a deployment with workers, since worker configurations are not currently supported. ([\matrix-org#13531](matrix-org#13531)) - Allow use of both `@trace` and `@tag_args` stacked on the same function. ([\matrix-org#13453](matrix-org#13453)) - Instrument the federation/backfill part of `/messages` for understandable traces in Jaeger. ([\matrix-org#13489](matrix-org#13489)) - Instrument `FederationStateIdsServlet` (`/state_ids`) for understandable traces in Jaeger. ([\matrix-org#13499](matrix-org#13499), [\matrix-org#13554](matrix-org#13554)) - Track HTTP response times over 10 seconds from `/messages` (`synapse_room_message_list_rest_servlet_response_time_seconds`). ([\matrix-org#13533](matrix-org#13533)) - Add metrics to track how the rate limiter is affecting requests (sleep/reject). ([\matrix-org#13534](matrix-org#13534), [\matrix-org#13541](matrix-org#13541)) - Add metrics to time how long it takes us to do backfill processing (`synapse_federation_backfill_processing_before_time_seconds`, `synapse_federation_backfill_processing_after_time_seconds`). ([\matrix-org#13535](matrix-org#13535), [\matrix-org#13584](matrix-org#13584)) - Add metrics to track rate limiter queue timing (`synapse_rate_limit_queue_wait_time_seconds`). ([\matrix-org#13544](matrix-org#13544)) - Update metrics to track `/messages` response time by room size. ([\matrix-org#13545](matrix-org#13545)) - Refactor methods in `synapse.api.auth.Auth` to use `Requester` objects everywhere instead of user IDs. ([\matrix-org#13024](matrix-org#13024)) - Clean-up tests for notifications. ([\matrix-org#13471](matrix-org#13471)) - Add some miscellaneous comments to document sync, especially around `compute_state_delta`. ([\matrix-org#13474](matrix-org#13474)) - Use literals in place of `HTTPStatus` constants in tests. ([\matrix-org#13479](matrix-org#13479), [\matrix-org#13488](matrix-org#13488)) - Add comments about how event push actions are rotated. ([\matrix-org#13485](matrix-org#13485)) - Modify HTML template content to better support mobile devices' screen sizes. ([\matrix-org#13493](matrix-org#13493)) - Add a linter script which will reject non-strict types in Pydantic models. ([\matrix-org#13502](matrix-org#13502)) - Reduce the number of tests using legacy TCP replication. ([\matrix-org#13543](matrix-org#13543)) - Allow specifying additional request fields when using the `HomeServerTestCase.login` helper method. ([\matrix-org#13549](matrix-org#13549)) - Make `HomeServerTestCase` load any configured homeserver modules automatically. ([\matrix-org#13558](matrix-org#13558)) # -----BEGIN PGP SIGNATURE----- # # iQGzBAABCgAdFiEEWMTnW8Z8khaaf90R+84KzgcyGG8FAmMPT8QACgkQ+84Kzgcy # GG9CUAv+Pv/iDpE2jKlV7zQ/cagaKCGsFK5jy0+K9Wr215nP89tuhU37bJXsgvVu # GP3A8k1c/ENPhXwYHLCnnxV3jick1FuVE0W6h0j2PMYeIGNCQhDswytnsQO4JExg # fGLL4ygCzpe8bFX9+mhIM4z8xkZjZX3lIa8CN2LtRLIo0m7qoT1ZWqdt7kAjj5yL # XMk+3Y1yq/Y4SHHqgKurBNdwNcwnv7ynchWxTYa12WVTINt26dLV0Syk3p8u2SLl # 5YNzcDs2TAM7+VxAu7E0AQl426+Ufi122Oj1ZBUG2FxTPLH8Xr18cN2M/at6WxoX # 8pOkGiuahKKvahw1iCoHAGIC66gFIPxBE9xW4R2SKrQtG4sDuKJI0kvunRV8+cy5 # TuJ9cmdDmJR2vj3P3OULqLXGkWsGNJqfZZF8OWkHEI8LUIXZLrAZocFtlonkr9rV # Y8r8LxL8Id1rbHAnCXcJnYdaJ6ol0RIObDFpitY/D8BDUONVw/byeOyAEkq/XPrZ # Ke/9K8sy # =eg1L # -----END PGP SIGNATURE----- # gpg: Signature made Wed Aug 31 13:10:44 2022 BST # gpg: using RSA key 58C4E75BC67C92169A7FDD11FBCE0ACE0732186F # gpg: Can't check signature: No public key # Conflicts: # synapse/api/auth.py # synapse/push/baserules.py # synapse/push/bulk_push_rule_evaluator.py # synapse/push/push_rule_evaluator.py # synapse/storage/databases/main/event_push_actions.py # tests/server_notices/test_resource_limits_server_notices.py

packaging changes: - Depend on pydantic summary of upstream changes: Synapse 1.66.0 (2022-08-31) =========================== This release removes the ability for homeservers to delegate email ownership verification and password reset confirmation to identity servers. This removal was originally planned for Synapse 1.64, but was later deferred until now. See the [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details. Deployments with multiple workers should note that the direct TCP replication configuration was deprecated in Synapse v1.18.0 and will be removed in Synapse v1.67.0. In particular, the TCP `replication` [listener](https://matrix-org.github.io/synapse/v1.66/usage/configuration/config_documentation.html#listeners) type (not to be confused with the `replication` resource on the `http` listener type) and the `worker_replication_port` config option will be removed . To migrate to Redis, add the [`redis` config](https://matrix-org.github.io/synapse/v1.66/workers.html#shared-configuration), then remove the TCP `replication` listener from config of the master and `worker_replication_port` from worker config. Note that a HTTP listener with a `replication` resource is still required. See the [worker documentation](https://matrix-org.github.io/synapse/v1.66/workers.html) for more details. Features -------- - Improve validation of request bodies for the following client-server -API endpoints: -[`/account/password`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountpassword), -[`/account/password/email/requestToken`](https://spec.matrix.org/v1.3/client-server -api/#post_matrixclientv3accountpasswordemailrequesttoken), -[`/account/deactivate`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3accountdeactivate) -and -[`/account/3pid/email/requestToken`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixclientv3account3pidemailrequesttoken). ([\#13188](matrix-org/synapse#13188), -[\#13563](matrix-org/synapse#13563)) - Add forgotten status to [Room Details Admin API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#room-details-api). ([\#13503](matrix-org/synapse#13503)) - Add an experimental implementation for [MSC3852 (Expose user agents on `Device`)](https://github.com/matrix-org/matrix-spec-proposals/pu ll/3852). ([\#13549](matrix-org/synapse#13549)) - Add `org.matrix.msc2716v4` experimental room version with updated content fields. Part of [MSC2716 (Importing history)](matrix-org/matrix-spec-proposals#2716). ([\#13551](matrix-org/synapse#13551)) - Add support for compression to federation responses. ([\#13537](matrix-org/synapse#13537)) - Improve performance of sending messages in rooms with thousands of local users. ([\#13522](matrix-org/synapse#13522), [\#13547](matrix-org/synapse#13547)) Deprecations and Removals ------------------------- - Remove the ability for homeservers to delegate email ownership verification and password reset confirmation to identity servers. See [upgrade notes](https://matrix-org.github.io/synapse/v1.66/upgrade.html#upgrading-to-v1660) for more details.

David Robertson added 4 commits July 4, 2022 19:12

Lock pydantic

c932ed5

Use pydantic mypy plugin

c40edbd

Helper for validating a json body with pydantic

52b0ef3

Validate client/account/deactivate

87a6e79

DMRobertson force-pushed the dmr/rest/client/account branch from 1c62f29 to 0ad81f6 Compare July 5, 2022 17:00

David Robertson added 6 commits July 5, 2022 18:51

Validate /client/account/password

c7caf5b

Changelog

3319732

Better errcode when pw reset emails are disabled

9a47994

Validate /client/account/passsword/email/requestToken

0a87861

move out RequestToken body to models

3e2d003

it's about to get reused

Validate /client/account/3pid/email/requestToken

ba71e02

DMRobertson force-pushed the dmr/rest/client/account branch from a37c103 to ba71e02 Compare July 5, 2022 17:52

Ooops, use Strict* types for EmailRequestTokenBody

e4926c2

reivilibre requested review from reivilibre and a team and removed request for a team July 5, 2022 18:07

clokep reviewed Jul 5, 2022

View reviewed changes

synapse/rest/client/account.py Outdated Show resolved Hide resolved

clokep reviewed Jul 5, 2022

View reviewed changes

synapse/rest/client/account.py Show resolved Hide resolved

Fix linter whoops

b81faf4

dklimpel reviewed Jul 5, 2022

View reviewed changes

synapse/rest/client/account.py Outdated Show resolved Hide resolved

reivilibre reviewed Jul 6, 2022

View reviewed changes

synapse/rest/client/account.py Outdated Show resolved Hide resolved

synapse/rest/client/account.py Outdated Show resolved Hide resolved

synapse/rest/client/account.py Show resolved Hide resolved

synapse/rest/client/models.py Outdated Show resolved Hide resolved

Revert "Better errcode when pw reset emails are disabled"

dc1f188

This reverts commit 9a47994.

DMRobertson commented Jul 20, 2022

View reviewed changes

synapse/rest/client/account.py Outdated Show resolved Hide resolved

David Robertson added 2 commits July 20, 2022 13:50

Move the models into the servlet classes

61f8f25

Revert another status code change.

296467e

DMRobertson force-pushed the dmr/rest/client/account branch from ad801d9 to 296467e Compare July 20, 2022 12:50

DMRobertson added this to the Validate data passed to REST endpoints milestone Jul 20, 2022

DMRobertson mentioned this pull request Jul 20, 2022

Prototype a means to detect non-strict types in Pydantic models at lint time #13336

Closed

DMRobertson commented Aug 8, 2022

View reviewed changes

synapse/rest/client/models.py Outdated Show resolved Hide resolved

David Robertson added 3 commits August 8, 2022 18:51

Require access token if id server is provided

df1f840

Ignore unused fields; create frozen instances

c4074aa

Fix mypy

fefb0a8

Fix lint

4dbafed

DMRobertson requested review from a team, reivilibre and richvdh August 8, 2022 18:17

richvdh approved these changes Aug 9, 2022

View reviewed changes

David Robertson added 3 commits August 10, 2022 14:27

Update changelog

bd917d3

Add a missing strict=True.

f8f6037

Really need a lint for this, see e.g. #13336

Move the BaseModel subclass to synapse.rest

c404f8f

and give it a rename to be a bit more specific

DMRobertson requested a review from richvdh August 10, 2022 15:19

David Robertson added 2 commits August 10, 2022 16:40

Try to break circular import?

1597977

Fix lint durr

22aee7a

DMRobertson mentioned this pull request Aug 11, 2022

Reject non-strict types in Pydantic models #13502

Merged

richvdh approved these changes Aug 15, 2022

View reviewed changes

DMRobertson enabled auto-merge (squash) August 15, 2022 19:02

DMRobertson merged commit d642ce4 into develop Aug 15, 2022

DMRobertson deleted the dmr/rest/client/account branch August 15, 2022 19:05

DMRobertson mentioned this pull request Aug 19, 2022

/account/deactivate is 500ing #13560

Closed

reivilibre mentioned this pull request Aug 19, 2022

Fix validation problem that occurs when a user tries to deactivate their account or change their password. #13563

Merged

clokep mentioned this pull request Sep 7, 2023

Avoid temporary storage of sensitive information. #16272

Merged

clokep mentioned this pull request Sep 21, 2023

Add missing license header. #16359

Merged

matrixbot mentioned this pull request Dec 21, 2023

Investigate if we can improve Pydantic's error messages for user consumption element-hq/synapse#13337

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use Pydantic to systematically validate a first batch of endpoints in `synapse.rest.client.account`. #13188

Use Pydantic to systematically validate a first batch of endpoints in `synapse.rest.client.account`. #13188

DMRobertson commented Jul 5, 2022 •

edited

Loading

DMRobertson commented Jul 5, 2022

reivilibre left a comment

DMRobertson commented Jul 20, 2022

DMRobertson Aug 8, 2022 •

edited

Loading

DMRobertson commented Aug 8, 2022

richvdh left a comment

richvdh Aug 9, 2022

DMRobertson Aug 10, 2022

richvdh Aug 9, 2022

DMRobertson Aug 10, 2022

richvdh Aug 9, 2022

DMRobertson Aug 10, 2022

richvdh Aug 9, 2022

DMRobertson Aug 10, 2022

DMRobertson Aug 10, 2022

richvdh left a comment

		@@ -0,0 +1 @@
		Improve validation of some account-related REST endpoints.

		from synapse.util.threepids import validate_email


		class AuthenticationData(SynapseBaseModel):

Use Pydantic to systematically validate a first batch of endpoints in synapse.rest.client.account. #13188

Use Pydantic to systematically validate a first batch of endpoints in synapse.rest.client.account. #13188

Conversation

DMRobertson commented Jul 5, 2022 • edited Loading

DMRobertson commented Jul 5, 2022

reivilibre left a comment

Choose a reason for hiding this comment

DMRobertson commented Jul 20, 2022

DMRobertson Aug 8, 2022 • edited Loading

Choose a reason for hiding this comment

DMRobertson commented Aug 8, 2022

richvdh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

Use Pydantic to systematically validate a first batch of endpoints in `synapse.rest.client.account`. #13188

Use Pydantic to systematically validate a first batch of endpoints in `synapse.rest.client.account`. #13188

DMRobertson commented Jul 5, 2022 •

edited

Loading

DMRobertson Aug 8, 2022 •

edited

Loading