Pull latest changes from dendrite fork (matrix-org#193)

* Verify that the user ID for registration matches the spec, and the auth data (#10)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* pull latest from dendrite-fork subtree

* refresh latest dendrite main

* pull dendrite subtree and resolve merge conflicts

* check that userID matches the signed message

* verify that the user ID for registration is CAIP-10 compliant and MXID compliant

* removed space

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Fix nats.go commit (matrix-org#2540)

Signed-off-by: Jean Lucas <jean@4ray.co>

* Don't return `end` if there are not more messages (matrix-org#2542)

* Be more spec compliant

* Move lazyLoadMembers to own method

* Return an error if trying to invite a malformed user ID (matrix-org#2543)

* Add `evacuateUser` endpoint, use it when deactivating accounts (matrix-org#2545)

* Add `evacuateUser` endpoint, use it when deactivating accounts

* Populate the API

* Clean up user devices when deactivating

* Include invites, delete pushers

* Silence presence logs (matrix-org#2547)

* Blacklist `Guest users can join guest_access rooms` test until it can be investigated

* Disable WebAssembly builds for now

* Try to fix backfilling (matrix-org#2548)

* Try to fix backfilling

* Return start/end to not confuse clients

* Update GMSL

* Update GMSL

* Roomserver producers package (matrix-org#2546)

* Give the roomserver a producers package

* Change init point

* Populate ACLs API

* Fix build issues

* `RoomEventProducer` naming

* Version 0.8.9 (matrix-org#2549)

* Version 0.8.9

* Update changelog

* Takwaiw/fix concurrent registration bug (#12)

* fix concurrent registration bug. Rename decentralizedid

* remove unused module

* add regressed test to blacklist

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Test_UserStatistics Fix expected results to match observed results

* Takwaiw/dendrite publickey (#2)

* Implementation of MSC 3782 Add publickey login as a new auth type.

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Implement EIP-4361 sign in with Ethereum (#5)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* refresh latest dendrite main

* dendrite implementation of eip-4361

* simplify nonce generation

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Use rand.Seed to seed the random function generator (#6)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* use rand.Seed to seed the random function

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Create session ID during registration (#8)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* pull latest from dendrite-fork subtree

* refresh latest dendrite main

* Create session ID during registration

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Verify that the user ID for registration matches the spec, and the auth data (#10)

* Blacklist some sytest tests that are failing in our environment

* Commenting out test that isn't reliably passing or failing, probably a race

* refresh latest dendrite main

* pull latest from dendrite-fork subtree

* refresh latest dendrite main

* pull dendrite subtree and resolve merge conflicts

* check that userID matches the signed message

* verify that the user ID for registration is CAIP-10 compliant and MXID compliant

* removed space

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* Takwaiw/fix concurrent registration bug (#12)

* fix concurrent registration bug. Rename decentralizedid

* remove unused module

* add regressed test to blacklist

Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>

* removed unused module

* feat+fix: Ignore unknown keys and verify required fields are present in appservice registration files (matrix-org#2550)

* fix: ignore unknown keys in appservice configs

fixes matrix-org#1567

* feat: verify required fields in appservice configs

* Use new testrig for key changes tests (matrix-org#2552)

* Use new testrig for tests

* Log the error message

* Fix QuerySharedUsers for the SyncAPI keychange consumer (matrix-org#2554)

* Make more use of base.BaseDendrite

* Fix QuerySharedUsers if no UserIDs are supplied

* Return clearer error when no state NID exists for an event (matrix-org#2555)

* Wrap error from `SnapshotNIDFromEventID`

* Hopefully fix read receipts timestamps (matrix-org#2557)

This should avoid coercions between signed and unsigned ints which might fix problems like `sql: converting argument $5 type: uint64 values with high bit set are not supported`.

* fix concurrency issue when checking session ID (#14)

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>

* merge latest changes from dendrite main (#15)

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>

* Login and Register tests for public key ethereum (#16)

* TestLoginPublicKeyNewSession

* use asserts

* setup, test, asserts

* TestLoginPublicKeyValidAuthTypeMissingSession

* invalid session id test

* create a helper newSession function

* TestLoginPublicKeyEthereumMissingUserId

* TestLoginPublicKeyEthereumAccountNotAvailable

* TestLoginPublicKeyEthereumInvalidUserId

* createEip4361TestMessage

* TestLoginPublicKeyEthereumMissingSignature

* TestLoginPublicKeyEthereum

* re-enable all publickey signin tests

* move common publickey test util to its own file

* register_public_key.go stub

* refactored common ethereum test helpers to its own folder

* refactor test helpers

* return error in test helpers

* fix regressions with ServerName

* TestRegistrationUnimplementedAlgo

* TestNewRegistration

* TestNewRegistrationSession

* verify new login session

* remove assert

* perform account creation

* TestRegisterEthereum

* Enable all tests

* move helper functions into test file

Co-authored-by: Tak Wai Wong <tak@hntlabs.com>

Co-authored-by: Brian Meek <brian@hntlabs.com>
Co-authored-by: Tak Wai Wong <takwaiw@gmail.com>
Co-authored-by: Jean Lucas <jean@4ray.co>
Co-authored-by: Till <2353100+S7evinK@users.noreply.github.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: Tak Wai Wong <tak@hntlabs.com>
Co-authored-by: Kabir Kwatra <kabir@kwatra.me>

.github/workflows/dendrite.yml

@@ -24,7 +24,7 @@ jobs:
       - name: Install Go
         uses: actions/setup-go@v2
         with:
-          go-version: 1.16
+          go-version: 1.18
 
       - uses: actions/cache@v2
         with:
@@ -97,7 +97,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        go: ["1.16", "1.17", "1.18"]
+        go: ["1.18"]
     steps:
       - uses: actions/checkout@v3
       - name: Setup go
@@ -127,7 +127,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        go: ["1.16", "1.17", "1.18"]
+        go: ["1.18"]
         goos: ["linux"]
         goarch: ["amd64", "386"]
     steps:
@@ -160,7 +160,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        go: ["1.16", "1.17", "1.18"]
+        go: ["1.18"]
         goos: ["windows"]
         goarch: ["amd64"]
     steps:
@@ -209,7 +209,7 @@ jobs:
       - name: Setup go
         uses: actions/setup-go@v2
         with:
-          go-version: "1.16"
+          go-version: "1.18"
       - uses: actions/cache@v3
         with:
           path: |

README.md

@@ -36,7 +36,7 @@ If you have further questions, please take a look at [our FAQ](docs/FAQ.md) or j
 
 ## Requirements
 
-To build Dendrite, you will need Go 1.16 or later.
+To build Dendrite, you will need Go 1.18 or later.
 
 For a usable federating Dendrite deployment, you will also need:
 

build/gobind-pinecone/monolith.go

@@ -239,7 +239,7 @@ func (m *DendriteMonolith) Start() {
 	m.PineconeRouter = pineconeRouter.NewRouter(logrus.WithField("pinecone", "router"), sk, false)
 	m.PineconeQUIC = pineconeSessions.NewSessions(logrus.WithField("pinecone", "sessions"), m.PineconeRouter, []string{"matrix"})
 	m.PineconeMulticast = pineconeMulticast.NewMulticast(logrus.WithField("pinecone", "multicast"), m.PineconeRouter)
-	m.PineconeManager = pineconeConnections.NewConnectionManager(m.PineconeRouter)
+	m.PineconeManager = pineconeConnections.NewConnectionManager(m.PineconeRouter, nil)
 
 	prefix := hex.EncodeToString(pk)
 	cfg := &config.Dendrite{}

build/scripts/Complement.Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.16-stretch as build
+FROM golang:1.18-stretch as build
 RUN apt-get update && apt-get install -y sqlite3
 WORKDIR /build
 
@@ -27,6 +27,6 @@ EXPOSE 8008 8448
 # At runtime, generate TLS cert based on the CA now mounted at /ca
 # At runtime, replace the SERVER_NAME with what we are told
 CMD ./generate-keys --server $SERVER_NAME --tls-cert server.crt --tls-key server.key --tls-authority-cert /complement/ca/ca.crt --tls-authority-key /complement/ca/ca.key && \
- ./generate-config -server $SERVER_NAME --ci > dendrite.yaml && \
- cp /complement/ca/ca.crt /usr/local/share/ca-certificates/ && update-ca-certificates && \
- ./dendrite-monolith-server --really-enable-open-registration --tls-cert server.crt --tls-key server.key --config dendrite.yaml -api=${API:-0}
+    ./generate-config -server $SERVER_NAME --ci > dendrite.yaml && \
+    cp /complement/ca/ca.crt /usr/local/share/ca-certificates/ && update-ca-certificates && \
+    ./dendrite-monolith-server --really-enable-open-registration --tls-cert server.crt --tls-key server.key --config dendrite.yaml -api=${API:-0}

build/scripts/ComplementLocal.Dockerfile

@@ -6,7 +6,7 @@
 #
 # Use these mounts to make use of this dockerfile:
 # COMPLEMENT_HOST_MOUNTS='/your/local/dendrite:/dendrite:ro;/your/go/path:/go:ro'
-FROM golang:1.16-stretch
+FROM golang:1.18-stretch
 RUN apt-get update && apt-get install -y sqlite3
 
 WORKDIR /runtime
@@ -16,24 +16,24 @@ EXPOSE 8008 8448
 
 # This script compiles Dendrite for us.
 RUN echo '\
-#!/bin/bash -eux \n\
-if test -f "/runtime/dendrite-monolith-server"; then \n\
+    #!/bin/bash -eux \n\
+    if test -f "/runtime/dendrite-monolith-server"; then \n\
     echo "Skipping compilation; binaries exist" \n\
     exit 0 \n\
-fi \n\
-cd /dendrite \n\
-go build -v -o /runtime /dendrite/cmd/dendrite-monolith-server \n\
-' > compile.sh && chmod +x compile.sh
+    fi \n\
+    cd /dendrite \n\
+    go build -v -o /runtime /dendrite/cmd/dendrite-monolith-server \n\
+    ' > compile.sh && chmod +x compile.sh
 
 # This script runs Dendrite for us. Must be run in the /runtime directory.
 RUN echo '\
-#!/bin/bash -eu \n\
-./generate-keys --private-key matrix_key.pem \n\
-./generate-keys --server $SERVER_NAME --tls-cert server.crt --tls-key server.key --tls-authority-cert /complement/ca/ca.crt --tls-authority-key /complement/ca/ca.key \n\
-./generate-config -server $SERVER_NAME --ci > dendrite.yaml \n\
-cp /complement/ca/ca.crt /usr/local/share/ca-certificates/ && update-ca-certificates \n\
-./dendrite-monolith-server --really-enable-open-registration --tls-cert server.crt --tls-key server.key --config dendrite.yaml \n\
-' > run.sh && chmod +x run.sh
+    #!/bin/bash -eu \n\
+    ./generate-keys --private-key matrix_key.pem \n\
+    ./generate-keys --server $SERVER_NAME --tls-cert server.crt --tls-key server.key --tls-authority-cert /complement/ca/ca.crt --tls-authority-key /complement/ca/ca.key \n\
+    ./generate-config -server $SERVER_NAME --ci > dendrite.yaml \n\
+    cp /complement/ca/ca.crt /usr/local/share/ca-certificates/ && update-ca-certificates \n\
+    ./dendrite-monolith-server --really-enable-open-registration --tls-cert server.crt --tls-key server.key --config dendrite.yaml \n\
+    ' > run.sh && chmod +x run.sh
 
 
 WORKDIR /cache

build/scripts/ComplementPostgres.Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.16-stretch as build
+FROM golang:1.18-stretch as build
 RUN apt-get update && apt-get install -y postgresql
 WORKDIR /build
 
@@ -9,16 +9,16 @@ RUN sed -i "s%127.0.0.1/32            md5%127.0.0.1/32            trust%g" /etc/
 
 # This entry script starts postgres, waits for it to be up then starts dendrite
 RUN echo '\
-#!/bin/bash -eu \n\
-pg_lsclusters \n\
-pg_ctlcluster 9.6 main start \n\
- \n\
-until pg_isready \n\
-do \n\
-  echo "Waiting for postgres"; \n\
-  sleep 1; \n\
-done \n\
-' > run_postgres.sh && chmod +x run_postgres.sh
+    #!/bin/bash -eu \n\
+    pg_lsclusters \n\
+    pg_ctlcluster 9.6 main start \n\
+    \n\
+    until pg_isready \n\
+    do \n\
+    echo "Waiting for postgres"; \n\
+    sleep 1; \n\
+    done \n\
+    ' > run_postgres.sh && chmod +x run_postgres.sh
 
 # we will dump the binaries and config file to this location to ensure any local untracked files
 # that come from the COPY . . file don't contaminate the build
@@ -46,9 +46,9 @@ EXPOSE 8008 8448
 # At runtime, generate TLS cert based on the CA now mounted at /ca
 # At runtime, replace the SERVER_NAME with what we are told
 CMD /build/run_postgres.sh && ./generate-keys --server $SERVER_NAME --tls-cert server.crt --tls-key server.key --tls-authority-cert /complement/ca/ca.crt --tls-authority-key /complement/ca/ca.key && \
- ./generate-config -server $SERVER_NAME --ci > dendrite.yaml && \
- # Replace the connection string with a single postgres DB, using user/db = 'postgres' and no password, bump max_conns
- sed -i "s%connection_string:.*$%connection_string: postgresql://postgres@localhost/postgres?sslmode=disable%g" dendrite.yaml && \
- sed -i 's/max_open_conns:.*$/max_open_conns: 100/g' dendrite.yaml && \
- cp /complement/ca/ca.crt /usr/local/share/ca-certificates/ && update-ca-certificates && \
- ./dendrite-monolith-server --really-enable-open-registration --tls-cert server.crt --tls-key server.key --config dendrite.yaml -api=${API:-0}
+    ./generate-config -server $SERVER_NAME --ci > dendrite.yaml && \
+    # Replace the connection string with a single postgres DB, using user/db = 'postgres' and no password, bump max_conns
+    sed -i "s%connection_string:.*$%connection_string: postgresql://postgres@localhost/postgres?sslmode=disable%g" dendrite.yaml && \
+    sed -i 's/max_open_conns:.*$/max_open_conns: 100/g' dendrite.yaml && \
+    cp /complement/ca/ca.crt /usr/local/share/ca-certificates/ && update-ca-certificates && \
+    ./dendrite-monolith-server --really-enable-open-registration --tls-cert server.crt --tls-key server.key --config dendrite.yaml -api=${API:-0}

clientapi/auth/login_publickey.go

@@ -30,10 +30,10 @@ import (
 
 type LoginPublicKeyHandler interface {
 	AccountExists(ctx context.Context) (string, *jsonerror.MatrixError)
-	IsValidUserIdForRegistration(userId string) bool
 	CreateLogin() *Login
 	GetSession() string
 	GetType() string
+	IsValidUserId(userId string) bool
 	ValidateLoginResponse() (bool, *jsonerror.MatrixError)
 }
 

clientapi/auth/login_publickey_ethereum.go

@@ -73,14 +73,18 @@ func (pk LoginPublicKeyEthereum) AccountExists(ctx context.Context) (string, *js
 		return "", jsonerror.Forbidden("the address is incorrect, or the account does not exist.")
 	}
 
+	if !pk.IsValidUserId(localPart) {
+		return "", jsonerror.InvalidUsername("the username is not valid.")
+	}
+
 	res := userapi.QueryAccountAvailabilityResponse{}
 	if err := pk.userAPI.QueryAccountAvailability(ctx, &userapi.QueryAccountAvailabilityRequest{
 		Localpart: localPart,
 	}, &res); err != nil {
 		return "", jsonerror.Unknown("failed to check availability: " + err.Error())
 	}
 
-	if res.Available {
+	if localPart == "" || res.Available {
 		return "", jsonerror.Forbidden("the address is incorrect, account does not exist")
 	}
 
@@ -89,7 +93,7 @@ func (pk LoginPublicKeyEthereum) AccountExists(ctx context.Context) (string, *js
 
 var validChainAgnosticIdRegex = regexp.MustCompile("^eip155=3a[0-9]+=3a0x[0-9a-fA-F]+$")
 
-func (pk LoginPublicKeyEthereum) IsValidUserIdForRegistration(userId string) bool {
+func (pk LoginPublicKeyEthereum) IsValidUserId(userId string) bool {
 	// Verify that the user ID is a valid one according to spec.
 	// https://github.com/ChainAgnostic/CAIPs/blob/master/CAIPs/caip-10.md
 
@@ -100,9 +104,9 @@ func (pk LoginPublicKeyEthereum) IsValidUserIdForRegistration(userId string) boo
 
 	isValid := validChainAgnosticIdRegex.MatchString(userId)
 
-	// In addition, double check that the user ID for registration
+	// In addition, double check that the user ID
 	// matches the authentication data in the request.
-	return isValid && userId == pk.UserId
+	return isValid && strings.ToLower(userId) == pk.UserId
 }
 
 func (pk LoginPublicKeyEthereum) ValidateLoginResponse() (bool, *jsonerror.MatrixError) {