Does the Synapse Complement test runner for workers work?

[MadLittleMods]

Spawning from #11220 (comment)

---

It looks like the Synapse Complement test runner has option to have the WORKERS environment variable set and go into worker mode but I am unable to get it working,

$ WORKERS=1 COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestImportHistoricalMessages
... see terminal/console output below

Full terminal/console output

$ WORKERS=1 COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestImportHistoricalMessages
[+] Building 1.7s (18/18) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                   0.1s
 => => transferring dockerfile: 2.70kB                                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                      0.0s
 => => transferring context: 35B                                                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/python:3.8-slim                                                                                                                                                     1.0s
 => [internal] load build context                                                                                                                                                                                      0.4s
 => => transferring context: 6.81MB                                                                                                                                                                                    0.3s
 => [stage-1 1/5] FROM docker.io/library/python:3.8-slim@sha256:d31a1beb6ccddbf5b5c72904853f5c2c4d1f49bb8186b623db0b80f8c37b5899                                                                                       0.0s
 => => resolve docker.io/library/python:3.8-slim@sha256:d31a1beb6ccddbf5b5c72904853f5c2c4d1f49bb8186b623db0b80f8c37b5899                                                                                               0.0s
 => CACHED [stage-1 2/5] RUN apt-get update && apt-get install -y     curl     gosu     libjpeg62-turbo     libpq5     libwebp6     xmlsec1     libjemalloc2     libssl-dev     openssl     && rm -rf /var/lib/apt/li  0.0s
 => CACHED [builder 2/9] RUN apt-get update && apt-get install -y     build-essential     libffi-dev     libjpeg-dev     libpq-dev     libssl-dev     libwebp-dev     libxml++2.6-dev     libxslt1-dev     openssl     0.0s
 => CACHED [builder 3/9] COPY scripts /synapse/scripts/                                                                                                                                                                0.0s
 => CACHED [builder 4/9] COPY MANIFEST.in README.rst setup.py synctl /synapse/                                                                                                                                         0.0s
 => CACHED [builder 5/9] COPY synapse/__init__.py /synapse/synapse/__init__.py                                                                                                                                         0.0s
 => CACHED [builder 6/9] COPY synapse/python_dependencies.py /synapse/synapse/python_dependencies.py                                                                                                                   0.0s
 => CACHED [builder 7/9] RUN pip install --prefix="/install" --no-warn-script-location     /synapse[all]                                                                                                               0.0s
 => CACHED [builder 8/9] COPY synapse /synapse/synapse/                                                                                                                                                                0.0s
 => CACHED [builder 9/9] RUN pip install --prefix="/install" --no-deps --no-warn-script-location /synapse                                                                                                              0.0s
 => CACHED [stage-1 3/5] COPY --from=builder /install /usr/local                                                                                                                                                       0.0s
 => CACHED [stage-1 4/5] COPY ./docker/start.py /start.py                                                                                                                                                              0.0s
 => CACHED [stage-1 5/5] COPY ./docker/conf /conf                                                                                                                                                                      0.0s
 => exporting to image                                                                                                                                                                                                 0.1s
 => => exporting layers                                                                                                                                                                                                0.0s
 => => writing image sha256:330e2b3377700f5b734911015a6512810db35e02aacd2b213edca7cd6e9b32f0                                                                                                                           0.0s
 => => naming to docker.io/matrixdotorg/synapse                                                                                                                                                                        0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Building 0.3s (11/11) FINISHED
 => [internal] load build definition from Dockerfile-workers                                                                                                                                                           0.1s
 => => transferring dockerfile: 772B                                                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                                                      0.0s
 => => transferring context: 35B                                                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/matrixdotorg/synapse:latest                                                                                                                                                 0.0s
 => [1/6] FROM docker.io/matrixdotorg/synapse                                                                                                                                                                          0.0s
 => [internal] load build context                                                                                                                                                                                      0.1s
 => => transferring context: 24.25kB                                                                                                                                                                                   0.0s
 => CACHED [2/6] RUN apt-get update                                                                                                                                                                                    0.0s
 => CACHED [3/6] RUN apt-get install -y supervisor redis nginx                                                                                                                                                         0.0s
 => CACHED [4/6] RUN rm /etc/nginx/sites-enabled/default                                                                                                                                                               0.0s
 => CACHED [5/6] COPY ./docker/conf-workers/* /conf/                                                                                                                                                                   0.0s
 => CACHED [6/6] COPY ./docker/configure_workers_and_start.py /configure_workers_and_start.py                                                                                                                          0.0s
 => exporting to image                                                                                                                                                                                                 0.1s
 => => exporting layers                                                                                                                                                                                                0.0s
 => => writing image sha256:7fc9754111e6270bc960d4c9325b73f4f1d4c9e534005b577eb809eeb991437f                                                                                                                           0.0s
 => => naming to docker.io/matrixdotorg/synapse-workers                                                                                                                                                                0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Building 0.2s (13/13) FINISHED
 => [internal] load build definition from SynapseWorkers.Dockerfile                                                                                                                                                    0.0s
 => => transferring dockerfile: 52B                                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/matrixdotorg/synapse-workers:latest                                                                                                                                         0.0s
 => [1/8] FROM docker.io/matrixdotorg/synapse-workers                                                                                                                                                                  0.0s
 => [internal] load build context                                                                                                                                                                                      0.0s
 => => transferring context: 125B                                                                                                                                                                                      0.0s
 => CACHED [2/8] RUN curl -OL "https://github.com/caddyserver/caddy/releases/download/v2.3.0/caddy_2.3.0_linux_amd64.tar.gz" &&   tar xzf caddy_2.3.0_linux_amd64.tar.gz && rm caddy_2.3.0_linux_amd64.tar.gz && mv c  0.0s
 => CACHED [3/8] RUN apt-get update                                                                                                                                                                                    0.0s
 => CACHED [4/8] RUN apt-get install -y postgresql                                                                                                                                                                     0.0s
 => CACHED [5/8] RUN pg_ctlcluster 13 main start &&  su postgres -c "echo  "ALTER USER postgres PASSWORD 'somesecret';  CREATE DATABASE synapse   ENCODING 'UTF8'   LC_COLLATE='C'   LC_CTYPE='C'   template=template  0.0s
 => CACHED [6/8] COPY synapse/workers-shared.yaml /conf/workers/shared.yaml                                                                                                                                            0.0s
 => CACHED [7/8] WORKDIR /data                                                                                                                                                                                         0.0s
 => CACHED [8/8] COPY synapse/caddy.complement.json /root/caddy.json                                                                                                                                                   0.0s
 => exporting to image                                                                                                                                                                                                 0.1s
 => => exporting layers                                                                                                                                                                                                0.0s
 => => writing image sha256:736926dfc8d82f9efbb44978e8087fa086f29070fecfeaa09f3b4d03813bc6c2                                                                                                                           0.0s
 => => naming to docker.io/library/complement-synapse-workers                                                                                                                                                          0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
2021/11/10 15:33:21 config: &{BaseImageURI:complement-synapse-workers BaseImageArgs:[] DebugLoggingEnabled:false AlwaysPrintServerLogs:true BestEffort:false VersionCheckIterations:500 KeepBlueprints:[] PackageNamespace:pkg}
2021/11/10 15:33:21 Running with Complement CA
=== RUN   TestImportHistoricalMessages
2021/11/10 15:34:19 pkg.hs_with_application_service.hs2 : failed to deployBaseImage: pkg.hs_with_application_service.hs2: failed to check server is up. GET http://localhost:55007/_matrix/client/versions => error: Get "http://localhost:55007/_matrix/client/versions": dial tcp: lookup localhost: no such host
2021/11/10 15:35:16 pkg.hs_with_application_service.hs2 : Failed to extract container logs: Error response from daemon: Bad response from Docker engine
    msc2716_test.go:62: Deploy: Failed to construct blueprint: pkg.hs_with_application_service.hs2: failed to check server is up. GET http://localhost:55007/_matrix/client/versions => error: Get "http://localhost:55007/_matrix/client/versions": dial tcp: lookup localhost: no such host
--- FAIL: TestImportHistoricalMessages (115.44s)
FAIL
FAIL	github.com/matrix-org/complement/tests	115.938s
2021/11/10 15:33:21 config: &{BaseImageURI:complement-synapse-workers BaseImageArgs:[] DebugLoggingEnabled:false AlwaysPrintServerLogs:true BestEffort:false VersionCheckIterations:500 KeepBlueprints:[] PackageNamespace:csapi}
2021/11/10 15:33:21 Running with Complement CA
testing: warning: no tests to run
PASS
ok  	github.com/matrix-org/complement/tests/csapi	0.250s [no tests to run]
FAIL

And results in my Docker crashing in different ways:

Fatal Error

Docker daemon failed to start

2021-11-10T21:35:10Z dockerd time="2021-11-10T21:35:10.691096500Z" level=info msg="ignoring event" container=364f726c48860bb2acd17e2e2cce226ca1458678b671362af70a78f34e88c8d6 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"

And other different errors:

2021-11-10T03:24:22Z dockerd time="2021-11-10T03:24:22.696959125Z" level=info msg="Container 2faad2c3366d177fba0d1df67959518ac66114e72ddca539434f6c53cdadc084 failed to exit within 10 seconds of signal 15 - using the force

2021-11-10T00:39:12Z dockerd time="2021-11-10T00:39:11.027705833Z" level=info msg="[graphdriver] using prior storage driver: overlay2"

I'm curious if anyone else is able to run the Complement tests against Synapse in worker mode?

---

NB: For reference, Complement runs just fine without the WORKERS=1 environment variable set.

Environment

• Running the develop branch of Synapse
• macOS 11.1
• Docker desktop 4.1.1
  • Engine 20.10.8
  • Compose 1.29.2

$ docker version
Client:
 Cloud integration: 1.0.17
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:55:20 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:31 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Related issues

• Synapse workers Complement image results in flaky tests due to inconsistent worker process init #10065
• Synapse's test suite doesn't cover workers #11111

[clokep]

Yes, it doesn't currently work. This is a duplicate of #10065.

[MadLittleMods]

Thanks for the confirmation @clokep 😀

It isn't obvious that it's a duplicate. #10065 seems to indicate that it works sometimes (flaky) and I don't see any mention of Docker crashes.

[clokep]

Fair, I think the gist is that it is unsupported / doesn't work right now and needs to be rethought. :(