[WIP] Dendrite as alternative Matrix Server Implementation

[Dekker1]

This PR is a start towards providing Dendrite as an optional alternative Matrix server (as asked in #800). Dendrite is the second generation "sample" implementation of a Matrix server and offers to be more performant alternative to Synapse, allowing administrators to run Matrix on more minimal hardware. Currently Dendrite is in beta and does not contain all features that Synapse does.

This Dendrite role is directly based on the matrix-synapse role to hopefully be easy to maintain side-by-side.

Using Dendrite

To use dendrite you have to enable Dendrite, disable Synapse (to not cause any conflict), and Dendrite, like Synapse, will need a shared secret to create accounts:

matrix_synapse_enabled: false
matrix_dendrite_enabled: true
matrix_dendrite_macaroon_secret_key: "some_generated_secret"

What is currently working

• Dendrite gets fully installed using a generated configuration (see roles/matrix-dendrite/defaults/main.yml), a generated server signing key, and the dendrite-monolith Docker image.
• All required Dendrite databases gets created (by default dendrite uses ~10 databases)
• A matrix-dendrite service gets added to the system, which will automatically start the dendrite images with the system.
• User registration (using the create-account binary provided) should be working, both using Ansible and a matrix-dendrite-create-account script added to the system.

What is not working

I got stuck (and ran out of time) trying to configure the correct TLS / Nginx settings to correctly proxy Dendrite through Nginx and allow outside connections to Dendrite. Once this is fixed, the hope is that this would provide correct Dendrite functionality.

Even more work

The most important work that has to happen thereafter is that the functionality is correctly documented so other users will know how to activate the Dendrite server, how/if it connects with the other components (which I'm not sure about), and maybe some general information about Dendrite.

Call for Help

I believe this is a good start to providing Dendrite functionality in these Ansible scripts, but there is still quite a bit of work to be done. Although I had fun working on this for a few hours and would love to continue, I'm not sure how much time I will have in the next few weeks. So if you feel like you would like to have a go at fixing the last parts, then feel free to take over and amend my work. I'm happy to answer any questions and help with any superseding PRs.

[aaronraimist]

Thanks for working on this. I'm really exited to get Dendrite up and running. I haven't really dug in but I tried this out and it does appear to be working for me. I did have a few notes from what I can see initially. You can probably go ahead and delete the homeserver.yaml.j2 file and the few files related to goofys. I assume those have just been copied over from the Synapse role and aren't needed here. I did that in aaronraimist@5aecebe and everything still seems to work.

I also wanted to mention that since we are using Postgres, Dendrite does have the option of running everything from a single database. You are only required to use multiple databases if you are using SQLite. It probably does make sense to use multiple databases since we have the capability to do it and it could provide room for future expansion but just thought I would mention it.

I haven't actually set things up correctly yet but on https://github.com/aaronraimist/matrix-docker-ansible-deploy/commits/dendrite-aaron I just hard coded nginx to use dendrite instead of synapse and all of the basics are working. The APIs are available and I was able to register for an account. Exciting!

[spantaleev]

I guess going for a single Postgres database is possible, and I probably would have done that for simplicity.

.. but given that you've made it use multiple databases already, I guess it's better if we kept that. People can then scale out - moving some heavier/busier databases more easily outside of matrix-postgres, etc.

---

@aaronraimist, pointing nginx (well matrix.DOMAIN) to something other than Synapse will be much simpler very soon (once #456 lands). I've redone things there, so that there's one more level of indirection (matrix.DOMAIN -> matrix-synapse-proxy vhost -> matrix-synapse). Once #456 gets merged (soon!), we should be able to redo this Dendrite pull request, so that it changes this to (matrix.DOMAIN -> matrix-dendrite-proxy -> matrix-dendrite). Nice and simple!

---

In addition to the the above forwarding change, we should also start the process of dynamically tweaking other things as well, based on which homeserver implementation is enabled (various changes in group_vars/matrix_servers).

But first.. let's get #456 merged, as it will be necessary for repointing nginx to Dendrite, instead of Synapse.

---

This is indeed very exciting!

[Dekker1]

I've updated the build with @aaronraimist's changes to remove the duplicate goofys installation and updated the docker image to the newest version. I think this PR should then be mostly ready once we can connect NGINX correctly.

[tecosaur]

I'm very interested in this being merged, and it looks like it's rather close 😃. @Dekker1 how's the NGINX?

[Dekker1]

@tecosaur It seems that #456 (which should make it easier to point NGINX in the right direction, which seems to be the last missing technical step) has been merged. I haven't had a chance yet to have a look (and might not be able to in the next few weeks).

If you feel you might be able to have a look sooner, then please feel free to add to continue this branch. I'm happy to apply any contributions to the branch when ready.

[Dekker1]

I've rebased the feature/dendrite branch with master to remove any conflicts.

[tecosaur]

Just to confirm, is it only the nginx configuration that's missing?

[spantaleev]

There's probably some rewiring work (in group_vars/matrix_servers) that needs to happen as well.

[thomwiggers]

Would there also be a migration path from Synapse to Dendrite?

[aaronraimist]

@thomwiggers once the Dendrite developers invent one. That's likely at least 6+ months away and could be multiple years away.

[thomwiggers]

Not entirely unexpected but rough, it seems quite attractive considering even bridging a few small IRC rooms takes like 2GB of virtual memory with Synapse.

[spantaleev]

I've upgraded Dendrite from 0.3.11 to 0.5.0 and brought the configuration up to date.

This still needs more work to integrate properly with matrix-nginx-proxy, etc., but at least it's some progress on bringing it up to date.

[spantaleev]

I believe that Dendrite support is now ready, so I just merged this manually.

Thanks to everyone involved! 🙇

To learn more, see the CHANGELOG.

---

If you've been testing this PR and using configuration like this:

matrix_synapse_enabled: false

matrix_dendrite_enabled: true
matrix_dendrite_generic_secret_key: "SOME_SECRET_KEY_HERE"

you'd need to change to this:

matrix_homeserver_implementation: dendrite
matrix_homeserver_generic_secret_key: "SOME_SECRET_KEY_HERE"

---

While the Dendrite team focuses on making Dendrite more production-ready, we should probably focus on documenting Dendrite support better.

Some other potential Dendrite-related work that I've identified is:

• adding support for using a single Postgres database, as opposed to one for each service. Might make backups simpler, etc.
• Prometheus metrics support
• making --tags=update-user-password work for Dendrite
• docs/ page for Dendrite, as mentioned above. Right now the only documentation we have about Dendrite is that changelog entry.
• updating other docs/ pages to not be so Synapse-centric (may be done later, as Dendrite is still beta and we don't expect people to use it)

[tecosaur]

Fantastic! Thanks for all your work.