Mautrix Twitter broken

[shalzz]

Describe the bug
Enabling mautrix twitter breaks the playbook

To Reproduce
Enable mautrix twitter

matrix_mautrix_twitter_enabled: true

I get the following error:

fatal: [matrix.redacted]: FAILED! =>
  msg: |-
    The task includes an option with an undefined variable. The error was: {{ matrix_mautrix_twitter_configuration_yaml|from_yaml|combine(matrix_mautrix_twitter_configuration_extension, recursive=True) }}: {{ lookup('template', 'templates/config.yaml.j2') }}: {{ { 'postgres': matrix_mautrix_twitter_database_connection_string, }[matrix_mautrix_twitter_database_engine] }}: 'dict object' has no attribute ''

    The error appears to be in '/home/shalzz/builds/network/matrix-docker-ansible-deploy/roles/matrix-bridge-mautrix-twitter/tasks/setup_install.yml': line 56, column 3, but may
    be elsewhere in the file depending on the exact syntax problem.

    The offending line appears to be:


    - name: Ensure mautrix-twitter config.yaml installed
      ^ here

Expected behavior
Mautrix twitter bridge should be configured and installed.

[shalzz]

Okay I see that we need to manually set the matrix_mautrix_twitter_database_* variables which is not described in the docs.
I would have thought the already specified database variables for synapse would be re-used here, but I guess it's not trivial to automatically create a new postgres database on an existing db server.

[shalzz]

Okay, even after setting the matrix_mautrix_twitter_database_* variables I'm still seeing the same error.

[spantaleev]

Perhaps you haven't defined matrix_mautrix_twitter_database_engine: postgres in your vars.yml file?

I don't think it's enough to have the following custom configuration to reproduce the bug:

matrix_mautrix_twitter_enabled: true

You're probably using an external Postgres server as well (matrix_postgres_enabled: false), which is causing all the trouble (manual work you need to do). Such a setup is not so well supported (something #1679 is aiming to address through documentation.. although.. things are not so simple).

---

The way group_vars/matrix_servers was configured, you'd end up with matrix_mautrix_twitter_database_engine: '' here:

matrix-docker-ansible-deploy/group_vars/matrix_servers

Line 563 in 1bea072

matrix_mautrix_twitter_database_engine: "{{ 'postgres' if matrix_postgres_enabled else '' }}"

.. which should have told you to define this empty matrix_mautrix_twitter_database_engine variable via this check:

matrix-docker-ansible-deploy/roles/matrix-bridge-mautrix-twitter/tasks/validate_config.yml

Lines 12 to 18 in 1bea072

	- name: Fail if database is not defined
	fail:
	msg: >-
	You need to define a need to set `matrix_mautrix_twitter_database_engine: postgres` and redefine the other `matrix_mautrix_twitter_database_*` variables
	when: "vars[item] == ''"
	with_items:
	- "matrix_mautrix_twitter_database_engine"

Perhaps this validation check did not work well and matrix_mautrix_twitter_database_engine still remains empty for you?

---

In 8c25ade, I've reworked how these variables work a bit. You should try and see if you have more success with that.

If you're using an external Postgres server, you'll need to define this additional vars.yml configuration:

matrix_mautrix_twitter_enabled: true

# Define your Postgres server credentials here (defaults are taken from roles/matrix-bridge-mautrix-twitter/defaults/main.yml)
matrix_mautrix_twitter_database_username: 'matrix_mautrix_twitter'
matrix_mautrix_twitter_database_password: ''
matrix_mautrix_twitter_database_hostname: ''
matrix_mautrix_twitter_database_port: 5432
matrix_mautrix_twitter_database_name: 'matrix_mautrix_twitter'

[shalzz]

Yes, indeed I'm using an external postgres database.

I didn't think to set matrix_mautrix_twitter_database_engineas it was already set to postgres in roles/matrix-bridge-mautrix-twitter/tasks/main.yml and the validation check certainly didn't warn me about it.
Manually setting it solves it.

It is certainly a bummer to realise now using an external postgres server made already configured bridges use an sqlite database. Especially since I didn't face any issues with using an external database as it's hostname is a domain name that is very well resolvable and accessible from within a docker container.

Anyways thanks for the quick reply, I guess I'll now start migrating my already setup bridges to use the postgres db from sqlite.

[spantaleev]

For external Postgres servers, we're defaulting to SQLite for bridges that support that.. Alternatively, we:

• either need to reach your Postgres server and create the appropriate databases on it for each additional service -- something that we do for matrix-postgres automatically
• or ask you to prepare your external Postgres server in a similar way (for each service) yourself and define the appropriate variables (a bunch of matrix_mautrix_twitter_database_* variables in this case)
• or just use SQLite (if the bridge supports it) and not bother you with doing manual Postgres work, unless you want to

Ideally, we'll be able to configure an external Postgres automatically just like we do it for matrix-postgres.

We create the additional databases here:

matrix-docker-ansible-deploy/roles/matrix-postgres/tasks/setup_postgres.yml

Lines 128 to 133 in 5a4dd2b

	- include_tasks:
	file: "{{ role_path }}/tasks/util/create_additional_databases.yml"
	apply:
	tags:
	- always
	when: "matrix_postgres_enabled|bool and matrix_postgres_additional_databases|length > 0"

This is a part of the matrix-postgres role and only runs when matrix_postgres_enabled: true, however.

If you're keen on making the matrix-postgres role perform initialization for an external Postgres server, we'll need to:

• adjust the Using an external PostgreSQL server (optional) docs, so that they don't just ask you to define Postgres variables for Synapse (matrix_synapse_database_*), but ask for a more privileged global Postgres user (matrix_postgres_connection_* variables)

• make changes to the matrix-postgres role, so that it performs database initialization for the configured Postgres server (define via matrix_postgres_connection_* variables) even if matrix_postgres_enabled: false. We'd better add another variable that controls whether this initialization is done or not, so that people can disable it if they so desire (e.g. matrix_postgres_additional_database_initialization_enabled: true)

• test that these changes work well -- requires an external Postgres server, time and interest in developing this feature -- something no one has had yet

If you'd like to work on the above, that'd be great though! It'd be very useful to have the playbook work with an external Postgres server seamlessly.

[shalzz]

Yes, having seamless support for external postgres server would be great, however just documenting that matrix_postgres_enabled: false has wildly different defaults than when it's enabled would be a good start.

[spantaleev]

I've adjusted the documentation. Thanks for reporting this!

[shalzz]

As for matrix_postgres_enabled: false is it possibly to not use postgres at all for synapse by not overriding matrix_synapse_database_* variables with external postgres settings?

If not then is there really any point of disabling postgres for other apps/services? Since the user is anyways required to have a working postgres db server running for at least one service?

[spantaleev]

I'm not sure I understand your question.

[shalzz]

Well, I'm just saying It seems setting matrix_postgres_enabled to false would disable postgres for everything, even synpase,
in a way that possible options would be:

• No Postgres
• Internal Postgres
• External Postgres

When its really just the last two options.

[spantaleev]

matrix_postgres_enabled controls whether the matrix-postgres role will install Postgres or not. It doesn't influence whether you can get a Postgress-less installation. Certain services require Postgres in any case.