If a service is enabled, a database for it is created in postgres with a uniqque password. The service can then use this database for data storage instead of relying on sqlite.

Moving it above the "uninstalling" set of tasks is better.
Extracting it out to another file at the same time, for readability,
especially given that it will probably have to become more complex in
the future (potentially installing `jq`, etc.)

We'll most likely use one that matches the database name, but
it's better to have it configurable.

Quotes are necessary around dictionary field names.

There was a missing `}` as well.

To avoid needing to have `jq` installed on the machine, we could:
- try to run jq in a Docker container using some small image providing
that
- better yet, avoid `jq` altogether

> Invalid data passed to 'loop', it requires a list, got this instead: matrix_postgres_additional_databases. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup.

Well, or working around it, as I've done in this commit (which seems
more sane than `wantlist=True` stuff).

The tasks in `create_additional_databases.yml` will likely
ensure `matrix-postgres.service` is started, etc.

If no additional databases are defined, we'd rather not execute that
file and all these tasks that it may do in the future.

While these modules are really nice and helpful, we can't use them
for at least 2 reasons:

- for us, Postgres runs in a container on a private Docker network
(`--network=matrix`) without usually being exposed to the host.
These modules execute on the host so they won't be able to reach it.

- these modules require `psycopg2`, so we need to install it before
using it. This might or might not be its own can of worms.

Being more explicit sounds better.

People can toggle between them now. The playbook also defaults
to using SQLite if an external Postgres server is used.

Ideally, we'd be able to create databases/users in external Postgres
servers as well, but our initialization logic (and `docker run` command,
etc.) hardcode too many things right now.

…users/databases

Using the result of `password_hash` works for creating them,
but authentication seems to be failing with some tools like pgloader.

It's possible that we're not escaping things properly somewhere.
Ideally, it'd be nice to solve that. But the easier (and still
relatively safe/good) solution is to just turn that password hash
into a UUID that's safe for passing around without worrying about
escaping.

This can be used by various bridges, etc., to import an SQLite
(or some other supported) database into Postgres.

…tgres_*`)

`matrix_SERVICE_database_engine` remains as it is - the only one with a
`_database_` infix. Postgres-related ones move to `_postgres_`.

…nnection string around

Instead of passing the connection string, we can now pass a name of a
variable, which contains a connection string.

Both are supported for having extra flexibility.

I don't use this bridge, so this is completely untested.

Should have been done during 2a99e84, but I've missed it.

Because of this, 9bf8ce8 was also incomplete and would have caused
an error.

…x_telegram_postgres_hostname)

Related to 9bf8ce8.

…es_pasword`

The only one that remains is `matrix_synapse_database_password`, but
that's something old and should be dealt with separately in the future
(unless it remains as it is).

I don't use this bridge, so this is completely untested.

Regression since 2d99ade and 9bf8ce8, respectively.

When SQLite is to be used, these bridges expect an `sqlite://`
connection string, and not a plain file name (path), like Appservice
Discord and mautrix-whatsapp do.

I don't use this bridge, so this is completely untested.

I was thinking that it makes sense to be more specific,
and using `_postgres_` also separated these variables
from the `_database_` variables that ended up in bridge configuration.

However, @jdreichmann makes a good point
(spantaleev#740 (comment))
that we don't need to be so specific and can allow for other engines (like MySQL) to use these variables.

…and migration notice

…tabase_*`

Looks like there are some that I missed in 087dbe4

This has been tested and appears to work.

…in role and migration notice

…cebook

…ngouts

…legram

…atsapp

…-discord

Auto-migration and everything seems to work. It's just that
matrix-registration cannot load the Python modules required
for talking to a Postgres database.

Tracked here: zeratax/matrix-registration#44

Until this gets fixed, we'll continue default to 'sqlite'.

We were referencing them from `group_vars/matrix_servers` since
recently, but there were no such variables and they weren't being put to
use.

…ension

We were running into conflicts, because having initialized
the roles (users) and databases, trying to import leads to
errors (role XXX already exists, etc.).

We were previously ignoring the Synapse database (`homeserver`)
when upgrading/importing, because that one gets created by default
whenever the container starts.

For our additional databases, it's a similar situation now.
It's not created by default as soon as Postgres starts with an empty
database, but rather we create it as part of running the playbook.

So we either need to skip those role/database creation statements
while upgrading/importing, or to avoid creating the additional database
and rely on the import for that. I've gone for the former, because
it's already similar to what we were doing and it's simpler
(it lets `setup_postgres.yml` be the same in all scenarios).

In cases where pgloader is not enough and we need to do some additional
migration work after it, we can now use
`additional_psql_statements_list` and
`additional_psql_statements_db_name`.

This is to be used when migrating `matrix-registration`'s data at the
very least.

Now that 0.7.2 is out, the Docker image supports Postgres
and we can do the (SQLite -> Postgres) migration.

I've also found out that we needed to fix up the `tokens.ex_date` column
data type a bit to prevent matrix-registration from raising exceptions
when comparing `datetime.now()` with `ex_date` coming from the database.

Example:

> File "/usr/local/lib/python3.8/site-packages/matrix_registration/tokens.py", line 58, in valid
> expired = self.ex_date < datetime.now()
> TypeError: can't compare offset-naive and offset-aware datetimes

…-discord

Should have been part of 149872e

…-instagram

…gration

Our old (base-path -> data-path) SQLite migration can't work otherwise.

It's probably not necessary to keep it anymore, but since we still do,
at least we should take care to ensure it works.

…-skype

…-slack

…-steam

…-twitter

…ce-irc

…ervice-slack