Problem with docker.asc in keyrings after last chore(deps): update ghcr.io/element-hq/synapse docker tag to v1.108.0

[UehgxEQbFiw4Tlw8]

Describe the bug
A clear and concise description of what the bug is.

After running the yesterday update about docker with just roles etc. I receive the message from Ansible:
TASK [galaxy/playbook_help : Update apt cache] *************************************************************************************************************************************************
fatal: [...]: FAILED! => changed=false
msg: 'E:Widersprüchliche Werte gesetzt für Option Signed-By betreffend die Quelle https://download.docker.com/linux/debian/ bullseye: /etc/apt/trusted.gpg.d/docker.asc != /etc/apt/keyrings/docker.asc, E: Die Liste der Quellen konnte nicht gelesen werden.'

I checekd on the server the two docker.asc files with a diff, they are equal, the first is older, the second was written yesterday in the keyring.
Checking also on the Debian download-website under gpg, it seems to me, also the file there has no deviations from the other twos.
Ergo: the ansible Warning seems to be wrong at first inspection.

To Reproduce
My vars.yml file looks like this:

Of no interest and it didn't change during the last few months.

Expected behavior
A clear and concise description of what you expected to happen.
Normal behavior by Ansible. The server successfully updated.

Matrix Server:

• OS: Raspian (Bullseye) with all patches / updates sofar installed.
• Architecture: uname -a: Linux <***> 6.1.21-v8+ fix typo #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux

Ansible:
On my Laptop with Ubuntu 22.04 LTS installed and up-to-date compatible with the packages on distribution
ansible [core 2.12.8]
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/myname/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3/dist-packages/ansible
ansible collection location = /home/myname/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/bin/ansible
python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
jinja version = 3.0.3
libyaml = True

Client:

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Possible workaround: to copy the older docker.asc

-rw-r--r-- 1 root root 3817 11. Dez 2022 docker.asc

into keyrings? Now still

-rw-r--r-- 1 root root 3817 28. Mai 07:24 docker.asc

Thanks for helping! Have a nice day
Best
M.

[spantaleev]

This should have not been an issue due to 3a4e58c, but.. Hmm..

[UehgxEQbFiw4Tlw8]

Sorry, I didn't offer a translation from German... here it is:

E:Contradictory values set for option Signed-By regarding the source https://download.docker.com/linux/debian/ bullseye: /etc/apt/trusted.gpg.d/docker.asc != /etc/apt/keyrings/docker.asc, E: The list of sources could not be read'.

[UehgxEQbFiw4Tlw8]

It seems I had two lines, both valid but conflicting with each other, in /etc/apt/source.list.d/docker.list.

Now with just:

deb [arch=arm64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian bullseye stable

(I've chosen the newest one, the other line as above but with /etc/apt/trusted.gpg.d/docker.asc) it seems to work till the end:

TASK [galaxy/playbook_help : Update apt cache] **************************************************************************************
changed: [mydomain.tdl]

A confirmation I guessed right would be helpful, because I'm not that keen when digging into /etc/apt/ ...
Thank you!

Best regards
M.

[spantaleev]

3a4e58c should have removed the whole /etc/apt/source.list.d/docker.list file because it contains the old line (having /etc/apt/trusted.gpg.d/docker.asc in it).

Removing the file will make the Docker role recreate it properly and there won't be a conflict.

The playbook is not automatically detecting this faulty line and is not removing the file for you, it seems. You are reporting being on the "upgrade Synapse to v1.108.0" commit, which is newer than 3a4e58c, so I would have expected it to work. Maybe there's something special in your setup (Raspberry Pi..) and it failed to do it correctly for you.

Anyway, I'm glad to hear that the problem has been fixed for you. If it happens again (to someone else), we may revisit this issue and figure out why 3a4e58c is not doing its job correctly in some instances.

[UehgxEQbFiw4Tlw8]

@spantaleev

As it seems, it is not a problem caused by RasperryPi with Raspian bullseye, but it happens also on a RaspberryPi with a fresh installed Ubuntu Server 24.04 LTS (via rpi-imager)

Ansible stops with message at:

TASK [galaxy/docker : Add Docker apt key.] *************************************************************************************************************************
fatal: [test_matrix.xxx.de]: FAILED! => changed=false
dest: /etc/apt/keyrings/docker.asc
elapsed: 0
msg: 'An unknown error occurred: ''CustomHTTPSConnection'' object has no attribute ''cert_file'''
url: https://download.docker.com/linux/ubuntu/gpg
...ignoring

TASK [galaxy/docker : Ensure curl is present (on older systems without SNI).] **************************************************************************************
ok: [test_matrix.xxx.de]

TASK [galaxy/docker : Add Docker apt key (alternative for older systems without SNI).] *****************************************************************************
changed: [test_matrix.xxx.de]

TASK [galaxy/docker : Add Docker repository.] **********************************************************************************************************************
fatal: [test_matrix.xxx.de]: FAILED! => changed=false
msg: 'Failed to update apt cache: W:Updating from such a repository can''t be done securely, and is therefore disabled by default., W:See apt-secure(8) manpage for repository creation and user configuration details., W:GPG error: https://download.docker.com/linux/ubuntu noble InRelease: The following signatures couldn''t be verified because the public key is not available: NO_PUBKEY 7EA0A9C3F273FCD8, E:The repository ''https://download.docker.com/linux/ubuntu noble InRelease'' is not signed.'

PLAY RECAP *********************************************************************************************************************************************************
test_matrix.xxx.de : ok=44 changed=2 unreachable=0 failed=1 skipped=30 rescued=0 ignored=1

Checking /etc/apt/keyrings/ one finds it empty.
Copying the content of the link above ( https://download.docker.com/linux/ubuntu/gpg ) in docker.asc and running again solves the problem (really?) or at least ansible has nothing to complain:

TASK [galaxy/docker : Add Docker apt key.] *************************************************************************************************************************
ok: [test_matrix.xxx.de]

TASK [galaxy/docker : Ensure curl is present (on older systems without SNI).] **************************************************************************************
skipping: [test_matrix.xxx.de]

TASK [galaxy/docker : Add Docker apt key (alternative for older systems without SNI).] *****************************************************************************
skipping: [test_matrix.xxx.de]

TASK [galaxy/docker : Add Docker repository.] **********************************************************************************************************************
changed: [test_matrix.xxx.de]

TASK [galaxy/docker : Install Docker packages.] ********************************************************************************************************************
skipping: [test_matrix.xxx.de]

TASK [galaxy/docker : Install Docker packages (with downgrade option).] ********************************************************************************************
changed: [test_matrix.xxx.de]

but than a new problem appears with the exim-relay pull (I think, the two problems are not correlated, but I may be mistaken):

TASK [galaxy/exim_relay : Ensure exim-relay image is pulled]
FAILED - RETRYING: [test_matrix.xxx.de]: Ensure exim-relay image is pulled (10 retries left).
FAILED - RETRYING: [test_matrix.xxx.de]: Ensure exim-relay image is pulled (9 retries left).
FAILED - RETRYING: [test_matrix.xxx.de]: Ensure exim-relay image is pulled (8 retries left).
FAILED - RETRYING: [test_matrix.xxx.de]: Ensure exim-relay image is pulled (7 retries left).
etc. etc.
fatal: [test_matrix.xxx.de]: FAILED! => changed=false
attempts: 10
msg: 'Error connecting: Error while fetching server API version: HTTPConnection.request() got an unexpected keyword argument ''chunked'''

A google search for the last msg above give some hints, but at this point I need definitely help...

E.g. at google-deepmind/alphafold#867 - still opened - it is suggested to use docker compose instead of docker-compose - possibly a solution also for this problem? But to be painfully honest, I don't really understand all what is said there...

Thank you for the kind help and suggestions! Highly appreciated.

Best regards
M.

[spantaleev]

An unknown error occurred: ''CustomHTTPSConnection'' object has no attribute ''cert_file''

.. sounds like too-old Ansible and/or other modules. See docs/ansible.md. I suggest trying to run Ansible from within a container, as described in that docs page.

[TwlyY29]

Hi,

I stumbled upon the same issue as @UehgxEQbFiw4Tlw8 above - the 'got an unexpected keyword argument "chunked"' error for a 24.04 LTS server.
After some digging, I learned that it is apparently an issue between python libraries requests>=2.29.0 and urllib3>=2.0. The issue should be resolved in the ansible docker repositories (like the geerlingguy/docker one) as the reply to this comment indicated: docker/docker-py#3113 (comment)
However, even after re-downloading all galaxy-roles, upgrading to the latest ansible, the issue still remained. So, for me, it wasn't a too-old ansible version, I think, @spantaleev ...

Luckily, I found a "solution", which is honestly rather a quick fix, that helped me: google-deepmind/alphafold#867 (comment)

Basically, the idea is to create a virtualenv on the matrix server

python3 -m venv ~/.venv
~/.venv/bin/python -m pip install 'requests<2.29.0' 'urllib3<2.0'

and in your inventory/hosts file specifying the venv-python-binary:

[group_name]
name ansible_host=matrix-server ansible_python_interpreter=/home/user/.venv/bin/python

Hope it helps...

[tobiasdiez]

Another solution might be to install the updated python-docker package:

sudo apt install python3-docker -t noble-updates