Upgrading Dangerzone Leaves Dangling Files, Preventing it from Starting

[sudwhiwdh]

Please see this comment for a workaround. - @eloquence, 2023-09-07

[deeplow]

@sudwhiwdh what platform are you on? We currently support the following:

Ubuntu 23.04 (lunar)
Ubuntu 22.04 (jammy)
Ubuntu 20.04 (focal)
Debian 13 (trixie)
Debian 12 (bookworm)
Debian 11 (bullseye)
Fedora 38
Fedora 37
Fedora 36
Qubes OS (alpha support)

(taken from https://github.com/freedomofpress/dangerzone/blob/main/INSTALL.md)

[deeplow]

Regarding the availability in the package manager, earlier in the year we changed where our software is hosted and users need to install the repositories again (instructions for Debian / Ubuntu and Fedora). Is this something you've done recently? If not, that could be the issue.

We pulled the plug on the old software repositories recently and that could be behind the software not showing on the gnome software application.

[sudwhiwdh]

Hi,
I am using Fedora 38.
And I had already used the new repository to do a fresh install of dangerzone 0.4.1 on my system.
Can it have something to do with updating to 0.4.2? About a week and a half ago I was still using dangerzone and there was no problem.

[sudwhiwdh]

Which command do I use to launch dangerzone from the terminal?

[deeplow]

I am using Fedora 38.

OK. That's a bit more problematic, because that is supposed to work. Let's follow your suggestion and start it from the terminal to see if something pops up.

Which command do I use to launch dangerzone from the terminal?

dangerzone or dangerzone-cli.

If it says that it doesn't exist, can you try installing it via the terminal? (sudo dnf install dangerzone). It could be that gnome-software somehow doesn't see the newly added Dangerzone repository.

[sudwhiwdh]

[user@fedora ~]$ dangerzone-cli
Traceback (most recent call last):
File "/usr/bin/dangerzone-cli", line 33, in
sys.exit(load_entry_point('dangerzone==0.4.2', 'console_scripts', 'dangerzone-cli')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/bin/dangerzone-cli", line 25, in importlib_load_entry_point
return next(matches).load()
^^^^^^^^^^^^^
StopIteration

[user@fedora ~]$ dangerzone
Traceback (most recent call last):
File "/usr/bin/dangerzone", line 33, in
sys.exit(load_entry_point('dangerzone==0.4.2', 'console_scripts', 'dangerzone')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/bin/dangerzone", line 25, in importlib_load_entry_point
return next(matches).load()
^^^^^^^^^^^^^
StopIteration

[sudwhiwdh]

[user@fedora ~]$ sudo dnf install dangerzone
[sudo] password for user:
Last metadata expiration check: 1:21:55 ago on Mo 14 Aug 2023 14:09:26.
Package dangerzone-0.4.2-1.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!

[deeplow]

That is odd. Does it get resolved if you remove it and install it again?

sudo dnf remove dangerzone
sudo dnf install dangerzone
dangerzone-cli

[sudwhiwdh]

After removing and reinstalling, the same terminal output appears for dangerzone and dangerzone-cli.

[deeplow]

Damn, I have seen similar issues happen when an application is built for one python version but installed on a system with another. And thank you for your patience.

If it's not too burdensome, would be able to run the following commands? So we check if Dangerzone was installed properly?

python3 -c "import dangerzone"
which python
python3 --version
ls /usr/lib/python3.11/site-packages/dangerzone/

For context I just installed Dangerzone well in a new Fedora 38 virtual machine but failed to replicate this issue.

[sudwhiwdh]

[user@fedora ~]$ python3 -c "import dangerzone"
[user@fedora ~]$ which python
/usr/bin/python
[user@fedora ~]$ python3 --version
Python 3.11.4
[user@fedora ~]$ ls /usr/lib/python3.11/site-packages/dangerzone/
args.py conversion errors.py init.py logic.py settings.py
cli.py document.py gui isolation_provider pycache util.py

[sudwhiwdh]

What exactly does python3 -c "import dangerzone" do?

[deeplow]

Thank you.

What exactly does python3 -c "import dangerzone" do?

Because it ran without any issues, that means that Dangerzone can be successfully loaded by python (the programming language Dangerzone uses). That means that the issue is somehow in the file /usr/bin/dangerzone, since that is the one that can't really find dangerzone.

[sudwhiwdh]

That means that the issue is somehow in the file /usr/bin/dangerzone, since that is the one that can't really find dangerzone.

Doesn't the error message already tell in which line(s) the error is caused?

[deeplow]

[user@fedora ~]$ ls /usr/lib/python3.11/site-packages/dangerzone/
args.py conversion errors.py init.py logic.py settings.py
cli.py document.py gui isolation_provider pycache util.py

I see something unexpected there. Theoretically there should be a file called __init__.py there. Can you also run the following command and see if anything comes out?

cat /usr/lib/python3.11/site-packages/dangerzone/__init__.py 

[deeplow]

That means that the issue is somehow in the file /usr/bin/dangerzone, since that is the one that can't really find dangerzone.

Doesn't the error message already tell in which line(s) the error is caused?

Normally it does, but in this case, /usr/bin/dangerzone is an autogenerated file that looks for a dangerzone python module, but it's not finding a match. And that's what's bugging me. If dangerzone were not present it would show something like:

no module named Dangerzone

[user@dz-temp-test ~]$ dangerzone --version
Traceback (most recent call last):
  File "/usr/bin/dangerzone", line 33, in <module>
    sys.exit(load_entry_point('dangerzone==0.4.2', 'console_scripts', 'dangerzone')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/dangerzone", line 25, in importlib_load_entry_point
    return next(matches).load()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/importlib/metadata/__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'dangerzone'

[sudwhiwdh]

init.py is init.py. I guess the underscores got lost during copying.

[sudwhiwdh]

[user@fedora ~]$ cat /usr/lib/python3.11/site-packages/dangerzone/__init__.py 
import os
import sys

if "DANGERZONE_MODE" in os.environ:
    mode = os.environ["DANGERZONE_MODE"]
else:
    basename = os.path.basename(sys.argv[0])
    if basename == "dangerzone-cli" or basename == "dangerzone-cli.exe":
        mode = "cli"
    else:
        mode = "gui"

if mode == "cli":
    from .cli import cli_main as main
else:
    from .gui import gui_main as main

[deeplow]

Ah, then this is not the issue. For some reason this file didn't show in a previous command. Unfortunately I'm heading out for the day so we might need to cut it short today. I'll be back on Wednesday.

Thanks again for the help and sorry that there wasn't much progress.

[sudwhiwdh]

[user@fedora ~]$ ls /usr/lib/python3.11/site-packages/dangerzone/
args.py conversion errors.py init.py logic.py settings.py
cli.py document.py gui isolation_provider pycache util.py

Is listed as init.py. I did not know yet that I have to represent __init__.py as a code block in GitHub so that everything is displayed completely.

Thank you for researching together and have a great day!

[deeplow]

Is listed as init.py. I did not know yet that I have to represent init.py as a code block in GitHub so that everything is displayed completely.

Ah! You're right. That was it then.

Thank you for researching together and have a great day!

You too!

[pizzadude]

Hi, I had the same issue.
I fixed it by doing:

cd /usr/lib/python3.11/site-packages
sudo rm -rfv dangerzone-0.4.1-py3.11.egg-info

For me, this happens every dangerzone update. IMO the rpm packaging should be fixed so it automatically removes the old files on upgrade / reinstall.

[apyrgio]

Thank you both for flagging this issue. We were always installing the latest Dangerzone artifacts in a vacuum (which has its merits), so we never stumbled on this bad behavior.

It's good that we have a workaround for now, but let's have a clear way forward as well. Ideally, I'd want us to:

1. See if this issue affects Debian-based distros as well.
2. Check how RPM/DEB upgrades handle the artifacts of previous versions. Leaving stale artifacts behind is something that should never happen, and may be related with the way we produce our package.
3. Create new RPMs and bump their Fedora patch versions to -2. so existing Fedora users do not have to resort to this workaround.
4. Update our CI with a job that installs the latest upstream Dangerzone version and then upgrades it with a locally produced Dangerzone package, and see that it works.

We'll keep this issue updated once we have news on this front.

[pizzadude]

oh btw, dangerzone still has the issue where it requires me to temporarily disable SELinux on fedora 38 for the conversion process to work, but now the latest version has a new issue: it says i have run out of space halfway through conversion when i have plenty of space

[apyrgio]

Ok, we definitely need two separate GitHub issues for these two things, since they may take some digging. I was under the impression that (1) was fixed, since we now mount directories with :Z (see #335). As for the second issue, I believe it has to do with the size of the latest container image, but we'll see.

Can I ask you to open these two issues for your findings, and maybe share some error messages there? Else I can open them and ping you.

[pizzadude]

I opened two issues here:
#517
#518

[deeplow]

Fedora-38 (on Qubes OS), I installed on Fedora 38 Dangerzone 0.4.1 and then upgrading to 0.4.2. And I was unable to reproduce this issue, since dangerzone 0.4.2 started up just fine from the terminal. I do see, however, the stale directory:

/usr/lib/python3.11/site-packages/dangerzone-0.4.1-py3.11.egg-info
/usr/lib/python3.11/site-packages/dangerzone-0.4.2-py3.11.egg-info

But somehow it doesn't interfere with the loading. I will try now with a full-blown Fedora 38.

[deeplow]

I think I'm pretty close to the bottom of this. Turns out the issue is a bit wider in scope. In short, doing dnf remove dangerzone doesn't properly remove all the directories. For example, /usr/lib/python3.11/dangerzone remains with a skeleton of the structure:

/usr/lib/python3.11/site-packages/dangerzone/
dangerzone
 ├─ conversion
 │ └─  __pycache__
 ├─ gui
 │ └─  __pycache__
 └─ isolation_provider
       └─  __pycache__

My guess is that it's not removed because these __pycache__ files were generated only when dangerzone was executed and thus are considered as user-modifications. But I'm not so versed in RPM packaging, so I'll need to look into this. (Update according to this thread, that's the case)

Most critically it leaves the *.egg-info directory, which creates conflicts when running newer versions.

[apyrgio]

1. See if this issue affects Debian-based distros as well.

This is not the case, Debian packages remove the __pycache__ files.

2. Create new RPMs and bump their Fedora patch versions to -2. so existing Fedora users do not have to resort to this workaround.

This is not so straightforward. We'd need to have a post-install script that removes these empty __pycache__ dirs.

[sudwhiwdh]

@apyrgio @deeplow Can you imagine doing a small release that fixes this bug?
It will probably take a while until 0.5 is released and I would like to be able to start and use the app again in the meantime.

[deeplow]

This is something we're definitely considering. It will require doing our own RPM packging, but that's in good timing because we have other reasons to do so. We'll should have more news on this by the end of the week.

[deeplow]

It will probably take a while until 0.5 is released and I would like to be able to start and use the app again in the meantime.

In theory you already fix this by following this suggestion:

cd /usr/lib/python3.11/site-packages
sudo rm -rfv dangerzone-0.4.1-py3.11.egg-info

If that doesn't work, please let us know.

[sudwhiwdh]

Tried #514 (comment).

After that, when I start using dangerzone, the application window opens and it says: "Installing Dangerzone container image...".

When I start via dangerzone-cli appears:
[user@fedora ~]$ dangerzone-cli
Usage: dangerzone-cli [OPTIONS] FILENAMES...
Try 'dangerzone-cli --help' for help.

Error: Missing argument 'FILENAMES...'.

[apyrgio]

After that, when I start using dangerzone, the application window opens and it says: "Installing Dangerzone container image...".

Right, this operation may take a few minutes, since it loads a ~1GiB container image. Did you see an error while waiting? I bet that it will work without issues.

As for the dangerzone-cli, that's the expected help message, when you don't pass a file argument. If you pass a file to it, you'll see it would load the same container image to Podman.

[deeplow]

But probably now you can launch its graphical version either via the applications menu or by calling dangerzone in the terminal.

[sudwhiwdh]

Did you see an error while waiting? I bet that it will work without issues.

[user@fedora ~]$ dangerzone
QSocketNotifier: Can only be used with threads started with QThread
qt.qpa.qgnomeplatform.theme: The desktop style for QtQuick Controls 2 applications is not available on the system (qqc2-desktop-style). The application may look broken.
[DEBUG] Setting up Dangezone updater
[DEBUG] Consulting updater settings before checking for updates
[DEBUG] Checking platform type
[DEBUG] Running on Linux, disabling updates
[DEBUG] Will not check for updates, based on updater settings

[deeplow]

[user@fedora ~]$ dangerzone
QSocketNotifier: Can only be used with threads started with QThread
qt.qpa.qgnomeplatform.theme: The desktop style for QtQuick Controls 2 applications is not available on the system (qqc2-desktop-style). The application may look broken.

I think I've seen this before, but I don't think it interferes with Dangerzone. Note you're seeing extra (debug) information because you're starting the graphical version from the terminal. Normally (starting by opening the application) you wouldn't see this.

Can you see the dangerzone graphical application this way?

[sudwhiwdh]

As before, dangerzone is no longer found in GNOME software.

[deeplow]

As before, dangerzone is no longer found in GNOME software.

Thanks. However, I think this appears to be an unrelated issue. I have opened a new one for it here. So let's continue the conversation there, if you don't mind.

[sudwhiwdh]

So let's continue the conversation there, if you don't mind.

I not only don't mind, I'm also very grateful that you immediately created a new issue for it!

[apyrgio]

If you don't mind, I'd like to focus a bit on this:

After that, when I start using dangerzone, the application window opens and it says: "Installing Dangerzone container image...".

Just to make sure I get it straight:

1. You start Dangerzone from the terminal, by running dangerzone.
2. Some debug messages appear in the terminal, but we can ignore them, as they are most probably benign.
3. The Dangerzone GUI pops up, and you see a message there that says "Installing Dangerzone container image"

My question here is, what happens if you wait 5-10 minutes for the installation to finish?

[sudwhiwdh]

If you don't mind, I'd like to focus a bit on this:

This only works again after #514 (comment)
And yes, when the container is in place, I get a functional GUI.

[apyrgio]

Ok, that's a relief. Then we have the issue of dangling files that we need to tackle, as well as the fact that Dangerzone cannot be found in the Gnome software center. We are actively working on these, and we will give updates soon.