Add a Deb packaging backend

[freakboy3742]

Adds a .deb packaging backend. Fixes #1062.

This is a work in progress. It doesn't currently have any tests, and there's a lot of refactoring that can be done to share pieces between the AppImage and Deb backends (as they're both docker builds; an RPM backend will likely also share this infrastructure). However, it's at the point where community feedback would be useful.

To test a Toga app named togatest, check out this PR branch and install it (pip install -e path/to/checkout), run briefcase new to generate a new project with an app name of togatest, then add the following to your project's pyproject.toml:

[tool.briefcase.app.togatest.linux.deb]

system_requires = [
    'libcairo2-dev',
    'libgirepository1.0-dev',
]

system_runtime_requires = [
    "libgtk-3-0",
    "libwebkit2gtk-4.0-37",
]

system_requires are the packages needed to build your app; system_runtime_requires are the packages that need to exist at runtime. If you're building a non-Toga app, or you have dependencies that aren't provided as manylinux wheels, you'll likely need to update both these settings. Manylinux wheels should work without any need to update either setting.

There is a deb branch on briefcase-template, plus a new briefcase-linux-deb-template. The deb template should be picked up without any additional configuration; you'll need to clone the briefcase-template locally if you want to test that one.

If you're on a Debian-derived operating system, you can build and run the app natively:

briefcase run linux deb --no-docker

If you're on macOS, or any Linux, and you have Docker:

briefcase package linux deb --target ubuntu:jammy

will produce a .deb file in the linux folder. ubuntu:jammy is the vendor and codename of any Debian-derived distribution that has a Docker image (so - debian:bullseye and ubuntu:focal will also work). If you're on the target platform, you can then run sudo apt install <name of deb pkg> to install the app. On installation, the app should appear in your desktop menus.

Some questions for testers:

• Does this work for you? If it doesn't, please describe exactly what you did, and provide the full Briefcase log file.
• Does it match your expectations on how a Debian package should "taste"? i.e., Does it put files in the right places, name them appropriately, etc?
• Does the developer experience make sense?
• If you're packaging an PySide or PPB app, what did you need to put in system_requires and system_runtime_requires?

PR Checklist:

• All new features have been tested
• All new features have been documented
• I have read the CONTRIBUTING.md file
• I will abide by the code of conduct

[flowerncsu]

• couldn't do editable pip install but dropping -e worked fine.

ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /home/charlotte/projects/briefcase
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)

• There's an extraneous [ at the beginning of the blob provided for the pyproject.toml, but removing that cleared the error

• Ran into a hiccup installing pycairo, but this command from the pycairo docs cleared that issue: sudo apt install libcairo2-dev pkg-config python3-dev

Now I'm getting what might be some python version confusion. When I run briefcase run linux deb --no-docker with my virtual environment activated, it tells me No module named 'toga' (I verified toga is installed in the virtual environment). Based on its PYTHONPATH output, it doesn't look like it's referencing my virtual environment, so I tried installing toga on my system python just as a test, but that didn't make a difference. Attaching most recent log file.
briefcase.2023_02_06-10_21_53.run.log

[freakboy3742]

• couldn't do editable pip install but dropping -e worked fine.

ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /home/charlotte/projects/briefcase
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)

Sounds like you might have an old version of pip? PEP517 editable installs need a fairly recent version (22.something, I think?)

• There's an extraneous [ at the beginning of the blob provided for the pyproject.toml, but removing that cleared the error

Oops - thanks; I've corrected the example.

• Ran into a hiccup installing pycairo, but this command from the pycairo docs cleared that issue: sudo apt install libcairo2-dev pkg-config python3-dev

Ah - ok; that's a good clarification. If you build in Docker, all the system requirements (system_requires) will be installed for you; if you build out of docker, you need to install those yourself. I'll update the docs to reflect this.

Now I'm getting what might be some python version confusion. When I run briefcase run linux deb --no-docker with my virtual environment activated, it tells me No module named 'toga' (I verified toga is installed in the virtual environment). Based on its PYTHONPATH output, it doesn't look like it's referencing my virtual environment, so I tried installing toga on my system python just as a test, but that didn't make a difference. Attaching most recent log file. briefcase.2023_02_06-10_21_53.run.log

Hrm - I can't see an obvious cause for this in the logs; my best guess is that something about the error you got with pycairo got the app into a weird state.

1. Does the linux/ubuntu/focal/togatest/togatest_0.0.1-1_amd64/usr/local/lib/app_packages folder contain toga, toga_gtk etc?
2. If not: does running briefcase run linux deb -u -r run pip and add those files? (This does a run, but explicitly updates the code and requirements, forcing their re-installation)
3. If not: does deleting the linux folder and re-running briefcase run linux deb work?

If toga is in the app_packages folder... then we've got a much bigger problem :-)

[flowerncsu]

Success!

• I checked my pip version and it's actually version 23.0, so not sure what's up there.
• I think you dropped a togatest out of the path, but assuming you were referring to linux/ubuntu/focal/togatest/togatest_0.0.1-1_amd64/usr/local/lib/togatest/app_packages/, it was empty.
• Using the -u -r (also with --no-docker) did work! And the resulting app did behave exactly like I would expect an app to behave on this system; I clicked around it to check behaviors and appearances and everything seemed functional and coherent!

[freakboy3742]

@flowerncsu Awesome - thanks for that testing!

[rmartin16]

My high-level thoughts so far:

Install Location: /usr/local

The internet contains a lot of thoughts about the /usr/local directory and exactly what its purpose is...but Debian seems to have some unequivocal words on the matter: "As mandated by the FHS, packages must not place any files in /usr/local".

Looking at /usr/local on my daily driver machine with loads of packages installed, indeed /usr/local only contains applications I have built myself.

Should we be embedding a Python with the app?

These debs noticeably do not contains a support package....although, it's clear the intention is to defer to the host Debian system to provide Python (along with any other assortment of dependencies...other than Python dependencies since we will be bringing those along).

Is this the intended final state for debs? Packaging an app for an arbitrary version of Python seems much more desirable from an end user perspective. Especially since we're not expecting this deb package to provide meaningful functionality for the system python like users would get from the plethora of python-* packages in the Debian repositories; this package just needs a (compatible) Python for it to run.

Building the deb Package

I'm not entirely convinced that dpkg-deb is the right/best tool to build a deb...but I'm not also convinced there's anything necessarily wrong with it. A lot of deb packaging seems to center around orchestrating the build of the application such that it can all be automated with Debian tools; actually creating the deb itself seems relatively straightforward.

Nonetheless, was there any particular reason you landed on dpkg-deb?

Debian Policy

There's parts of the Debian Policy this isn't currently in compliance with.

Some examples:

❯ lintian linux/helloworld_0.0.1-1~ubuntu-jammy_amd64.deb 
E: helloworld: dir-in-usr-local usr/local/bin/
E: helloworld: dir-in-usr-local usr/local/lib/
E: helloworld: dir-in-usr-local usr/local/lib/helloworld/
E: helloworld: dir-in-usr-local ... use --no-tag-display-limit to see all (or pipe to a file/program)
E: helloworld: extended-description-is-empty
E: helloworld: file-in-usr-local usr/local/bin/helloworld
E: helloworld: file-in-usr-local usr/local/lib/helloworld/app/helloworld-0.0.1.dist-info/INSTALLER
E: helloworld: file-in-usr-local usr/local/lib/helloworld/app/helloworld-0.0.1.dist-info/METADATA
E: helloworld: file-in-usr-local ... use --no-tag-display-limit to see all (or pipe to a file/program)
E: helloworld: missing-dependency-on-libc needed by usr/local/bin/helloworld and 3 others
E: helloworld: no-changelog usr/share/doc/helloworld/changelog.gz (native package)
E: helloworld: no-copyright-file
E: helloworld: package-installs-python-pycache-dir usr/local/lib/helloworld/app_packages/cairo/__pycache__/
E: helloworld: package-installs-python-pycache-dir usr/local/lib/helloworld/app_packages/gbulb/__pycache__/
E: helloworld: package-installs-python-pycache-dir usr/local/lib/helloworld/app_packages/gi/__pycache__/
E: helloworld: package-installs-python-pycache-dir ... use --no-tag-display-limit to see all (or pipe to a file/program)
E: helloworld: unstripped-binary-or-object usr/local/bin/helloworld
E: helloworld: unstripped-binary-or-object usr/local/lib/helloworld/app_packages/cairo/_cairo.cpython-310-x86_64-linux-gnu.so
E: helloworld: unstripped-binary-or-object usr/local/lib/helloworld/app_packages/gi/_gi.cpython-310-x86_64-linux-gnu.so
E: helloworld: unstripped-binary-or-object ... use --no-tag-display-limit to see all (or pipe to a file/program)
W: helloworld: file-in-unusual-dir usr/local/bin/helloworld
W: helloworld: file-in-unusual-dir usr/local/lib/helloworld/app/helloworld-0.0.1.dist-info/INSTALLER
W: helloworld: file-in-unusual-dir usr/local/lib/helloworld/app/helloworld-0.0.1.dist-info/METADATA
W: helloworld: file-in-unusual-dir ... use --no-tag-display-limit to see all (or pipe to a file/program)
W: helloworld: non-standard-dir-perm usr/ 0775 != 0755
W: helloworld: non-standard-dir-perm usr/local/ 0775 != 0755
W: helloworld: non-standard-dir-perm usr/local/lib/ 0775 != 0755
W: helloworld: non-standard-dir-perm ... use --no-tag-display-limit to see all (or pipe to a file/program)
W: helloworld: non-standard-file-perm usr/local/lib/helloworld/app/helloworld-0.0.1.dist-info/INSTALLER 0664 != 0644
W: helloworld: non-standard-file-perm usr/local/lib/helloworld/app/helloworld-0.0.1.dist-info/METADATA 0664 != 0644
W: helloworld: non-standard-file-perm usr/local/lib/helloworld/app/helloworld-0.0.1.dist-info/WHEEL 0664 != 0644
W: helloworld: non-standard-file-perm ... use --no-tag-display-limit to see all (or pipe to a file/program)
W: helloworld: recommended-field helloworld_0.0.1-1~ubuntu-jammy_amd64.deb Priority
W: helloworld: recommended-field helloworld_0.0.1-1~ubuntu-jammy_amd64.deb Section

debian/control....or is it DEBIAN/control

Currently hitting this error in Docker, pop os 22.04, and Debian 11.

[helloworld] Building .deb package...
dpkg-deb: error: failed to open package info file 'helloworld_0.0.1-1_amd64/DEBIAN/control' for reading: No such file or directory
Building...

briefcase.2023_02_07-17_31_58.package.log

I haven't gained a full understanding yet but Debian seems to be differentiating "source packages" and "binary packages" where one uses a capitalized directory and the other is not...

[freakboy3742]

Thanks for this feedback - it's really helpful.

Looking at /usr/local on my daily driver machine with loads of packages installed, indeed /usr/local only contains applications I have built myself.

Yeah - and that was what I was following by picking /usr/local as a packaged location. However, it appears that this is in violation on the dpkg guidelines, so I guess that means the app needs to be moved to /usr. That's easy enough to do - there's nothing in the app that is bound to the /local path that can't be easily modified.

These debs noticeably do not contains a support package.... Is this the intended final state for debs?

Yes, but I'm definitely open for debate on the topic.

Here's my reasoning:

• The Debian world has a different relationship with Python to everyone else. Completely aside from the way they... "fix"... Python from the official sources - there is no official "get Python 3.X" solution. The end-user expectation is that the only official Python is the one the system provides.
• Authors are essentially going to need to publish a deb for every OS they want to support. My original thought was that we might be able to build a single package referencing a nebulous "Python3" requirement; but it turns out other binary dependencies imposed by PyCairo and PyGobject become a problem. So - we're going to have Debian 10, Debian 11, Ubuntu 20.04, Ubuntu 22.04 (and more), ... packages anyway.
• Although I can't point at a specific policy statement, the Debian (and, to a lesser extent, Linux generally) worldview seems philosophically opposed to the idea of an app shipping its own version of Python. My read is that distributions are aiming for (and users expect) a "single integrated system". As an example - I recently got chewed out (on a different forum) by a Linux user for suggesting that macOS shipping fat binaries was a good user experience compromise; they were of the opinion that it wasted disk space. This is the same sort of worldview that leads to people thinking "building your entire system from source" is a productive use of time (Gentoo, Arch, etc). Working "with the system" seems to be a lot closer aligned with platform expectations than an app developers desire to use a specific Python version.
• Python is a lot more compatible across versions than Debian is :-) With a Python3.8 floor, there's a limited number of syntactic changes, and almost all the stdlib deprecations can be easily accommodated on a 3.8-3.11 range.
• It makes the build process a lot faster, and the downloaded package a lot smaller :-)

The alternatives I can see would be:

• Bundling a Python distro with the app. This is what Flatpak does - download and compile Python from source as part of the build process, then install into a specific location. This results in less of an "integrated system", and bigger packages; but exact Python versions. However, this would cause problems for the build process; we wouldn't be able to build into the Briefcase folder and run from that location, because Linux binaries aren't inherently relocatable. briefcase build --no-docker would need to run as root so it can install the python libraries into /usr (or we'd need to do some sort of chroot jail...) - and that means the development process taints the /usr folder in non-auditable ways.
• Allow for the use of deadsnakes. Add deadsnakes to the Docker image and build for a specific Python, and add that specific python version as a dependency; it then becomes an end-user issue telling users that they need to add the deadsnakes PPA. This isn't a great end-user experience (it's no longer just "install the deb", its "run these three commands then install the deb), but at some point, we have to cook with the ingredients we've been given.

To my mind, "use the system Python" is the best of a bunch of bad options. Deadsnakes could even be added as an option on top of the current implementation; By default, package for the system python; but with an opt-in option on DEB configs (use_system_python=False), that enables the use of arbitrary Python versions.

Looking to #1063 and #1064 for related concerns: On Redhat (and variants) there are more options (RHEL8 provides Python3.6, 3.8 and 3.9 for example), but there's no analog of deadsnakes (that I'm aware of); Arch et al provide source packages for everything, so I'm fairly certain you can reference a specific version in package dependencies.

Nonetheless, was there any particular reason you landed on dpkg-deb?

Ummm.... because all the online tutorials I found used dkpg-deb? What's the alternative?

There's parts of the Debian Policy this isn't currently in compliance with.

Yeah - I wasn't aware of Lintian until after I asked publicly for feedback (the same tutorials that told me about dpkg-deb neglected to mention it 😄)

Most of these should be fairly straightforward to accomodate; moving to /usr and adding some extra metadata should silence 90% of them.

debian/control....or is it DEBIAN/control

Yeah - this is definitely something that needs to be corrected... as soon as I can work out what "correct" is 😄

[rmartin16]

Nonetheless, was there any particular reason you landed on dpkg-deb?

Ummm.... because all the online tutorials I found used dkpg-deb? What's the alternative?

So, from everything I've read, dpkg-deb feels like a particularly low-level tool; one that will basically do what it's told without much validation or feedback. Alternatively, if we consider something like debuild from devscripts, it's a wrapper on dpkg-buildpackage. And dpkg-buildpackage seems to be the more holistic tool for building Debian packages. Although, it seems highly likely that dpkg-buildpackage eventually calls dpkg-deb.

Honestly, though, this feels a bit like when I started using Python and first attempted to create a package for PyPI. You find all these tools for managing the whole life cycle of your project and it isn't really clear what's the best option.

[rmartin16]

These debs noticeably do not contains a support package.... Is this the intended final state for debs?

Yes, but I'm definitely open for debate on the topic.

I don't have a particularly strong conviction we shouldn't use the System Python. I sympathize with the "integrated system" approach and am aligned on the principle. However, IMO, when the rubber hits the road for distro-shipped Python...it introduces a layer of limitations for the developer that are really quite artificial. (e.g. Ubuntu 20.04 LTS doesn't run out until 2030 but its Python is done in 20 months :/).

On the other hand, I think any future state should allow users to disable including a whole Python in their app when they know the System Python addresses their needs; so, if this potentially ends up just being the first version of deb support, I'm on board. On the third hand, this may be a good use case for Greg's standalone Python...

p.s. i did find these folks advertising support for bundling python in a deb....just as doc of prior art.

[freakboy3742]

I don't have a particularly strong conviction we shouldn't use the System Python. I sympathize with the "integrated system" approach and am aligned on the principle. However, IMO, when the rubber hits the road for distro-shipped Python...it introduces a layer of limitations for the developer that are really quite artificial. (e.g. Ubuntu 20.04 LTS doesn't run out until 2030 but its Python is done in 20 months :/).

Absolutely. From my non-linux-user perspective, I don't understand how Debian/Ubuntu considers this situation tenable. I hope they enjoy taking responsibility for backporting all the security patches (they are backporting security patches, right...?). But - not my circus, not my monkeys... :-)

On the other hand, I think any future state should allow users to disable including a whole Python in their app when they know the System Python addresses their needs; so, if this potentially ends up just being the first version of deb support, I'm on board. On the third hand, this may be a good use case for Greg's standalone Python...

How about this as a proposal: we introduce a python_source setting:

1. python_source = 'system' means use the system Python, with the same warning if there's a mismatch, and a system dependency on "python3" in the DEB bundle
2. python_source = 'deadsnakes' means add the deadsnakes Python to the build image, and add a system dependency on "python3.X" (as appropriate) in the DEB bundle
3. python_source = 'standalone' means download the support package from Greg's repo (URL hard-coded in the template, but overridable with support_package), and bundle Python with the app

My inclination is to use system as the default (as that's the option that allows run --no-docker without any large downloads), but I could be convinced otherwise on that.

This does mean we need to test 3 versions of the deb app,

p.s. i did find these folks advertising support for bundling python in a deb....just as doc of prior art.

Interesting... I'll check it out to see if there's something we can use/adapt/copy.

[freakboy3742]

p.s. i did find these folks advertising support for bundling python in a deb....just as doc of prior art.

Interesting... I'll check it out to see if there's something we can use/adapt/copy.

I asked my friend (who it turns out wrote a bunch of omnibus) - and it turns out it won't help us that much. Firstly, that repo is "last modified 10 years ago"; secondly, Omnibus isn't really doing much more than we're doing right now.

Omnibus is a tool for building a bunch of packages once you've already sorted out the "how to lay out your project on the filesystem" problem. From a packaging perspective, it's wrapping fpm which might be a marginal helper around building deb and rpm packages, but adding a dependency on a third party tool to avoid writing a single metadata file (losing the direct control over that file in the process) and invoking dpkg-deb directly isn't really worth it.

[rmartin16]

On the other hand, I think any future state should allow users to disable including a whole Python in their app when they know the System Python addresses their needs; so, if this potentially ends up just being the first version of deb support, I'm on board. On the third hand, this may be a good use case for Greg's standalone Python...

How about this as a proposal: we introduce a python_source setting:

1. python_source = 'system' means use the system Python, with the same warning if there's a mismatch, and a system dependency on "python3" in the DEB bundle
2. python_source = 'deadsnakes' means add the deadsnakes Python to the build image, and add a system dependency on "python3.X" (as appropriate) in the DEB bundle
3. python_source = 'standalone' means download the support package from Greg's repo (URL hard-coded in the template, but overridable with support_package), and bundle Python with the app

My inclination is to use system as the default (as that's the option that allows run --no-docker without any large downloads), but I could be convinced otherwise on that.

I like this idea. I think system is a reasonable default; I imagine the average situation would be people wanting to distribute to latest version of the distro. The warnings will tip them off once they try building for other versions.

This does mean we need to test 3 versions of the deb app,

:)

Firstly, that repo is "last modified 10 years ago";

whoops....how did i miss that, haha

it's wrapping fpm which might be a marginal helper around building deb and rpm packages, but adding a dependency on a third party tool to avoid writing a single metadata file (losing the direct control over that file in the process) and invoking dpkg-deb directly isn't really worth it.

Agreed; it looks like if we can get these control files populated properly, building the actual deb isn't that bad.

[freakboy3742]

My inclination is to use system as the default (as that's the option that allows run --no-docker without any large downloads), but I could be convinced otherwise on that.

I like this idea. I think system is a reasonable default; I imagine the average situation would be people wanting to distribute to latest version of the distro. The warnings will tip them off once they try building for other versions.

Ok - I'll update the PR to head in this direction.

As an aside - it occurs to me that converging on standalone would benefit both the AppImage backend (since we no longer need to maintain the Python-linux-support repo), and the Flatpak backend (as we would no longer need a full CPython source build, which is a significant part of the Flatpak build time). Sharing a common Linux build resource across all three would also be a nice from an architectural point of view.

Windows is already using an "official" embedded install; so I don't think there's a whole lot to be gained by switching there; Greg's macOS builds aren't universal, so they're not an option (at least until macOS drops x86_64 support). Plus, we'll have to maintain the Apple build chain for iOS anyway, as that's a platform he doesn't support at present.

[mhsmith]

e.g. Ubuntu 20.04 LTS doesn't run out until 2030 but its Python is done in 20 months :/).

Absolutely. From my non-linux-user perspective, I don't understand how Debian/Ubuntu considers this situation tenable. I hope they enjoy taking responsibility for backporting all the security patches (they are backporting security patches, right...?).

Yes, they are, and I think this is one of the main motivations for their rule of only having one copy of each component on the system. Once they release a security update for Python, they can be confident that every Python-based package on the system immediately benefits from it. It's very good for stability, but not so good for being able to run fast-moving projects like BeeWare.

If we follow our usual principle that the Python version of the app matches the version the user is running Briefcase with, then maybe we can detect the situation where that doesn't match the version supported by the distribution they're targeting, and suggest that they use the standalone mode instead.

[freakboy3742]

If we follow our usual principle that the Python version of the app matches the version the user is running Briefcase with, then maybe we can detect the situation where that doesn't match the version supported by the distribution they're targeting, and suggest that they use the standalone mode instead.

In the current "always use system" implementation, there's already a warning in place advising that the python being used for packaging doesn't match the briefcase python; it definitely makes sense to extend that message to include the tip that using standalone/deadsnakes is also an option.

In terms of exposure to this particular bug:

• If you're on Ubuntu 22.04 using system python, and you build a deb (with or without docker), you'll get a compliant package for 22.04, with no warning.
• If you're on Ubuntu 22.04 using a deadsnakes python, and you build a deb (with or without docker), you'll get a compliant package for 22.04, but with a warning.
• If you're on Ubuntu 22.04 and packaging for (say) 20.04, you'll get a warning.

Given that we're going to need to advise people to build multiple DEBs, one for each distro (Debian {10, 11} and Ubuntu {20.04, 22.04, ...}) then warnings are going to be mostly inevitable for the system config. The only way to avoid those warnings would be to install deadsnakes and use the Python for the target distro to build that package's deb; and you should probably be testing any DEBs that are produced before release anyway.

[freakboy3742]

I've now audited the output of the deb package with lintian; it's now runnining clean for me. I've also added in support for deadsnakes as a Python source. Still no tests yet, but feedback welcome.

[mhsmith]

I've also added in support for deadsnakes as a Python source

I can see this mentioned in the doucmentation, but is there an implementation yet?

Last time I checked, deadsnakes was only compatible with Ubuntu. There were some similar unofficial Python backports for Debian, but they weren't being updated so consistently.

So instead of hardcoding deadsnakes, how about allowing the user to specify an arbitrary repository address, and including instructions in the documentation for how to point this at deadsnakes?

[mhsmith]

I can see this mentioned in the doucmentation, but is there an implementation yet?

Cancel that: "large diffs are not rendered by default" strikes again.

[freakboy3742]

Last time I checked, deadsnakes was only compatible with Ubuntu. There were some similar unofficial Python backports for Debian, but they weren't being updated so consistently.

Oh bother - you're right. I had forgotten that it was an Ubuntu-specific thing.

So instead of hardcoding deadsnakes, how about allowing the user to specify an arbitrary repository address, and including instructions in the documentation for how to point this at deadsnakes?

I guess we could do that... but what other repository address are people going to use? Is there another repo of Python installs that Debian users are likely to know and trust? And - more importantly - one where the only change required is replacing ppa:deadsnakes/ppa with ppa:something-else? Deadsnakes has one notable change between "system" in that there's no python3.X-pip package (relying on ensure-pip instead) - are there any guarantees that a Debian deadsnakes-alike will have exactly the same configuration requirements?

I guess we could modify the code to accept any source starting with ppa: as an eternal repository, and leave it up to the user to find (or build) one that satisfies Debian's needs and is a behavioural mirror of Deadsnakes. However, my inclination is that unless there's a likely alternative that we actually anticipate people using, it's not worth exposing ourselves to the footgun. We'd be better served catching the use of debian+deadsnakes as an error, and documenting that limitation.

We can then focus on adding support for python-standalone as a "real" alternative. I was originally thinking of landing this PR with just "system" and "deadsnakes", and then tackle introducing python-standalone as a separate feature that all the Linux backends can benefit from; perhaps it's worth rolling standalone into this PR, and treat extending standalone into appImage and flatpak as follow-on work.

[mhsmith]

The original point of this PR was to allow developers to build packages that are trivial for an end-user to install, and "add this extra package repository to your system configuration" doesn't achieve that. So maybe the whole deadsnakes option is more trouble than it's worth, and we should just provide system and standalone.

[freakboy3742]

@mhsmith I guess this is the point where I don't have enough practical experience as a Linux user to say for certain.

My impression is that Linux native app packaging is always a bit eldritch. Linux users have already accepted a level of system administration responsibility that is higher than your average macOS/Windows user, but adding a PPA is (at least in the Ubuntu world) a not entirely unheard of as a practice. The DEB distribution process is already going to involve picking from multiple binaries. Having 1-2 extra sysadmin lines to add a PPA as a pre-requisite - something that end users may have already done if they're actively using Python - doesn't seem that onerous given the headaches that seem to exist with distributing packages in general.

From our end, the implementation is fairly straightforward - the same 2 sysadmin lines and one extra pip accomodation in the Dockerfile, and some handling on the build backend that will be shared with standalone. We may need to add some additional protection so that users on Debian are adequately warned, but that's a simple platform check using properties we've already extracted (app.target_vendor).

Overall - my impression is that deadsnakes support is something that enough users could find useful, and the implementation isn't that complex to maintain. I agree we should add standalone as well, but I don't think the situation is bad enough that we should strip deadsnakes.

[flowerncsu]

I guess this is the point where I don't have enough practical experience as a Linux user to say for certain.

Speaking as someone who's used a Linux system primarily for over a decade, my feeling tends to be that if I really need/want a given app, and I need to add a package repo to do it, I'll do it, but I definitely look for an alternative before doing that. On average I'd say I install something that needs one about once a year. So it may make sense to include as an option for the occasional use case where it matters, but I'm not sure if the feature maintenance is worth the small category of edge cases where system python wouldn't do. It might be good to nudge app creators towards figuring out how to work with system python instead.

[freakboy3742]

Thanks for that data point @flowerncsu.

Reading between the lines - is the implication that as a Python user, your general strategy for ensuring you have an up-to-date user-space Python is "update your distro", rather than "use Deadsnakes" - and that Deadsnakes is more of a cover-all for CI-type situations where you need to test against different Python versions?

[flowerncsu]

Thanks for that data point @flowerncsu.

Reading between the lines - is the implication that as a Python user, your general strategy for ensuring you have an up-to-date user-space Python is "update your distro", rather than "use Deadsnakes" - and that Deadsnakes is more of a cover-all for CI-type situations where you need to test against different Python versions?

Yes, typically I just install python from the standard package repo. It tends to lag about a version behind the most current stable python version but is plenty sufficient the majority of the time. I don't think I've ever put deadsnakes on my personal computer, though I've used it for work before.

[freakboy3742]

I've done some experimentation with python-standalone, and I'm not convinced it's actually a way forward - at least, not for native system packages.

Firstly, it's not quite as "standalone" as I thought it was. The Linux binaries appear to have lots of hard-coded paths that assume it's going to be installed in /usr/lib/python3.X, which doesn't really help us. These can be worked around, but it's hardly a "drop in embedded Python" base.

Secondly - and this is perhaps the bigger issue - packaging an embedded Python interpreter seems to cause the DFSG guidelines a minor hernia. Running lintian on a standalone-based package causes all manner of python3-script-but-no-python-dep errors and unusual-interpreter warnings.

It also throws embedded-library warnings, because the standard library binary modules (all of which are static linked into libpython) statically link their dependencies, such as bzip2. This is entirely understandable - but violates the DFSG because security updates to (for example) bzip2 won't be reflected in the version embedded in the Briefcase-packaged app.

This comes back to the "Debian really wants you to build an integrated system" argument - and my impression is that if we try to fight that, we're going to end up with a package that doesn't actually meet the expectations of Debian users - even if that does mean violating expectations they might have a Briefcase users.

To that end - I think Briefcase is the one that has to blink in this game of chicken. We're trying to follow the "when in Rome" philosophy, and build packages that appear "as native as possible"; in this case, I get the distinct impression that trying to do the Briefcase thing of adhering to a constant Python version will get us identified as tourists :-)

To be clear - I think it's still work adding standalone support for AppImage and Flatpak (and not that much effort, either); in both those situations, the Python interpreter is working in a sandbox, so the /usr/lib environment is at least closer to the actual runtime location (and to the extent it isn't, it's not competing with a system python).

[rmartin16]

I've done some experimentation with python-standalone, and I'm not convinced it's actually a way forward - at least, not for native system packages.

Firstly, it's not quite as "standalone" as I thought it was. The Linux binaries appear to have lots of hard-coded paths that assume it's going to be installed in /usr/lib/python3.X, which doesn't really help us. These can be worked around, but it's hardly a "drop in embedded Python" base.

Hmm....this seems to contradict a lot of the purpose of the standalone python project. Although, discussing this is probably moot given the rest of the post.

Secondly - and this is perhaps the bigger issue - packaging an embedded Python interpreter seems to cause the DFSG guidelines a minor hernia. Running lintian on a standalone-based package causes all manner of python3-script-but-no-python-dep errors and unusual-interpreter warnings.

It also throws embedded-library warnings, because the standard library binary modules (all of which are static linked into libpython) statically link their dependencies, such as bzip2. This is entirely understandable - but violates the DFSG because security updates to (for example) bzip2 won't be reflected in the version embedded in the Briefcase-packaged app.

This comes back to the "Debian really wants you to build an integrated system" argument - and my impression is that if we try to fight that, we're going to end up with a package that doesn't actually meet the expectations of Debian users - even if that does mean violating expectations they might have a Briefcase users.

To that end - I think Briefcase is the one that has to blink in this game of chicken. We're trying to follow the "when in Rome" philosophy, and build packages that appear "as native as possible"; in this case, I get the distinct impression that trying to do the Briefcase thing of adhering to a constant Python version will get us identified as tourists :-)

Or at least, this is unlikely to meet the expectations of a Debian administrator....albeit that's arguably just pedantic semantics.

At any rate, if one continues following Greg's path after python-build-standalone, we'll find PyOxidizer which he offered an explanation for and of....ultimately arguing that bundling Python is the most correct solution....even when a "system python" is available. While the "integrated system" debate remains, I wonder if Python (or perhaps any interpreter) could be reasonably considered external to the debate.

Nonetheless, we're already wearing togas, so why not also use the local thermae :) It seems most likely the majority of Briefcase projects targeting recent distro releases won't have terrible compatibility issues. Additionally, I can't find good evidence that bundling Python (whether for pure python projects or otherwise) is more popular than leveraging a system python. Personally, I run most things in a docker container....which I guess is nearly the opposite of integrated system in many ways.

p.s. This additionally makes we wonder if just creating a fat binary with pyinstaller or pyoxidizer to have dpkg drop in to a user's /usr/bin from a deb would be an acceptable alternative to trying to get python-build-standalone working. This would still go against the integrated system idea but would probably not frustrate lintian so much at least :/. I know you've expressed thoughts on this in the past; so, I won't ask you to reiterate them if the feelings are the same here.

[freakboy3742]

I've now got test coverage for a "system + dead snakes" version of this PR. The latest update has also done some refactoring, pulling out some common parts of the "running in Docker" code (mostly so I have less tests to write :-)

• There's no app verification at this point. That will need a change to the .github repo; I can't see a way to add deb app verification steps without breaking other PRs.
• This PR has deadsnakes support, but honestly, after all the discussion about it being an edge case, I'm not that wedded to it. Removing dead snakes support would simplify a bunch of things - most notably, the directory heirarchy.
• In the process of writing up the tests for this, it (re) occured to me how much this PR will be almost identical to an RPM and PACMAN patch. The only real differences are (a) the format of packaging metadata, and (b) the invocation of the package build process. It does make me wonder whether the right approach would be to have a "system" packaging backend, with deb, rpm, etc being format choices for that packaging. This wouldn't be that much work, as the Debian-specific pieces are actually fairly minimal, and could easily co-exist with other backends, and would open the door to "system" being a more appropriate default Linux build backend, with appImage and Flatpak being available as options.

[freakboy3742]

At any rate, if one continues following Greg's path after python-build-standalone, we'll find PyOxidizer which he offered an explanation for and of....ultimately arguing that bundling Python is the most correct solution....even when a "system python" is available.

Yeah - this is an extension of the "system python is not your python" debate; but I think our case is a little different, because what we're shipping is a binary app that uses the standard library, but none of the system-installed packages. We're just using the system Python for its interpreter.

Nonetheless, we're already wearing togas, so why not also use the local thermae :) It seems most likely the majority of Briefcase projects targeting recent distro releases won't have terrible compatibility issues.

This, for me, is the biggest reason why I don't see a huge issue with matching system Python, rather than Briefcase python.

p.s. This additionally makes we wonder if just creating a fat binary with pyinstaller or pyoxidizer to have dpkg drop in to a user's /usr/bin from a deb would be an acceptable alternative to trying to get python-build-standalone working. This would still go against the integrated system idea but would probably not frustrate lintian so much at least :/. I know you've expressed thoughts on this in the past; so, I won't ask you to reiterate them if the feelings are the same here.

FWIW: They are :-)

[freakboy3742]

Closing this in favour of #1106.