Debian 12 Bookworm - bump dependencies to uptodate versions of cx_Freeze, zipp and importlib_metadata #23

as365n4 · 2023-01-22T11:46:11Z

MLAT client stopped working on Debian 12 Bookworm due to outdated cx_Freeze dependency.

This is becomming an issue soon as Debian 12 has entered Toolchain freeze on January 19th and is due for general release this year. https://lists.debian.org/debian-devel-announce/2023/01/msg00004.html

https://github.com/marcelotduarte/cx_Freeze/archive/refs/tags/6.13.1.tar.gz
c302e15bc9c488704f0b79b3896101ae6fc81b405ce489c29b15937111bc18f3

https://github.com/jaraco/zipp/archive/refs/tags/v3.11.0.tar.gz
d439d06f10385871258c29fb40839677a99b13720c2572b2673c14e73f2a550d

https://github.com/python/importlib_metadata/archive/refs/tags/v6.0.0.tar.gz
9e6fafdbf0601a5825c53a842f7bb6928c8e8eb6c46a228199ca02396cf6007a

as365n4 · 2023-01-22T13:41:10Z

output from systemctl status piaware.service:
piaware.service

For reference, this is a contiuation of https://discussions.flightaware.com/t/debian-bookworm-kills-piaware-service-minutes-after-starting/86018

mutability · 2023-02-01T11:07:13Z

Do you have a PR for this?

as365n4 · 2023-02-02T11:35:04Z

No, I don't have a Pull request for this.

I am building piaware for arm64 platform and whilst the script builds without an error, as soon as piaware is started it complains about MLAT Client not working and refers to broken cx-Freeze as shown in systemctl output in previous comment.

Others building for amd64 platform and having exactly the same issue, as referenced in the FlightAware discussion.

Unfortunately, I do not have the knowlegde to provide a fix for this.

mutability · 2023-02-02T18:43:37Z

Okay. Can you verify that building with the newer cxfreeze fixes the problem? I do not have a bookworm system to verify against.

as365n4 · 2023-02-04T12:18:59Z

This is not as easy as it seems to be...

zipp-3.12.0 does not have setup.py as specified in line 72 of the rules file
importlib_metadata-6.0.0 does not have setup.py as specified in line 73 of the rules file

keeping zipp-0.5.0 and importlib_metadat-4.3.1 and just using cx_Freeze-6.14.2 results in:

Processing dependencies for cx-Freeze==6.14.2
Searching for patchelf>=0.14
Reading https://pypi.org/simple/patchelf/
Download error on https://pypi.org/simple/patchelf/: [Errno 111] Connection refused -- Some packages may not be found!
Couldn't find index page for 'patchelf' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.org/simple/
Download error on https://pypi.org/simple/: [Errno 111] Connection refused -- Some packages may not be found!
No local packages or working download links found for patchelf>=0.14
error: Could not find suitable distribution for Requirement.parse('patchelf>=0.14')
make[1]: *** [debian/rules:74: build_cxfreeze] Error 1
make[1]: Leaving directory '/***/piaware_builder/package-bookworm'
make: *** [debian/rules:104: build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

but patchelf 0.14 is installed already

ADSB-Receiver-test:~/piaware_builder/package-bookworm$ apt search patchelf
Sorting... Done
Full Text Search... Done
patchelf/testing,now 0.14.3-1+b1 arm64 [installed]
modify properties of ELF executables

pypi.org has patchelf 0.17.2.0 so installed this via pip as well, but still no success.
Reinstalled with sudo pip and the build was successfully, no errors reported.

But there are warnings:

Finished processing dependencies for cx-Freeze==6.14.2
cd mlat-client && /home/user/piaware_builder/package-bookworm/debian/venv/bin/python setup.py install
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/usr/lib/python3/dist-packages/setuptools/command/easy_install.py:146: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(

And after installing piaware and starting the service via systemctl it reports that the mlat-client is still not working.

● piaware.service - FlightAware ADS-B uploader
Loaded: loaded (/lib/systemd/system/piaware.service; enabled; preset: enabled)
Active: active (running) since Sat 2023-02-04 12:44:08 GMT; 3min 7s ago
Docs: https://flightaware.com/adsb/piaware/
Main PID: 23673 (piaware)
Tasks: 3 (limit: 1045)
Memory: 14.4M
CPU: 19.761s
CGroup: /system.slice/piaware.service
├─23673 /usr/bin/piaware -p /run/piaware/piaware.pid -plainlog -statusfile /run/piaware/status.json
└─23750 /usr/lib/piaware/helpers/faup1090 --net-bo-ipaddr localhost --net-bo-port 30005 --stdout --lat 50.791 --lon 0.326

Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): module_init.run(name + "main")
Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): File "/home/user/piaware_builder/package-bookworm/debian/venv/lib/python3.11/site-packages/cx_Freeze-6.14.2-py3.11-linux-aarch64.egg/cx_Freeze/initscripts/console.py", line 18, in run
Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): exec(code, module_main.dict) # pylint: disable=exec-used
Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): File "/home/user/piaware_builder/package-bookworm/debian/venv/bin/fa-mlat-client", line 4, in
Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): import('pkg_resources').run_script('MlatClient==0.2.12', 'fa-mlat-client')
Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: mlat-client(24077): ModuleNotFoundError: No module named 'pkg_resources'
Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: got EOF from multilateration client
Feb 04 12:47:14 ADSB-Receiver-test piaware[23673]: fa-mlat-client exited with EXIT 255

So something else is broken, any ideas?

mutability · 2023-02-04T18:47:09Z

As you've discovered, getting this right is not as easy as it looks. A lot of the hoops that piaware_builder has to jump through are to do with getting cxfreeze working with the older versions of Python that Debian usually ships. Maybe bookworm is new enough that a recent cxfreeze can be installed directly with pip without needing to do the manual dependency dance.

That said, given that you still get the same error, it's presumably not the cxfreeze version that's causing problems at all. I have not looked into this at all yet, but maybe you need to force cxfreeze to include pkg_resources in the frozen binary.

as365n4 · 2023-02-04T19:05:57Z

I wouldn't blame Debian on this one, because Bookworm ships with Python 3.11 which is the latest release of Python.

Which pkg_resources need to be included? pypi.org gives over 1300 hits when searched for pkg_resources, but cx_Freeze seems not to have a resource/dev package.

mutability · 2023-02-05T09:04:48Z

Sure, so if bookworm has a recent python you probably don't need to jump through hoops to get recent packages installed there and much of the work that piaware_builder does with mangling specific versions of packages to get them to work with the older python toolchain is unnecessary.

Re pkg_resources I mean tell cxfreeze to include the version that's already present in the venv, in the frozen binary, by using e.g. --includes or --packages on the cxfreeze command line. See https://cx-freeze.readthedocs.io/en/latest/script.html

(The issue here is that it looks like the version of setuptools et al that bookworm has, is synthesizing a fa-mlat-client script that uses pkg_resources, but pkg_resources is not being included by cxfreeze in the frozen binary)

as365n4 · 2023-02-06T14:52:19Z

I've amended line 76/77 of the rules script to:

build_mlat-client:
cd mlat-client && $(VENV)/bin/python setup.py install --includes=pkg_resources

And now gcc exits with a fatal error:

Finished processing dependencies for importlib-metadata==4.3.1
cd cx_Freeze-6.8.3 && /home/user/piaware_builder/package-bullseye/debian/venv/bin/python setup.py install
running install
/usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/usr/lib/python3/dist-packages/setuptools/command/easy_install.py:146: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running bdist_egg
running egg_info
writing cx_Freeze.egg-info/PKG-INFO
writing dependency_links to cx_Freeze.egg-info/dependency_links.txt
writing entry points to cx_Freeze.egg-info/entry_points.txt
writing requirements to cx_Freeze.egg-info/requires.txt
writing top-level names to cx_Freeze.egg-info/top_level.txt
reading manifest file 'cx_Freeze.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'doc/src/license.rst'
writing manifest file 'cx_Freeze.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-aarch64/egg
running install_lib
running build_py
running build_ext
running install_include
aarch64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -O2 -ffile-prefix-map=/home/user/piaware_builder/package-bullseye=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/user/piaware_builder/package-bullseye/debian/venv/include -I/usr/include/python3.11 -c source/bases/Console.c -o build/temp.linux-aarch64-cpython-311/source/bases/Console.o
In file included from source/bases/Console.c:45:
source/bases/Common.c:7:10: fatal error: eval.h: No such file or directory
7 | #include <eval.h>
| ^~~~~~~~
compilation terminated.
error: command '/usr/bin/aarch64-linux-gnu-gcc' failed with exit code 1
make[1]: *** [debian/rules:74: build_cxfreeze] Error 1
make[1]: Leaving directory '/home/user/piaware_builder/package-bullseye'
make: *** [debian/rules:104: build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

I think this needs more work to get piaware_builder ready for Bookworm.
(I've used the Bullseye build script for simplicity this time)

Edit: Even with --includes=pkg_resources removed from the rules script the build fails with the above error.

mutability · 2023-02-07T05:42:01Z

Does /usr/include/python3.11/eval.h exist?

mutability · 2023-02-07T05:42:48Z

Also --includes needs to go on the invocation of cxfreeze, not the mlat-client install step.

as365n4 · 2023-02-07T09:31:03Z

content of /usr/include/python3.11

user@ADSB-Receiver-test:~/piaware_builder/package-bullseye/debian$ ls /usr/include/python3.11
abstract.h descrobject.h graminit.h moduleobject.h pydtrace.h pystate.h structseq.h
bltinmodule.h dictobject.h import.h object.h pyerrors.h pystrcmp.h sysmodule.h
boolobject.h dynamic_annotations.h internal objimpl.h pyexpat.h pystrtod.h token.h
bytearrayobject.h enumobject.h intrcheck.h opcode.h pyframe.h Python.h traceback.h
bytesobject.h errcode.h iterobject.h osdefs.h pyhash.h pythonrun.h tracemalloc.h
ceval.h exports.h listobject.h osmodule.h pylifecycle.h pythread.h tupleobject.h
codecs.h fileobject.h longobject.h patchlevel.h pymacconfig.h pytypedefs.h typeslots.h
compile.h fileutils.h marshal.h pybuffer.h pymacro.h rangeobject.h unicodeobject.h
complexobject.h floatobject.h memoryobject.h pycapsule.h pymath.h setobject.h warnings.h
cpython frameobject.h methodobject.h pyconfig.h pymem.h sliceobject.h weakrefobject.h
datetime.h genericaliasobject.h modsupport.h py_curses.h pyport.h structmember.h

as365n4 · 2023-02-07T09:49:26Z

Ok, I've put --includes=pkg_resources in line 74 of the rules script and the build fails with:

Finished processing dependencies for importlib-metadata==4.3.1
cd cx_Freeze-6.8.3 && /home/user/piaware_builder/package-bullseye/debian/venv/bin/python --includes=pkg_resources setup.py install
unknown option --includes=pkg_resources
usage: /home/user/piaware_builder/package-bullseye/debian/venv/bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.
make[1]: *** [debian/rules:74: build_cxfreeze] Error 2
make[1]: Leaving directory '/home/user/piaware_builder/package-bullseye'
make: *** [debian/rules:104: build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2

Edit: even when changed to cd cx_Freeze-6.8.3 && /home/user/piaware_builder/package-bullseye/debian/venv/bin/python setup.py install --includes=pkg_resources or cd cx_Freeze-6.8.3 && /home/user/piaware_builder/package-bullseye/debian/venv/bin/python setup.py --includes=pkg_resources install results in the above error.
Sorry, but I do not have the knowledge to code.

mutability · 2023-02-09T04:50:24Z

Well, 6.8.3 isn't going to work with Python 3.11 anyway? See https://cx-freeze.readthedocs.io/en/latest/versions.html
You need 6.14.

Anyway, I can't really continue to remotely debug this one. If you do manage to get it working, I'll take a PR. Otherwise I think you'll need to wait until we officially look at bookworm support (which is more likely to be around the time that the Pi Foundation switches to basing their stuff on bookworm)

WhoAmI0501 · 2023-07-19T12:43:37Z

I would like to draw your attention to my just opened pull request #24, in which context I am still looking for some more testers.

iwikus · 2023-12-08T14:14:38Z

I would like to draw your attention to my just opened pull request #24, in which context I am still looking for some more testers.

I have successfully used your repo to build and install on bookworm. Thank you.

ii piaware 8.2 amd64 Open Aviation Data Exchange Protocol for ADS-B FlightFeeders

mutability · 2023-12-08T14:53:05Z

9.0 has bookworm support now, so I'm going to close this.

iwikus · 2023-12-08T15:09:19Z

9.0 has bookworm support now, so I'm going to close this.

hm, but sensible-build.sh have no bookworm option

root@rx:~# git clone https://github.com/flightaware/piaware_builder
Cloning into 'piaware_builder'...
remote: Enumerating objects: 1252, done.
remote: Counting objects: 100% (312/312), done.
remote: Compressing objects: 100% (133/133), done.
remote: Total 1252 (delta 231), reused 255 (delta 179), pack-reused 940
Receiving objects: 100% (1252/1252), 223.59 KiB | 3.49 MiB/s, done.
Resolving deltas: 100% (841/841), done.
root@rx:~# cd piaware_builder
root@rx:~/piaware_builder# ls -la
total 104
drwxr-xr-x  8 root root  4096 Dec  8 16:06 .
drwx------ 13 root root  4096 Dec  8 16:06 ..
drwxr-xr-x  3 root root  4096 Dec  8 16:06 bookworm
drwxr-xr-x  3 root root  4096 Dec  8 16:06 bullseye
drwxr-xr-x  3 root root  4096 Dec  8 16:06 buster
-rw-r--r--  1 root root 38138 Dec  8 16:06 changelog
drwxr-xr-x  3 root root  4096 Dec  8 16:06 common
drwxr-xr-x  8 root root  4096 Dec  8 16:06 .git
-rw-r--r--  1 root root   120 Dec  8 16:06 .gitignore
-rw-r--r--  1 root root   883 Dec  8 16:06 importlib_metadata-2.1.2.patch
-rw-r--r--  1 root root   259 Dec  8 16:06 importlib_metadata-4.3.1.patch
-rw-r--r--  1 root root  2430 Dec  8 16:06 Jenkinsfile
-rw-r--r--  1 root root  2966 Dec  8 16:06 README.md
-rwxr-xr-x  1 root root  7190 Dec  8 16:06 sensible-build.sh
drwxr-xr-x  3 root root  4096 Dec  8 16:06 stretch
-rw-r--r--  1 root root   901 Dec  8 16:06 zipp-0.5.0.patch
root@rx:~/piaware_builder# ./s
sensible-build.sh  stretch/
root@rx:~/piaware_builder# ./sensible-build.sh
syntax: ./sensible-build.sh <stretch|buster|bullseye|xenial|bionic|disco>

Ah, seems it is only missing in that help message and accepting bookworm option.

mutability · 2023-12-08T15:18:29Z

It's there, I just forgot to update the usage message, yeah

WhoAmI0501 mentioned this issue Jul 19, 2023

Introduce Debian Bookworm support #24

Closed

mutability closed this as completed Dec 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Debian 12 Bookworm - bump dependencies to uptodate versions of cx_Freeze, zipp and importlib_metadata #23

Debian 12 Bookworm - bump dependencies to uptodate versions of cx_Freeze, zipp and importlib_metadata #23

as365n4 commented Jan 22, 2023 •

edited

Loading

as365n4 commented Jan 22, 2023 •

edited

Loading

mutability commented Feb 1, 2023

as365n4 commented Feb 2, 2023 •

edited

Loading

mutability commented Feb 2, 2023

as365n4 commented Feb 4, 2023 •

edited

Loading

mutability commented Feb 4, 2023

as365n4 commented Feb 4, 2023

mutability commented Feb 5, 2023 •

edited

Loading

as365n4 commented Feb 6, 2023 •

edited

Loading

mutability commented Feb 7, 2023

mutability commented Feb 7, 2023

as365n4 commented Feb 7, 2023

as365n4 commented Feb 7, 2023 •

edited

Loading

mutability commented Feb 9, 2023

WhoAmI0501 commented Jul 19, 2023

iwikus commented Dec 8, 2023

mutability commented Dec 8, 2023

iwikus commented Dec 8, 2023 •

edited

Loading

mutability commented Dec 8, 2023

Debian 12 Bookworm - bump dependencies to uptodate versions of cx_Freeze, zipp and importlib_metadata #23

Debian 12 Bookworm - bump dependencies to uptodate versions of cx_Freeze, zipp and importlib_metadata #23

Comments

as365n4 commented Jan 22, 2023 • edited Loading

as365n4 commented Jan 22, 2023 • edited Loading

mutability commented Feb 1, 2023

as365n4 commented Feb 2, 2023 • edited Loading

mutability commented Feb 2, 2023

as365n4 commented Feb 4, 2023 • edited Loading

mutability commented Feb 4, 2023

as365n4 commented Feb 4, 2023

mutability commented Feb 5, 2023 • edited Loading

as365n4 commented Feb 6, 2023 • edited Loading

mutability commented Feb 7, 2023

mutability commented Feb 7, 2023

as365n4 commented Feb 7, 2023

as365n4 commented Feb 7, 2023 • edited Loading

mutability commented Feb 9, 2023

WhoAmI0501 commented Jul 19, 2023

iwikus commented Dec 8, 2023

mutability commented Dec 8, 2023

iwikus commented Dec 8, 2023 • edited Loading

mutability commented Dec 8, 2023

as365n4 commented Jan 22, 2023 •

edited

Loading

as365n4 commented Jan 22, 2023 •

edited

Loading

as365n4 commented Feb 2, 2023 •

edited

Loading

as365n4 commented Feb 4, 2023 •

edited

Loading

mutability commented Feb 5, 2023 •

edited

Loading

as365n4 commented Feb 6, 2023 •

edited

Loading

as365n4 commented Feb 7, 2023 •

edited

Loading

iwikus commented Dec 8, 2023 •

edited

Loading