Skip to content

Bundling Python apps

Jump to bottom
probonopd edited this page May 1, 2020 · 31 revisions

Here is an example on how to use it:

# Ideally run this inside the manylinux Docker container
# so that dependencies get bundled from that very container

# Download an AppImage of Python 3.8 built for manylinux
wget https://github.com/niess/python-appimage/releases/download/python3.8/python3.8.2-cp38-cp38-manylinux1_x86_64.AppImage

# Extract this AppImage
chmod +x python*-manylinux1_x86_64.AppImage
./python*-manylinux1_x86_64.AppImage --appimage-extract

# Install taguette into the extracted AppDir
./squashfs-root/AppRun -m pip install taguette

# Change AppRun so that it launches taguette
sed -i -e 's|/opt/python3.8/bin/python3.8|/usr/bin/taguette|g' ./squashfs-root/AppRun

# Test that it works
./squashfs-root/AppRun

# Edit the desktop file
mv squashfs-root/usr/share/applications/python3.8.2.desktop squashfs-root/usr/share/applications/taguette.desktop
sed -i -e 's|^Name=.*|Name=Taguette|g' squashfs-root/usr/share/applications/*.desktop
sed -i -e 's|^Exec=.*|Exec=taguette|g' squashfs-root/usr/share/applications/*.desktop
sed -i -e 's|^Icon=.*|Icon=taguette|g' squashfs-root/usr/share/applications/*.desktop
sed -i -e 's|^Comment=.*|Comment=Free and open source qualitative research tool|g' squashfs-root/usr/share/applications/*.desktop
sed -i -e 's|^Terminal=.*|Terminal=false|g' squashfs-root/usr/share/applications/*.desktop
sed -i -e 's|^Categories=.*|Categories=Science;|g' squashfs-root/usr/share/applications/*.desktop
rm squashfs-root/*.desktop
cp squashfs-root/usr/share/applications/*.desktop squashfs-root/

# Add icon
sudo apt-get -y install imagemagick
wget https://assets.gitlab-static.net/uploads/-/system/project/avatar/8339211/taguette.png
mkdir -p squashfs-root/usr/share/icons/hicolor/128x128/apps/
convert -resize 128x128 taguette.png squashfs-root/usr/share/icons/hicolor/128x128/apps/taguette.png
cp squashfs-root/usr/share/icons/hicolor/128x128/apps/taguette.png squashfs-root/

# Remove unneeded parts. TODO: This may need to be fine-tuned.
find squashfs-root/opt/python3.8/lib/python3.8/site-packages/PyQt5/Qt/plugins/platforms/ -type f -not -name libqxcb.so -delete
rm -rf squashfs-root/opt/python3.8/lib/python3.8/site-packages/PyQt5/Qt/plugins/egldeviceintegrations
rm -rf squashfs-root/opt/python3.8/lib/python3.8/site-packages/PyQt5/Qt/plugins/{audio,gamepads,geometryloaders,geoservices,mediaservice,playlistformats,position,renderplugins,sceneparsers,sensorgestures,sensors,sqldrivers,texttospeech,wayland*,webview,xcbglintegrations}
rm -rf squashfs-root/opt/python3.8/lib/python3.8/site-packages/PyQt5/Qt/qml
rm squashfs-root/opt/python3.8/lib/python3.8/site-packages/PyQt5/Qt/lib/libQt5{Bluetooth,Concurrent,Designer,Help,Location,Multimedia,Network,Nfc,OpenGL,Positioning,Qml,Quick,RemoteObjects,Sensors,SerialPort,Sql,Test,WaylandClient,WebChannel,WebSockets,Xml}*

# Convert back into an AppImage
export VERSION=$(cat squashfs-root/opt/python3.8/lib/python3.8/site-packages/taguette-*.dist-info/METADATA | grep "^Version:.*" | cut -d " " -f 2)
wget -c https://github.com/$(wget -q https://github.com/probonopd/go-appimage/releases -O - | grep "appimagetool-.*-x86_64.AppImage" | head -n 1 | cut -d '"' -f 2)
chmod +x appimagetool-*.AppImage
# 
# The following line does not work quite yet due to https://github.com/probonopd/go-appimage/issues/30
# ./appimagetool-*-x86_64.AppImage deploy squashfs-root/usr/share/applications/taguette.desktop
./appimagetool-*-x86_64.AppImage squashfs-root/ # Replace "1" with the actual version/build number

# Test it
./Taguette-1-x86_64.AppImage

It is actually possible to make the AppImage smaller by removing unneeded bits in a process known as "thinning".

Python knowledge is required to bundle Python apps. This page is for developers, not for users

Due to the way Python imports work and due to the way Python packages are installed on Debian and Ubuntu, it is not trivial to create working AppDirs from them.

To find out what Python files a Python app (let's say "SpiderOak") accesses, you can use

strace -eopen -f ./SpiderOak 2>&1 | grep / | grep -v ENOENT | cut -d "\"" -f 2 | sort | uniq > openedfiles

Instead, it is advised that you use workingenv.py, a script that sets up an isolated environment into which you can install your Python app and its dependencies that are not part of the standard library.

##  https://github.com/AppImage/AppImageKit/wiki/Bundling-Python-apps

python2 -m pip   install --user workingenv.py

## python     workingenv.py MyNewEnvironment
   python2 -m workingenv    MyNewEnvironment

cd MyNewEnvironment/
SITEPY=$(find -name site.py | head -n 1)

cat >> $SITEPY.new <<EOF
global USER_BASE, USER_SITE, ENABLE_USER_SITE
USER_BASE = None
USER_SITE = None
EOF

cat $SITEPY >>   $SITEPY.new
mv  $SITEPY.new  $SITEPY

source easy_install bin/activate
cd src/

# wget ...

tar xzf * ; cd * ; python2 setup.py install

#  or
## easy_install *.egg

Then, in your AppRun file, set $PYTHONPATH to point at MyNewEnvironment/lib/python2.6

Please let me know if there is an easier way to bundle Python apps properly.

Using pip

This is an example for making an AppImage from a Python 3 PyQt application using virtualenv and pip3 and the pkg2appimage tool:

app: mu.codewith.editor

ingredients:
  dist: trusty
  sources: 
    - deb http://us.archive.ubuntu.com/ubuntu/ trusty trusty-updates trusty-security main universe
    - deb http://us.archive.ubuntu.com/ubuntu/ trusty-updates main universe
    - deb http://us.archive.ubuntu.com/ubuntu/ trusty-security main universe
  packages:
    - python3.4-venv
  script:
    -  wget -c https://raw.githubusercontent.com/mu-editor/mu/master/conf/mu.codewith.editor.png
    -  wget -c https://raw.githubusercontent.com/mu-editor/mu/master/conf/mu.appdata.xml

script:
  - cp ../mu.codewith.editor.png ./usr/share/icons/hicolor/256x256/
  - cp ../mu.codewith.editor.png .
  - mkdir -p usr/share/metainfo/ ; cp ../mu.appdata.xml usr/share/metainfo/
  - virtualenv --python=python3 usr
  - ./usr/bin/pip3 install mu-editor
  - cat > usr/share/applications/mu.codewith.editor.desktop <<\EOF
  - [Desktop Entry]
(...)
  - cp usr/share/applications/mu.codewith.editor.desktop .
  - usr/bin/pip3 freeze | grep "mu-editor" | cut -d "=" -f 3 >> ../VERSION

Full, working example: https://github.com/AppImage/AppImages/blob/9249a99e653272416c8ee8f42cecdde12573ba3e/recipes/Mu.yml

Conda

Miniconda is a version of Python that can run on multiple distributions. It might be a good idea to use that, because it is supposed to be isolated from the Python in the base system (Linux distribution). https://github.com/HelloZeroNet/ZeroBundle/ appears to be using this. Let us know if you try this.

Here is a script that installs FreeCAD using Conda:

https://github.com/looooo/FreeCAD_Conda/blob/master/utils/appimage/freecad_app_image.sh

For more information, see https://conda-forge.github.io/

It looks like https://conda-forge.github.io/ has grown beyond "just" Python and may evolve into an option to populate any kind of AppDir.

AppImages converted from debs

If you convert a Python application from debs, then you should bundle Python inside the AppImage and make sure it does not load Python modules from the outside by the following trick:

We can do that by modifying usr/lib/python2.7/sitecustomize.py like this:

import sys,os
prefix = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(sys.path[0]))))
sys.path = [ prefix+s for s in sys.path if not s.startswith(prefix) ]

This works because sys.path[0] contains the directory from which Python was launched.

Avoid ctypes.util.find_library

The usual way that many wrapper libraries search for system libraries is via ctypes own function which on normal systems returns the .so filename. so in the case of VidCutter, for example:

import ctypes.util
sofile = ctypes.util.find_library('mpv')

the value of sofile should return as libmpv.so.1 but within the AppDir folder structure, and thus the resulting AppImage, this wouldn't work, returning None. It seems like ctypes.util.find_library isn't AppImage friendly (only searches outside of, but not inside the AppImage).You can work around it with a simple exception clause hardcoding the lib's filename like so:

sofile = ctypes.util.find_library('mpv')
if sofile is None:
    sofile = 'libmpv.so.1'

Please see https://github.com/ozmartian/vidcutter/issues/12 for more information.

Doing everything in Python

See https://github.com/benoit-pierre/plover/blob/9dc9fd5b2fb29b740eeb40f23d4a0630e8689b77/linux/appimage.sh for an example for how to create everything using Python, including downloading and building Python itself.

References

Clone this wiki locally