Upgrade to Python 3.8 or later

[mssalvatore]

Description

Python 3.7 goes EOL in June 2023. We need to upgrade to at least Python 3.8 (or preferably later).

Note: We should be able to use the manylinux1 distro of v3.9 from https://github.com/niess/python-appimage/releases on an Ubuntu Xenial build machine.

Considerations

• What is the oldest OS we want to support
• What is the newest version of Python we can make work on that system?
• How do our Agent and Island build systems need to change?

Notes

GLIBC

I think the oldest OS we can reasonable support is Ubuntu Xenial. It uses glibc 2.23 and still receives security updates through Ubuntu Advantage (until April 2026).

Xenial - glibc 2.23
Rhel/Rocky 8 - glibc 2.28
Rhel/Rocky 9 - glibc 2.34
Opensuse 15.4 - 2.31
Debian 9 stretch - glibc 2.24

Building Python on Xenial

It is possible to build Python 3.11 on Xenial. I've tried preliminarily running unit tests but run into issues. We could also try building 3.10. In order to build 3.10 or 3.11, we need to build openssl 1.1.1 from source.

It may be a smoother transition to go to 3.10 instead of 3.11.

Another option is to use the Python 3.9 AppImage. Only one unit test failed when I tried this approach. Later versions of Python are obviously preferable, but python 3.9 is good for 2 years, 7 months (which is almost as long as Xenial is supported).

Build notes

Here are some notes for building Python3.11 on Xenial

Install openssl from source
    ./config
    make
    make test
    sudo make install

install python deps
    sudo apt-get install libffi-dev

    sudo apt-get install build-essential gdb lcov pkg-config \
      libbz2-dev libffi-dev libgdbm-dev libgdbm-dev liblzma-dev \
      libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev \
      lzma lzma-dev tk-dev uuid-dev zlib1g-dev

Build Python
    Uncomment "static openssl" section in Modules/Setup
    Uncomment _bz2, _lzma, _uuid, _zlib, readline in Modules/Setup
    ./configure --with-pydebug --enable-shared # NOTE: --with-pydebug seems to cause issues with the encryption unit tests.
    make -s -j

Other notes:
    Need to `pip install cython` <- This should be added to Pipfile or Pipfile.lock
    Need to install freetds-dev in order to build pymssql

Python 3.9 AppImage

I was able to get unit tests to pass with the Python3.9 manylinux2014 AppImage from https://github.com/niess/python-appimage/releases. A couple of notes:

1. You need to unpin cryptography and cryptography-related dependencies
2. You need to run pipenv upgrade --dev for both the Island and Agent

WARNING You can't build a pyinstaller binary with the AppImage python, because PyInstaller requires that python be compiled with --enable-shared

Python 3.11.2 with pyenv

Installing 3.11.2 with pyenv is way easier than all of the above.

Tasks

Agent

• Create a new branch with python agent dependencies resynced with Python3.11 @mssalvatore
  • Remove pinned versions constrained by cryptography
  • Make sure you perform the sync on Windows
• Create the agent build container based on Ubuntu 16.04 @mssalvatore
  • Enable ESM (maybe this can help)
  • Install Python3.11 with pyenv (see cdrx/pyinstaller-linux:python3 to get started)
• Create an Infection Monkey repository on dockerhub (we'll need IT to help us securely store shared credentials) (0d) @cakekoa
  • Grant individual team members access (if possible)
  • Generate a token that jenkins can use to upload new containers
  • Share credentials securely
• Create an Ubuntu Advantage account for Infection monkey (0d)
  • Share credentials securely
• Create a jenkins job that will automatically recreate and republish the agent build image periodically (0d) @cakekoa
  • Finalize the docker image build scrips
  • Make the UA token a "credential" in jenkins so it doesn't get committed to any git repository
• Create an agent build script that pulls the agent build container from dockerhub and builds the agent (0d) @mssalvatore
  • Put script in the monkey repository so community members can easily build agents
• Modify the agent build job in jenkins to use the new agent build script (0d) @mssalvatore
• Fix "expanduser" unit tests on Windows (0d) - @shreyamalviya

Island

• Create a new branch with python Island dependencies resynced with Python3.11 (0d) @cakekoa
  • Remove pinned versions constrained by cryptography
  • Make sure you perform the sync on Windows
• Rework Docker build script to use python v3.11 (0d) @mssalvatore
• Rework the appimage build script to use python v3.11 (0d) @cakekoa
  • Use the latest manylinux version that runs on Ubuntu 18.04
• Update the Windows build machine to have python 3.11 (0d) @cakekoa
  • Update the windows agent build script if necessary
• Update deployment scripts to use python 3.11
  • Linux (0d) @ilija-lazoroski
  • Windows (0d) @cakekoa

Other

• Update documentation (0d)
  • Replace all references to python3.7 with python3.11 in documentation - @shreyamalviya
  • Update supported OSes page in documentation
• Update .travis.yml to use python 3.11 (0d) @mssalvatore
• Fix pre-commit (0d)
• Update hadoop build script and Pipfile (0d) @mssalvatore
• Test like mad (0d)