Missing dependency 'distro' in Python 3.8 or higher

[roaldnefs]

Description of Issue

Python's original platform.linux_distribution function will be removed in Python 3.8. In the #50822 and #51541, this is fixed by using the distro library on an ImportError in both salt/version.py and salt/grains/core.py. The distro library isn't mentioned as a dependency and therefore the install of salt>=2017.7.0 will result in an error.

Snippet importing the distro library:

# linux_distribution deprecated in py3.7
try:
    from platform import linux_distribution
    from platform import linux_distribution as _deprecated_linux_distribution

    def linux_distribution(**kwargs):
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
            return _deprecated_linux_distribution(**kwargs)
except ImportError:
    from distro import linux_distribution # <-- used but not installed by default

Setup

• python>=3.8
• salt>=2017.7.0

Steps to Reproduce Issue

The issue can be reproduced by installing Salt (via pip) in a Docker Python 3.8 container, by running the following command:

docker run -it python:3.8 /bin/bash -c "pip install salt"

This results in the following error message:

Collecting salt
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-zt9okj4a/salt/setup.py'"'"'; __file__='"'"'/tmp/pip-install-zt9okj4a/salt/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-zt9okj4a/salt/pip-egg-info
         cwd: /tmp/pip-install-zt9okj4a/salt/
    Complete output (14 lines):
    Traceback (most recent call last):
      File "/tmp/pip-install-zt9okj4a/salt/salt/version.py", line 15, in <module>
        from platform import linux_distribution as _deprecated_linux_distribution
    ImportError: cannot import name 'linux_distribution' from 'platform' (/usr/local/lib/python3.8/platform.py)
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-zt9okj4a/salt/setup.py", line 137, in <module>
        exec(compile(open(SALT_VERSION).read(), SALT_VERSION, 'exec'))
      File "/tmp/pip-install-zt9okj4a/salt/salt/version.py", line 22, in <module>
        from distro import linux_distribution
    ModuleNotFoundError: No module named 'distro'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Affected Versions

Quick test shows that Salt 2017.7.0 and higher are affected when being installed on Python 3.8.

[garethgreenaway]

@roaldnefs Thanks for the report.
@dmurphy18 @felippeb @waynew FYI, probably a good idea to update our dependencies to ensure distro is installed now that 3.8 has been released.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[roaldnefs]

Don't think this issue should be closed as I'm still able to reproduce the issue using docker run -it python:3.8 /bin/bash -c "pip install salt".

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[dmurphy18]

@roaldnefs The correct spec file used for salt resides in salt-pack-py3, for example: https://github.com/saltstack/salt-pack-py3/blob/develop/file_roots/pkg/salt/2019_2_2/rhel8/spec/salt.spec

I have a task to update the spec file current in salt itself for the latest version used, and now with Python 2 EOL'd, perhaps the issue of which to use is resolved.

see #50770

[tommyhp2]

Any chance of fixing this for FreeBSD 12.1?
`===> Configuring for py38-salt-2019.2.2_1
Traceback (most recent call last):
File "/wrkdirs/usr/ports/sysutils/py-salt/work-py38/salt-2019.2.2/salt/version.py", line 15, in
from platform import linux_distribution as _deprecated_linux_distribution
ImportError: cannot import name 'linux_distribution' from 'platform' (/usr/local/lib/python3.8/platform.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "setup.py", line 137, in
exec(compile(open(SALT_VERSION).read(), SALT_VERSION, 'exec'))
File "/wrkdirs/usr/ports/sysutils/py-salt/work-py38/salt-2019.2.2/salt/version.py", line 22, in
from distro import linux_distribution
ModuleNotFoundError: No module named 'distro'
*** Error code 1

Stop.
make: stopped in /usr/ports/sysutils/py-salt
`

Please advise. TIA!

[roaldnefs]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "setup.py", line 137, in
exec(compile(open(SALT_VERSION).read(), SALT_VERSION, 'exec'))
File "/wrkdirs/usr/ports/sysutils/py-salt/work-py38/salt-2019.2.2/salt/version.py", line 22, in
from distro import linux_distribution
ModuleNotFoundError: No module named 'distro'
*** Error code 1

@tommyhp2, as a workaround you can install distro before installing SaltStack. If the package isn't available you can use pip install distro for your corresponding Python version.

[dmurphy18]

@roaldnefs Currently Salt does not fully support Python 3.8, this is being addressed in the Sodium release. The distro package will be supplied as part of packaging at that time. It is currently being provided with the RHEL 8 Py 3 support and is a dependency with Debian 10, that is, it is loaded on Debian 10 by Salt.

Note: Python 3 was only fully support in version 2018.3.0 or higher.

If this information is sufficient, please consider closing this issue

[tommyhp2]

@dmurphy18 Thank you for the clarification. I was thinking of testing out salt capabilities. With the python2 at EOL, I'd figure to try with 3.8 and see what happens. I've already file a bug report at FreeBSD and it's addressed accordingly. Thank you.

[roaldnefs]

Thanks @dmurphy18 for your reply!

Closing this issue as Python 3.8 isn't officially supported.

[myii]

@roaldnefs Could you re-open this issue? It's now affecting all builds of the master branch using the salt-bootstrap:

• https://travis-ci.com/github/netmanagers/salt-image-builder/builds/165825747
  • UPDATE: Jobs have since passed after using rerunning a few hours later based on 9673de4. Originally were failing with 788834c.

[roaldnefs]

@roaldnefs Could you re-open this issue?

I've reopened the issue as it is now affecting builds of the master branch using the salt-bootstrap.

[myii]

@roaldnefs And it looks like it's been fixed by the latest commits that were merged since then! I got some advice from one of the devs about a commit that might resolve the issue. As I was adjusting the bootstrap to use another source for building Salt, I found that the original failures were now passing. So something between 788834c...9673de4.

[roaldnefs]

@myii, thanks for the feedback. Closing this issue again.

[asvany]

the dependency is still missing.

[alan-cugler]

dependency is still missing