Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ModuleNotFoundError installing Superset 2.0.1 in a fresh virtual environment #22571

Closed
3 tasks done
alanorth opened this issue Jan 3, 2023 · 22 comments
Closed
3 tasks done
Assignees
Labels
#bug Bug report

Comments

@alanorth
Copy link

alanorth commented Jan 3, 2023

Trying to install Superset 2.0.1 in a fresh Python 3.8.10 virtual environment fails with ModuleNotFoundError.

How to reproduce the bug

# su - superset
$ python3.8 -m venv venv
$ source ./venv/bin/activate
$ pip install --upgrade setuptools pip wheel
$ pip install apache-superset==2.0.1
$ export PYTHONPATH=/home/superset
$ export FLASK_APP=superset
$ superset
Traceback (most recent call last):
  File "/home/superset/venv/bin/superset", line 5, in <module>
    from superset.cli.main import superset
  File "/home/superset/venv/lib/python3.8/site-packages/superset/__init__.py", line 21, in <module>
    from superset.app import create_app
  File "/home/superset/venv/lib/python3.8/site-packages/superset/app.py", line 23, in <module>
    from superset.initialization import SupersetAppInitializer
  File "/home/superset/venv/lib/python3.8/site-packages/superset/initialization/__init__.py", line 33, in <module>
    from superset.extensions import (
  File "/home/superset/venv/lib/python3.8/site-packages/superset/extensions/__init__.py", line 32, in <module>
    from superset.utils.cache_manager import CacheManager
  File "/home/superset/venv/lib/python3.8/site-packages/superset/utils/cache_manager.py", line 24, in <module>
    from superset.utils.core import DatasourceType
  File "/home/superset/venv/lib/python3.8/site-packages/superset/utils/core.py", line 76, in <module>
    from cryptography.hazmat.backends.openssl.x509 import _Certificate
ModuleNotFoundError: No module named 'cryptography.hazmat.backends.openssl.x509'

But clearly cryptography is installed:

$ pip install cryptography
Requirement already satisfied: cryptography in ./venv/lib/python3.8/site-packages (39.0.0)
Requirement already satisfied: cffi>=1.12 in ./venv/lib/python3.8/site-packages (from cryptography) (1.15.1)
Requirement already satisfied: pycparser in ./venv/lib/python3.8/site-packages (from cffi>=1.12->cryptography) (2.21)

Expected results

Superset runs.

Actual results

ModuleNotFoundError from Python.

Environment

System is Ubuntu 20.04 "focal" with Python 3.8.10.

Checklist

  • I have checked the superset logs for python stacktraces and included it here as text if there are any.
  • I have reproduced the issue with at least the latest released version of superset.
  • I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

The system is currently running Superset 2.0.0 fine in another virtualenv.

@alanorth alanorth added the #bug Bug report label Jan 3, 2023
@alanorth
Copy link
Author

alanorth commented Jan 3, 2023

I just noticed that Superset 2.0.1's requirements/base.txt says it wants cryptography==3.4.7. If I install that manually then superset runs as expected.

I have different ModuleNotFoundErrors every time I install a new version of Superset (I always file an issue here). In fact, looking at pip list in my virtual environment, I see several other packages that are the wrong version. What is going on with the Python packaging of the pip version that causes pip install apache-superset==2.0.1 to not install the exact version dependencies? Why do the Superset docs still recommend using the pip install method at all?

@io-ma
Copy link

io-ma commented Jan 3, 2023

I am having the exact same issue:
ModuleNotFoundError: No module named 'cryptography.hazmat.backends.openssl.x509'

Python 3.8.16, Ubuntu 18.04 LTS

superset db upgrade
Traceback (most recent call last):
  File "/home/io/superset/venv/bin/superset", line 33, in <module>
    sys.exit(load_entry_point('apache-superset==2.0.1', 'console_scripts', 'superset')())
  File "/home/io/superset/venv/bin/superset", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/io/superset/venv/lib/python3.8/site-packages/superset/__init__.py", line 21, in <module>
    from superset.app import create_app
  File "/home/io/superset/venv/lib/python3.8/site-packages/superset/app.py", line 23, in <module>
    from superset.initialization import SupersetAppInitializer
  File "/home/io/superset/venv/lib/python3.8/site-packages/superset/initialization/__init__.py", line 33, in <module>
    from superset.extensions import (
  File "/home/io/superset/venv/lib/python3.8/site-packages/superset/extensions/__init__.py", line 32, in <module>
    from superset.utils.cache_manager import CacheManager
  File "/home/io/superset/venv/lib/python3.8/site-packages/superset/utils/cache_manager.py", line 24, in <module>
    from superset.utils.core import DatasourceType
  File "/home/io/superset/venv/lib/python3.8/site-packages/superset/utils/core.py", line 76, in <module>
    from cryptography.hazmat.backends.openssl.x509 import _Certificate
ModuleNotFoundError: No module named 'cryptography.hazmat.backends.openssl.x509'

@zhaoyongjie
Copy link
Member

@alanorth try this snippet

pip uninstall cryptography
pip uninstall pyopenssl
pip install cryptography==3.4.7

@alanorth
Copy link
Author

alanorth commented Jan 4, 2023

@zhaoyongjie I ended up using this one liner shared in another issue. It installs the exact dependencies for Apache Superset 2.0.1, from its requirements/base.txt:

$ curl -sS https://raw.githubusercontent.com/apache/superset/2.0.1/requirements/base.txt | \
       tail -n +10 | \
       awk -v ORS=" " '/^[A-z]/{print}' | \
       xargs pip install apache-superset==2.0.1

I still don't understand why pip installing apache-superset==2.0.1 doesn't work. This is a recurring issue with Superset for me.

@io-ma
Copy link

io-ma commented Jan 4, 2023

this one liner worked for me too, thank you!

@Jika97
Copy link

Jika97 commented Jan 4, 2023

Same problem

Apply the solution of @zhaoyongjie, now i have :

from wtforms.ext.sqlalchemy.fields import QuerySelectField ModuleNotFoundError: No module named 'wtforms.ext'

@zhaoyongjie
Copy link
Member

Same problem

Apply the solution of @zhaoyongjie, now i have :

from wtforms.ext.sqlalchemy.fields import QuerySelectField ModuleNotFoundError: No module named 'wtforms.ext'

Are you on the Superset 2.0.1? I can't reproduce it on the master and 2.0.1.

@Jika97
Copy link

Jika97 commented Jan 4, 2023

Yes, I'm trying install Superset 2.0.1 on Rocky Linux 9.1 with Python 3.9 @zhaoyongjie, but i will try on Ubuntu 20.04

@xinbeiG
Copy link

xinbeiG commented Jan 4, 2023

Same problem

Apply the solution of @zhaoyongjie, now i have :

from wtforms.ext.sqlalchemy.fields import QuerySelectField ModuleNotFoundError: No module named 'wtforms.ext'

I got the same issue here, and this solution fixed it.

pip uninstall WTForms 
pip install WTForms==2.3.0

@Jika97
Copy link

Jika97 commented Jan 5, 2023

Thanks @XinbeiGong, it's worked !

I think it would be necessary to update the installation documentation, because it is a bit tedious to search all over the internet to install and operate this magnificent tool.

Indirectly this repels a lot of potential users.

@michael-s-molina
Copy link
Member

Tagging @eschutho @eric-briscoe @AAfghahi as owners of 2.0.1

@jsanko9
Copy link

jsanko9 commented Jan 17, 2023

Same problem
Apply the solution of @zhaoyongjie, now i have :
from wtforms.ext.sqlalchemy.fields import QuerySelectField ModuleNotFoundError: No module named 'wtforms.ext'

I got the same issue here, and this solution fixed it.

pip uninstall WTForms 
pip install WTForms==2.3.0

I was having this issue plus an issue with validate function:
validate() takes 1 positional argument but 2 were given

I think culprit was with WTForms, updating to 2.3.2 seems to have helped, looking at the change log they may have changed something in 2.3.1
https://wtforms.readthedocs.io/en/2.3.x/changes/#version-2-3-1
Posting it here in case someone encounters same thing.

pip uninstall WTForms 
pip install WTForms==2.3.2

@eschutho
Copy link
Member

Thanks for that info @jsanko9. It looks like we can pin WTForms to >=2.3.2 <3.0 and that should help with that issue.
It looks like there may be a similar issue with the cryptography package. We know that cryptography==3.4.7 works. We currently require "cryptography>=3.3.2". Does anyone know which version is failing? cc @zhaoyongjie?

@noahyetter-nb
Copy link

Thanks for that info @jsanko9. It looks like we can pin WTForms to >=2.3.2 <3.0 and that should help with that issue. It looks like there may be a similar issue with the cryptography package. We know that cryptography==3.4.7 works. We currently require "cryptography>=3.3.2". Does anyone know which version is failing? cc @zhaoyongjie?

I just did a fresh install and got cryptography 39.0.0
See also this issue #22743
I applied the fix reported there, which works, but now I have the problem with wtforms described above

(I can't help but notice that the dependencies that get installed by the process in Dockerfile, and the dependencies specified in setup.py, are significantly different. Also the non-Docker installation instructions are incorrect.)

@alonsogtouch
Copy link

alonsogtouch commented Jan 26, 2023

I got the same error on Fedora 37.

Just so you know I couldn't get it to work with Python default version on Fedora (3.11) since it's not compatible with lots of libraries and dependencies in Superset.

In case anyone is interested I first had to build Python 3.8 from source:

sudo dnf groupinstall "Development Tools"
sudo dnf install python3-devel openssl-devel zlib-devel bzip2-devel sqlite-devel libffi-devel
wget https://www.python.org/ftp/python/3.8.16/Python-3.8.16.tgz
tar xzf Python-3.8.16.tgz Python-3.8.16/
cd Python-3.8.16/
./configure --enable-optimizations --with-ensurepip=install --with-lto=full --enable-profiling
sudo make altinstall

This last line helps you leave your system-wide Python 3.11 intact. You can run python3.8 after that
and create a superset virtual enrivonment using

python3.8 -m venv superset

After activating the new virtual environment and installing apache-superset with pip I got errors when I ran

superset db upgrade

as stated in the documentation (https://superset.apache.org/docs/installation/installing-superset-from-scratch/):

(superset) [alonso@fedora pythonvenvs]$ superset db upgrade
/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/pandas/compat/__init__.py:124: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.
  warnings.warn(msg)
Traceback (most recent call last):
  File "/home/alonso/pythonvenvs/superset/bin/superset", line 5, in <module>
    from superset.cli.main import superset
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/__init__.py", line 21, in <module>
    from superset.app import create_app
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/app.py", line 23, in <module>
    from superset.initialization import SupersetAppInitializer
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/initialization/__init__.py", line 33, in <module>
    from superset.extensions import (
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/extensions/__init__.py", line 32, in <module>
    from superset.utils.cache_manager import CacheManager
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/utils/cache_manager.py", line 24, in <module>
    from superset.utils.core import DatasourceType
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/utils/core.py", line 76, in <module>
    from cryptography.hazmat.backends.openssl.x509 import _Certificate
ModuleNotFoundError: No module named 'cryptography.hazmat.backends.openssl.x509'

In order to fix this I had to run the lines @zhaoyongjie provided, which I reproduce below:

@alanorth try this snippet

pip uninstall cryptography
pip uninstall pyopenssl
pip install cryptography==3.4.7

After that I ran

export FLASK_APP=superset
superset db upgrade

And I got the following errors:

  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/initialization/__init__.py", line 476, in configure_data_sources
    ConnectorRegistry.register_sources(module_datasource_map)
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/connectors/connector_registry.py", line 42, in register_sources
    module_obj = __import__(module_name, fromlist=class_names)
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/connectors/sqla/__init__.py", line 17, in <module>
    from . import models, views
  File "/home/alonso/pythonvenvs/superset/lib/python3.8/site-packages/superset/connectors/sqla/views.py", line 28, in <module>
    from wtforms.ext.sqlalchemy.fields import QuerySelectField
ModuleNotFoundError: No module named 'wtforms.ext'

@jsanko9 helped with this, I'm partly quoting the reply:

pip uninstall WTForms 
pip install WTForms==2.3.2

With that I could successfully run

superset db upgrade

@zeynepkoyun
Copy link

@alanorth Hi,
You can try with pip install pyopenssl==22.1.0. Downloading this library fixed it for me.

@cwegener
Copy link
Contributor

cwegener commented Feb 15, 2023

FWIW, there is a merged PR that removes the use of the private cryptography API _Certificate class by replacing it with the public version (Certificate) and at the same time pins the cryptography version.

#22744

@prakharcode
Copy link

prakharcode commented Feb 17, 2023

Fixed installation, working on ubuntu 22.10

pyenv version

3.9.16 (set by /root/.pyenv/version)
pip freeze

aiohttp==3.8.4
aiosignal==1.3.1
alembic==1.9.4
amqp==5.1.1
apache-superset==2.0.1
apispec==5.2.2
async-generator==1.10
async-timeout==4.0.2
attrs==22.2.0
Babel==2.11.0
backoff==2.2.1
billiard==3.6.4.0
bleach==3.3.1
Brotli==1.0.9
cachelib==0.4.1
celery==5.2.7
certifi==2022.12.7
cffi==1.15.1
charset-normalizer==3.0.1
click==8.1.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
colorama==0.4.6
convertdate==2.4.0
cron-descriptor==1.2.35
croniter==1.3.8
cryptography==38.0.2
deprecation==2.1.0
dnspython==2.3.0
email-validator==1.3.1
exceptiongroup==1.1.0
Flask==2.1.0
Flask-AppBuilder==4.2.0
Flask-Babel==2.0.0
Flask-Caching==1.11.1
Flask-Compress==1.13
Flask-JWT-Extended==4.4.4
Flask-Login==0.6.2
Flask-Migrate==4.0.4
Flask-SQLAlchemy==2.5.1
flask-talisman==1.0.0
Flask-WTF==1.1.1
frozenlist==1.3.3
func-timeout==4.3.5
geographiclib==2.0
geopy==2.3.0
graphlib-backport==1.0.3
gunicorn==20.1.0
h11==0.14.0
hashids==1.3.1
hijri-converter==2.2.4
holidays==0.14.2
humanize==4.6.0
idna==3.4
importlib-metadata==6.0.0
isodate==0.6.1
itsdangerous==2.1.2
Jinja2==3.1.2
jsonschema==4.17.3
kombu==5.2.4
korean-lunar-calendar==0.3.1
Mako==1.2.4
Markdown==3.4.1
MarkupSafe==2.0.1
marshmallow==3.19.0
marshmallow-enum==1.5.1
marshmallow-sqlalchemy==0.26.1
msgpack==1.0.4
multidict==6.0.4
numpy==1.22.1
outcome==1.2.0
packaging==23.0
pandas==1.3.5
parsedatetime==2.6
pgsanity==0.2.9
polyline==2.0.0
prison==0.2.1
prompt-toolkit==3.0.36
psycopg2-binary==2.9.5
pyarrow==5.0.0
pycparser==2.21
PyJWT==2.6.0
PyMeeus==0.5.12
pyparsing==3.0.9
pyrsistent==0.19.3
PySocks==1.7.1
python-dateutil==2.8.2
python-dotenv==0.21.1
python-geohash==0.8.5
pytz==2022.7.1
PyYAML==6.0
redis==4.5.1
selenium==4.8.0
simplejson==3.18.3
six==1.16.0
slackclient==2.5.0
sniffio==1.3.0
sortedcontainers==2.4.0
SQLAlchemy==1.3.24
SQLAlchemy-Utils==0.37.9
sqlparse==0.3.0
tabulate==0.8.9
trio==0.22.0
trio-websocket==0.9.2
typing-extensions==3.10.0.2
urllib3==1.26.14
vine==5.0.0
wcwidth==0.2.6
webencodings==0.5.1
Werkzeug==2.0.0
wsproto==1.2.0
WTForms==2.3.3
WTForms-JSON==0.3.5
yarl==1.8.2
zipp==3.13.0

@EMCP
Copy link

EMCP commented Feb 26, 2023

Can confirm, Ubuntu 20.04 py3.8 got stood up successfully with the following

pip install --upgrade pip
sudo -u ubuntu pip install --upgrade pip --no-warn-script-location
sudo -u ubuntu pip install testresources --no-warn-script-location
sudo -u ubuntu pip install aiohttp==3.8.1 aiosignal==1.2.0 alembic==1.6.5 amqp==5.1.0 apispec[yaml]==3.3.2 async-timeout==4.0.2 attrs==21.2.0 babel==2.9.1 backoff==1.11.1 billiard==3.6.4.0 bleach==3.3.1 brotli==1.0.9 cachelib==0.4.1 celery==5.2.2 cffi==1.14.6 charset-normalizer==2.0.4 click==8.0.4 click-didyoumean==0.3.0 click-plugins==1.1.1 click-repl==0.2.0 colorama==0.4.4 convertdate==2.4.0 cron-descriptor==1.2.24 croniter==1.0.15 cryptography==3.4.7 deprecation==2.1.0 dnspython==2.1.0 email-validator==1.1.3 flask==2.0.3 flask-appbuilder==4.1.3 flask-babel==1.0.0 flask-caching==1.10.1 flask-compress==1.10.1 flask-jwt-extended==4.3.1 flask-login==0.4.1 flask-migrate==3.1.0 flask-sqlalchemy==2.5.1 flask-talisman==0.8.1 flask-wtf==0.14.3 frozenlist==1.3.0 func-timeout==4.3.5 geographiclib==1.52 geopy==2.2.0 graphlib-backport==1.0.3 gunicorn==20.1.0 hashids==1.3.1 holidays==0.14.2 humanize==3.11.0 idna==3.2 isodate==0.6.0 itsdangerous==2.1.1 jinja2==3.0.3 jsonschema==3.2.0 kombu==5.2.4 korean-lunar-calendar==0.2.1 mako==1.1.4 markdown==3.3.4 markupsafe==2.0.1 marshmallow==3.13.0 marshmallow-enum==1.5.1 marshmallow-sqlalchemy==0.23.1 msgpack==1.0.2 multidict==5.1.0 numpy==1.22.1 packaging==21.3 pandas==1.3.4 parsedatetime==2.6 pgsanity==0.2.9 polyline==1.4.0 prison==0.2.1 prompt-toolkit==3.0.28 pyarrow==5.0.0 pycparser==2.20 pyjwt==2.4.0 pymeeus==0.5.11 pyparsing==3.0.6 pyrsistent==0.16.1 python-dateutil==2.8.2 python-dotenv==0.19.0 python-editor==1.0.4 python-geohash==0.8.5 pytz==2021.3 pyyaml==5.4.1 redis==3.5.3 selenium==3.141.0 simplejson==3.17.3 six==1.16.0 slackclient==2.5.0 sqlalchemy==1.3.24 sqlalchemy-utils==0.37.8 sqlparse==0.3.0 tabulate==0.8.9 typing-extensions==3.10.0.0 urllib3==1.26.6 vine==5.0.0 wcwidth==0.2.5 webencodings==0.5.1 werkzeug==2.0.3 wtforms==2.3.3 wtforms-json==0.3.3 yarl==1.6.3 --no-warn-script-location
sudo -u ubuntu pip install psycopg2-binary pillow gunicorn gevent --no-warn-script-location
sudo -u ubuntu pip install requests --upgrade --no-warn-script-location
sudo -u ubuntu pip install trino thrift pyhive --no-warn-script-location

status-set maintenance "About to pip install apache-superset $(date +"%H:%M")"
juju-log -l "WARNING" "About to pip install apache-superset"
sudo -u ubuntu pip install apache-superset --no-warn-script-location

@TrabajadorPraOleo
Copy link

TrabajadorPraOleo commented Apr 14, 2023

I install specifics the versions for the frameworks (WTForms, cryptography, pyopenssl ...) but it has not been solved. I have checked the versions in the file requirements/base.txt.

The error comes out when I enter this command:
-sudo docker-compose up

Captura desde 2023-04-14 13-27-30

@CharlesYe8848
Copy link

CharlesYe8848 commented Apr 14, 2023 via email

@alanorth
Copy link
Author

alanorth commented May 5, 2023

Closing because we realized that pip install apache-superset is apparently not meant to work. In other threads the maintainers have commented that we're supposed to be using the requirements file with pip. 🤷

For better or worse, now I always install Superset using a one liner based on the example given earlier in this issue:

$ curl -sS https://raw.githubusercontent.com/apache/superset/2.0.1/requirements/base.txt | \
       tail -n +10 | \
       awk -v ORS=" " '/^[A-z]/{print}' | \
       xargs pip install apache-superset==2.0.1

@alanorth alanorth closed this as completed May 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
#bug Bug report
Projects
None yet
Development

No branches or pull requests