superset 3.0.2 does again use old holidays which does not work with prophet 1.1.4/1.1.5

[nigzak]

A clear and concise description of what the bug is.

How to reproduce the bug

pull the official 3.0.2 container
install prophet

Expected results

it works

Actual results

it does not work

what actually happens.

 superset db upgrade
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 568, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 886, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 777, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (holidays 0.41 (/usr/local/lib/python3.9/site-packages), Requirement.parse('holidays<0.24,>=0.23'), {'apache-superset'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/superset", line 33, in <module>
    sys.exit(load_entry_point('apache-superset', 'console_scripts', 'superset')())
  File "/usr/local/bin/superset", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/local/lib/python3.9/importlib/metadata.py", line 86, in load
    module = import_module(match.group('module'))
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/app/superset/cli/main.py", line 28, in <module>
    from superset.cli.lib import normalize_token
  File "/app/superset/cli/lib.py", line 20, in <module>
    from superset import config
  File "/app/superset/config.py", line 38, in <module>
    import pkg_resources
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3243, in <module>
    def _initialize_master_working_set():
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3226, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3255, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 570, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 583, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 772, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'holidays<0.24,>=0.23' distribution was not found and is required by apache-superse

Checklist

Make sure to follow these steps before submitting your issue - thank you!

• I have checked the superset logs for python stacktraces and included it here as text if there are any.
• I have reproduced the issue with at least the latest released version of superset.
• I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

with prophet 1.1.5 also a much more newer version of holidays is required ... probably think about allowing more versions in superset than only one fixed version? This will always kill the integration

Refer old ticket #24619 where it was fixed for 2.1.1, seems it was not integrated in 3.x
At all same issue with prophet update now happens again (cannot install 1.1.5)

workaround

install prophet and holiday with a fixed version (1.1.3 prophet)

HINT

prophet 1.1.3 is two versions behind and probably should not being used anymore
prophet 1.1.4 is not compatible with holidays 0.23

ERROR: Cannot install holidays==0.23 and prophet==1.1.4 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested holidays==0.23
    prophet 1.1.4 depends on holidays>=0.25

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

might be good idea as asked to make not a fixed holiday package required in superset?

[michael-s-molina]

Hi @nigzak. Superset 3.0.2 uses Prophet 1.1.1. Check this PR for more context. It looks like the nasty bug affecting Prophet versions is fixed in 1.1.5 and this PR was recently merged to master and will be available in a future patch.

[nigzak]

Hi @michael-s-molina

did I understand correct that now prophet is in the docker image pre-installed and I don't need to manual install it?
In this case I could also remove the installation and it would not mind me anymore which holiday/prophet package is there or not ... For my cases also the prophet 1.1.4 worked perfectly fine, I think it depends what you are using where the "known bug" jumps into in prophet ...

[nigzak]

Hi @michael-s-molina

I downloaded the actual image (3.0.2) and there seems no prophet installed.
In general: would it not be good to include this package also in the official dockerhub image if the installation dependency is that bad working? If the installation is done "without thinking" it breaks the image complete
(pod is still running - but non-working)

I don't know if this would make sense or if this would have bad dependencies ... to me it smells a little bit that this can result in a view strange scenarios where the pod gets non-working ...

Ex: if you install the package default it kills the pod because the holiday package from prophet is newer and is updated which kills superset

pip install prophet   <== installs 1.1.5 with incompatibel version
pip install prophet==1.1.3    <== potential could (?) also installs incompatibel versio nbecause even 1.1.3 supports newer version
pip install holiday==... prophet==1.1.3  <== solves this issue for now

Problem: when superset updates the holiday package this results again into incompatibility ... I think this is a very bad dependency if you use the dockerhub image
Additional hint: if you execute "superset ..." in incompatible way via postexec you don't see that there is a incompatibility, you only see in the UI that there is something very faulty (we updated from 2.x to 3.0.2 and only have seen that some DB error was shown ... I exec'ed into the pod and executed "superset db upgrade" manual and only there saw the issue)

$ pip list
Package                Version     Editable project location
---------------------- ----------- -------------------------
alembic                1.6.5
amqp                   5.1.0
apache-superset        3.0.2       /app
apispec                6.3.0
appnope                0.1.3
astroid                2.6.6
asttokens              2.2.1
async-timeout          4.0.2
attrs                  23.1.0
Babel                  2.9.1
backcall               0.2.0
backoff                1.11.1
bcrypt                 4.0.1
billiard               3.6.4.0
boto3                  1.26.130
botocore               1.29.130
Brotli                 1.0.9
cached-property        1.5.2
cachelib               0.9.0
celery                 5.2.2
certifi                2023.5.7
cffi                   1.15.1
chardet                5.1.0
charset-normalizer     3.1.0
click                  8.1.3
click-didyoumean       0.3.0
click-option-group     0.5.5
click-plugins          1.1.1
click-repl             0.2.0
colorama               0.4.6
convertdate            2.4.0
cron-descriptor        1.2.24
croniter               1.0.15
cryptography           39.0.1
decorator              5.1.1
Deprecated             1.2.13
deprecation            2.1.0
dnspython              2.1.0
email-validator        1.1.3
et-xmlfile             1.1.0
executing              1.2.0
Flask                  2.2.5
Flask-AppBuilder       4.3.10
Flask-Babel            1.0.0
Flask-Caching          2.1.0
Flask-Compress         1.13
Flask-Cors             3.0.10
Flask-JWT-Extended     4.3.1
Flask-Limiter          3.3.1
Flask-Login            0.6.0
Flask-Migrate          3.1.0
Flask-SQLAlchemy       2.5.1
flask-talisman         1.0.0
Flask-WTF              1.1.1
func-timeout           4.3.5
future                 0.18.3
geographiclib          1.52
geopy                  2.2.0
greenlet               3.0.1
gunicorn               20.1.0
hashids                1.3.1
hijri-converter        2.3.1
holidays               0.23
humanize               3.11.0
idna                   3.2
ijson                  3.2.0.post0
importlib-metadata     6.6.0
importlib-resources    5.12.0
ipython                8.12.2
isodate                0.6.0
isort                  5.12.0
itsdangerous           2.1.2
jedi                   0.18.2
Jinja2                 3.1.2
jmespath               1.0.1
jsonlines              3.1.0
jsonschema             4.17.3
kombu                  5.2.4
korean-lunar-calendar  0.3.1
lazy-object-proxy      1.9.0
limits                 3.4.0
linear-tsv             1.1.0
Mako                   1.2.4
Markdown               3.3.4
markdown-it-py         2.2.0
MarkupSafe             2.1.1
marshmallow            3.19.0
marshmallow-sqlalchemy 0.23.1
matplotlib-inline      0.1.6
mccabe                 0.6.1
mdurl                  0.1.2
msgpack                1.0.2
mysqlclient            2.1.0
nh3                    0.2.11
numpy                  1.23.5
openpyxl               3.1.2
ordered-set            4.1.0
packaging              23.1
pandas                 1.5.3
paramiko               2.11.0
parsedatetime          2.6
parso                  0.8.3
pexpect                4.8.0
pgsanity               0.2.9
pickleshare            0.7.5
Pillow                 9.5.0
pip                    23.0.1
polyline               2.0.0
prison                 0.2.1
progress               1.6
prompt-toolkit         3.0.38
psycopg2-binary        2.9.6
ptyprocess             0.7.0
pure-eval              0.2.2
pure-sasl              0.6.2
pyarrow                12.0.0
pyasn1                 0.5.0
pyasn1-modules         0.3.0
pycparser              2.20
pydruid                0.6.5
Pygments               2.15.0
PyHive                 0.6.5
pyinstrument           4.4.0
PyJWT                  2.4.0
pylint                 2.9.6
PyMeeus                0.5.12
PyNaCl                 1.5.0
pyparsing              3.0.6
pyrsistent             0.19.3
python-dateutil        2.8.2
python-dotenv          0.19.0
python-editor          1.0.4
python-geohash         0.8.5
python-ldap            3.4.3
pytz                   2021.3
PyYAML                 6.0.1
redis                  4.5.4
requests               2.30.0
rfc3986                2.0.0
rich                   13.3.4
s3transfer             0.6.1
sasl                   0.3.1
selenium               3.141.0
setuptools             58.1.0
shortid                0.1.2
simplejson             3.17.3
six                    1.16.0
slack-sdk              3.21.3
SQLAlchemy             1.4.36
SQLAlchemy-Utils       0.38.3
sqloxide               0.1.33
sqlparse               0.4.4
sshtunnel              0.4.0
stack-data             0.6.2
tableschema            1.20.2
tabulate               0.8.9
tabulator              1.53.5
thrift                 0.16.0
thrift-sasl            0.4.3
toml                   0.10.2
traitlets              5.9.0
typing_extensions      4.4.0
unicodecsv             0.14.1
urllib3                1.26.6
vine                   5.0.0
wcwidth                0.2.5
Werkzeug               2.3.3
wheel                  0.41.3
wrapt                  1.12.1
WTForms                2.3.3
WTForms-JSON           0.3.5
xlrd                   2.0.1
XlsxWriter             3.0.7
zipp                   3.15.0

[michael-s-molina]

In general: would it not be good to include this package also in the official dockerhub image if the installation dependency is that bad working? If the installation is done "without thinking" it breaks the image complete

Hi @nigzak. We don't include Prophet because it's an optional dependency. If someone wishes to use it, they will need to install a compatible version taking into account Superset's dependencies.

[nigzak]

Hi @michael-s-molina

ok, that is sadly for me a bad dependency but it is as it is ...
Thanks

[michael-s-molina]

@nigzak #26431 is coming soon. Probably in 3.0.4 or 3.1.1.