Update dependency "holidays" from ">=1.17 <18" to actual "0.28" as dependency for actual prophet 1.1.4 => using newer holidays killing superset V2.1.0

[nigzak]

If you install standard Prophet 1.1.4 (current version) it also installs as dependency holidays 0.28
This holiday version is incompatible with superset and results to crashing pod (superset DB update & more does not work)

How to reproduce the bug

pull image 2.1.0
install prophet inside image

Expected results

Installation works, no issues

Actual results

pod is not working, init/db upgrade failing

kubectl -n <....> exec supers-54fbb77f8-mtbgn superset db upgrade
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 568, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 886, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 777, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (holidays 0.28 (/usr/local/lib/python3.8/site-packages), Requirement.parse('holidays<0.18,>=0.17.2'), {'apache-superset'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/superset", line 33, in <module>
    sys.exit(load_entry_point('apache-superset', 'console_scripts', 'superset')())
  File "/usr/local/bin/superset", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/local/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/superset/cli/main.py", line 28, in <module>
    from superset.cli.lib import normalize_token
  File "/app/superset/cli/lib.py", line 20, in <module>
    from superset import config
  File "/app/superset/config.py", line 51, in <module>
    import pkg_resources
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3243, in <module>
    def _initialize_master_working_set():
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3226, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3255, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 570, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 583, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 772, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'holidays<0.18,>=0.17.2' distribution was not found and is required by apache-superset
command terminated with exit code 1

Additional Hints

Prophet 1.1.4 uses minimum holidays 0.20, default installing 0.28 (latest version)

https://github.com/facebook/prophet/releases

Holidays Release latest versions currently brings more regions
https://python-holidays.readthedocs.io/en/latest/changelog.html

Environment

(please complete the following information):

• superset version: 2.1.0 docker image

Checklist

Make sure to follow these steps before submitting your issue - thank you!

• I have checked the superset logs for python stacktraces and included it here as text if there are any.
• I have reproduced the issue with at least the latest released version of superset.
• I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

recommend to update holiday package in superset to the actual version 0.28 which also includes more regions

Workaround

Installing prophet and superset with dedicated versions avoids killing the pod completly. This should only be temporary used, don't know if prophet 1.1.3 getting problems in case of security (?)

pip install --no-cache ...<other packages>... holidays==0.17.2 prophet==1.1.3 

[rusackas]

Have you already investigated a fix for this? The title sounds like the fix if I'minterpreting this correctly. If so, a PR would be warmly welcomed :)

[EugeneTorap]

Hi @arkid15r @sebastianliebscher! Do you want to fix the issue?

[arkid15r]

Hi @arkid15r @sebastianliebscher! Do you want to fix the issue?

Hey @EugeneTorap,
I may be able to look into this over the weekend or on Monday.

If the fix only requires

to update holiday package in superset to the actual version 0.28 which also includes more regions

then it shouldn't be a hard one.

[EugeneTorap]

@arkid15r Thanks so much!

[arkid15r]

@arkid15r Thanks so much!

Thanks for keeping stakeholders notified, Eugene!

[nigzak]

Hi @rusackas

I am rather new to the superset on this level and also not very common with using GIT Pull requests so sorry that I did not try to make a pull request ..
My hope was that if I make everything I know in this ticket it would not hard to fix this issue for somebody who can make the change in minutes ... for me it would be currently rather hours ;)
If I find something similar somewhere next time I will check if I could also make a pull request :)

thanks for the quick responses and reacts here!

[arkid15r]

It seems that the issue is caused by the difference of approaches Superset (carefully pinned) and Prophet (minimum version) have been using for their requirements management. I filed #24647 but I don't think it provides anything but just a temporary holidays library version match for both packages.

I believe the easiest workaround for the issue would be something like pip install "holidays < 0.18" prophet (tested in apache/superset:2.1.1rc1). I hope this helps.

Let me know if you have ideas on how to tackle this in a better way.

Thank you!

[EugeneTorap]

The PR is merged which fixes the issue.
@rusackas @nigzak Pls, close the issue.

[nigzak]

Fix for this issue is merged, closing this ticket now (should be fixed in V2.1.1)