[BUG] Datapusher service do not start correctly

[konopkakamil]

Describe the bug
Datapusher is not working even though datapusher container is up. Cannot access datapusher service via localhost:8000, got "Internal Server Error" message. In ckan UI when trying to "Upload to DataStore" got red message: "Upload error: An Error occurred while sending the job: 500 Server Error: Internal Server Error for url: http://datapusher:8000/job"

Docker images affected

1. datapusher image, 2.10

Expected behaviour
Datapusher service is working correctly and available under localhost:8000

Logs
2024-04-26 16:52:06 *** Starting uWSGI 2.0.19.1 (64bit) on [Fri Apr 26 14:52:06 2024] ***
2024-04-26 16:52:06 compiled with version: 10.2.1 20201203 on 15 January 2024 15:31:15
2024-04-26 16:52:06 os: Linux-5.15.146.1-microsoft-standard-WSL2 #1 SMP Thu Jan 11 04:09:03 UTC 2024
2024-04-26 16:52:06 nodename: de54da6ced8a
2024-04-26 16:52:06 machine: x86_64
2024-04-26 16:52:06 clock source: unix
2024-04-26 16:52:06 detected number of CPU cores: 20
2024-04-26 16:52:06 current working directory: /srv/app
2024-04-26 16:52:06 detected binary path: /usr/bin/uwsgi
2024-04-26 16:52:06 !!! no internal routing support, rebuild with pcre support !!!
2024-04-26 16:52:06 your memory page size is 4096 bytes
2024-04-26 16:52:06 detected max file descriptor number: 1048576
2024-04-26 16:52:06 - async cores set to 2000 - fd table size: 1048576
2024-04-26 16:52:06 lock engine: pthread robust mutexes
2024-04-26 16:52:06 thunder lock: disabled (you can enable it with --thunder-lock)
2024-04-26 16:52:06 uWSGI http bound on :8000 fd 3
2024-04-26 16:52:06 uwsgi socket 0 bound to UNIX address /tmp/uwsgi.sock fd 6
2024-04-26 16:52:06 Python version: 3.8.15 (default, Nov 10 2022, 13:35:13) [GCC 10.2.1 20201203]
2024-04-26 16:52:06 Python main interpreter initialized at 0x55df27ca86a0
2024-04-26 16:52:06 python threads support enabled
2024-04-26 16:52:06 your server socket listen backlog is limited to 100 connections
2024-04-26 16:52:06 your mercy for graceful operations on workers is 60 seconds
2024-04-26 16:52:06 mapped 62923392 bytes (61448 KB) for 4000 cores
2024-04-26 16:52:06 *** Operational MODE: preforking+async ***
2024-04-26 16:52:07 Traceback (most recent call last):
2024-04-26 16:52:07 File "wsgi.py", line 20, in
2024-04-26 16:52:07 import ckanserviceprovider.web as web
2024-04-26 16:52:07 File "/usr/lib/python3.8/site-packages/ckanserviceprovider/web.py", line 13, in
2024-04-26 16:52:07 import flask_login as flogin
2024-04-26 16:52:07 File "/usr/lib/python3.8/site-packages/flask_login/init.py", line 12, in
2024-04-26 16:52:07 from .login_manager import LoginManager
2024-04-26 16:52:07 File "/usr/lib/python3.8/site-packages/flask_login/login_manager.py", line 5, in
2024-04-26 16:52:07 from flask import _request_ctx_stack
2024-04-26 16:52:07 ImportError: cannot import name '_request_ctx_stack' from 'flask' (/usr/lib/python3.8/site-packages/flask/init.py)
2024-04-26 16:52:07 unable to load app 0 (mountpoint='') (callable not found or import error)
2024-04-26 16:52:07 *** no app loaded. going in full dynamic mode ***
2024-04-26 16:52:07 *** uWSGI is running in multiple interpreter mode ***
2024-04-26 16:52:07 spawned uWSGI master process (pid: 1)
2024-04-26 16:52:07 spawned uWSGI worker 1 (pid: 8, cores: 2000)
2024-04-26 16:52:07 spawned uWSGI worker 2 (pid: 9, cores: 2000)
2024-04-26 16:52:07 spawned uWSGI http 1 (pid: 10)
2024-04-26 16:52:07 *** running gevent loop engine [addr:0x55df27c28d30] ***

[stojanovskis1]

@konopkakamil for ckan 2.10 i would recommend using xloader instead of datapusher

[BENETNATH]

Issue is related to Flask 2.4 deprecation of '_request_ctx_stack'
ImportError: cannot import name '_request_ctx_stack' from 'flask'

[BENETNATH]

it could be great to have a keitaroinc update on that, including all changes

@konopkakamil for ckan 2.10 i would recommend using xloader instead of datapusher

[stojanovskis1]

@BENETNATH there is a docker image with xloader installed, just add this line to the config for database to store jobs, but with the same value as you have for sqlalchemy.url
ckanext.xloader.jobs_db.uri = postgresql://ckan_default:pass@localhost/ckan_default
for more info check xloader documentation

[BENETNATH]

thanks @stojanovskis1 , no 2.11 version yet ?

[stojanovskis1]

@BENETNATH there is now 2.11.0-xloader version available

[BENETNATH]

Thanks !
Meanwhile I went the package road and have a working system in 2.1 with xloader.