Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛Computational backend: disappearing logs #4265

Merged
merged 32 commits into from
May 25, 2023
Merged

🐛Computational backend: disappearing logs #4265

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
57fa2a7
refactor
sanderegg May 22, 2023
2fe24d2
cleanup
sanderegg May 22, 2023
5e6b40d
better logs check
sanderegg May 22, 2023
1fee582
cleanup
sanderegg May 22, 2023
a503cba
cleanup
sanderegg May 23, 2023
470ca12
upgrade dask requirements
sanderegg May 23, 2023
8ef7ea5
add test for progress logs
sanderegg May 23, 2023
0334357
ensure progress 0 and 1 are always sent
sanderegg May 23, 2023
3d20256
refactor
sanderegg May 23, 2023
188309d
rename
sanderegg May 23, 2023
c8ac8cc
refactor and prepare for testing
sanderegg May 23, 2023
fd9ed41
simplify
sanderegg May 23, 2023
232c984
test that logs are going through
sanderegg May 23, 2023
1a4f1ae
refactor
sanderegg May 23, 2023
a8e2f9a
refactor
sanderegg May 23, 2023
0d61f65
fix test
sanderegg May 23, 2023
0e008a9
upgrade dask in director-v2
sanderegg May 23, 2023
59a98d4
upgraded dask gateway
sanderegg May 23, 2023
69d1064
upgrade dask-sidecar
sanderegg May 23, 2023
be670e7
upgrade director-v2
sanderegg May 23, 2023
24e80f4
update dask-sidecar
sanderegg May 24, 2023
df05626
upgraded dask
sanderegg May 24, 2023
b174a73
dask-gateway upgrade
sanderegg May 24, 2023
1d4b59f
refactor
sanderegg May 24, 2023
fb5d322
fix test after upgrade to latest dask
sanderegg May 24, 2023
8db93d1
checking pubsub
sanderegg May 24, 2023
c00f852
added test for checking a lot of messages get correctly transmitted
sanderegg May 24, 2023
dcd22a1
fixing new dask version issues
sanderegg May 24, 2023
7444ba7
fix dask to 2023.3.x
sanderegg May 25, 2023
60ccb02
block director-v2 dask version
sanderegg May 25, 2023
833458c
fix dask version in gateway to 2023.3
sanderegg May 25, 2023
acaef9b
fix version of dask
sanderegg May 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions services/api-server/requirements/_test.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,7 @@ pyyaml==5.4.1
# responses
requests==2.31.0
# via
# -c requirements/_base.txt
# docker
# moto
# responses
Expand Down
4 changes: 3 additions & 1 deletion services/catalog/requirements/_test.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,9 @@ python-dateutil==2.8.2
# -c requirements/_base.txt
# faker
requests==2.31.0
# via docker
# via
# -c requirements/_base.txt
# docker
respx==0.20.1
# via -r requirements/_test.in
six==1.15.0
Expand Down
53 changes: 28 additions & 25 deletions services/dask-sidecar/requirements/_base.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
#
# pip-compile --output-file=requirements/_base.txt --resolver=backtracking --strip-extras requirements/_base.in
#
aio-pika==9.0.4
aio-pika==9.0.7
# via -r requirements/../../../packages/service-library/requirements/_base.in
aiobotocore==2.4.2
aiobotocore==2.5.0
# via s3fs
aiodebug==2.3.0
# via -r requirements/../../../packages/service-library/requirements/_base.in
Expand All @@ -26,7 +26,7 @@ aiohttp==3.8.4
# s3fs
aioitertools==0.11.0
# via aiobotocore
aiormq==6.7.3
aiormq==6.7.6
# via aio-pika
aiosignal==1.3.1
# via aiohttp
Expand All @@ -47,9 +47,9 @@ blosc==1.11.1
# via -r requirements/_base.in
bokeh==2.4.3
# via dask
botocore==1.27.59
botocore==1.29.76
# via aiobotocore
certifi==2022.12.7
certifi==2023.5.7
# via requests
charset-normalizer==3.1.0
# via
Expand All @@ -65,38 +65,38 @@ cloudpickle==2.2.1
# via
# dask
# distributed
dask==2023.3.0
dask==2023.3.2
# via
# -r requirements/../../../packages/dask-task-models-library/requirements/_base.in
# -r requirements/_base.in
# dask-gateway
# distributed
dask-gateway==2023.1.1
# via -r requirements/_base.in
distributed==2023.3.0
distributed==2023.3.2
# via
# dask
# dask-gateway
dnspython==2.3.0
# via email-validator
email-validator==1.3.1
email-validator==2.0.0.post2
# via pydantic
frozenlist==1.3.3
# via
# aiohttp
# aiosignal
fsspec==2023.3.0
fsspec==2023.5.0
# via
# -r requirements/_base.in
# dask
# s3fs
heapdict==1.0.1
# via zict
idna==3.4
# via
# email-validator
# requests
# yarl
importlib-metadata==6.6.0
# via dask
jinja2==3.1.2
# via
# -c requirements/../../../packages/settings-library/requirements/../../../requirements/constraints.txt
Expand All @@ -123,22 +123,22 @@ multidict==6.0.4
# via
# aiohttp
# yarl
numpy==1.24.2
numpy==1.24.3
# via bokeh
packaging==23.0
packaging==23.1
# via
# bokeh
# dask
# distributed
pamqp==3.2.1
# via aiormq
partd==1.3.0
partd==1.4.0
# via dask
pillow==9.4.0
pillow==9.5.0
# via bokeh
psutil==5.9.4
psutil==5.9.5
# via distributed
pydantic==1.10.2
pydantic==1.10.8
# via
# -r requirements/../../../packages/dask-task-models-library/requirements/../../../packages/models-library/requirements/_base.in
# -r requirements/_base.in
Expand All @@ -160,13 +160,13 @@ pyyaml==5.4.1
# dask
# dask-gateway
# distributed
redis==4.5.4
redis==4.5.5
# via
# -c requirements/../../../packages/settings-library/requirements/../../../requirements/constraints.txt
# -r requirements/../../../packages/service-library/requirements/_base.in
requests==2.31.0
# via fsspec
s3fs==2023.3.0
s3fs==2023.5.0
# via fsspec
six==1.16.0
# via
Expand All @@ -183,36 +183,39 @@ toolz==0.12.0
# dask
# distributed
# partd
tornado==6.2
tornado==6.3.2
# via
# bokeh
# dask-gateway
# distributed
tqdm==4.65.0
# via -r requirements/../../../packages/service-library/requirements/_base.in
typer==0.7.0
typer==0.9.0
# via -r requirements/../../../packages/settings-library/requirements/_base.in
typing-extensions==4.5.0
typing-extensions==4.6.1
# via
# aiodebug
# aiodocker
# bokeh
# pydantic
urllib3==1.26.14
# typer
urllib3==1.26.16
# via
# -c requirements/../../../packages/settings-library/requirements/../../../requirements/constraints.txt
# botocore
# distributed
# requests
wrapt==1.15.0
# via aiobotocore
yarl==1.8.2
yarl==1.9.2
# via
# aio-pika
# aiohttp
# aiormq
zict==2.2.0
zict==3.0.0
# via distributed
zipp==3.15.0
# via importlib-metadata

# The following packages are considered to be unsafe in a requirements file:
# setuptools
26 changes: 15 additions & 11 deletions services/dask-sidecar/requirements/_dask-distributed.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,22 +16,22 @@ cloudpickle==2.2.1
# -c requirements/./_base.txt
# dask
# distributed
dask==2023.3.0
dask==2023.3.2
# via
# -r requirements/_dask-distributed.in
# distributed
distributed==2023.3.0
distributed==2023.3.2
# via
# -c requirements/./_base.txt
# dask
fsspec==2023.3.0
fsspec==2023.5.0
# via
# -c requirements/./_base.txt
# dask
heapdict==1.0.1
importlib-metadata==6.6.0
# via
# -c requirements/./_base.txt
# zict
# dask
jinja2==3.1.2
# via
# -c requirements/./_base.txt
Expand All @@ -51,16 +51,16 @@ msgpack==1.0.5
# via
# -c requirements/./_base.txt
# distributed
packaging==23.0
packaging==23.1
# via
# -c requirements/./_base.txt
# dask
# distributed
partd==1.3.0
partd==1.4.0
# via
# -c requirements/./_base.txt
# dask
psutil==5.9.4
psutil==5.9.5
# via
# -c requirements/./_base.txt
# distributed
Expand All @@ -83,15 +83,19 @@ toolz==0.12.0
# dask
# distributed
# partd
tornado==6.2
tornado==6.3.2
# via
# -c requirements/./_base.txt
# distributed
urllib3==1.26.14
urllib3==1.26.16
# via
# -c requirements/./_base.txt
# distributed
zict==2.2.0
zict==3.0.0
# via
# -c requirements/./_base.txt
# distributed
zipp==3.15.0
# via
# -c requirements/./_base.txt
# importlib-metadata
31 changes: 16 additions & 15 deletions services/dask-sidecar/requirements/_test.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,24 +23,24 @@ attrs==21.4.0
# jschema-to-python
# jsonschema
# sarif-om
aws-sam-translator==1.66.0
aws-sam-translator==1.67.0
# via cfn-lint
aws-xray-sdk==2.12.0
# via moto
blinker==1.6.2
# via flask
boto3==1.24.59
boto3==1.26.76
# via
# aws-sam-translator
# moto
botocore==1.27.59
botocore==1.29.76
# via
# -c requirements/_base.txt
# aws-xray-sdk
# boto3
# moto
# s3transfer
certifi==2022.12.7
certifi==2023.5.7
# via
# -c requirements/_base.txt
# requests
Expand All @@ -57,7 +57,7 @@ click==8.1.3
# via
# -c requirements/_base.txt
# flask
coverage==7.2.5
coverage==7.2.6
# via
# -r requirements/_test.in
# pytest-cov
Expand All @@ -79,7 +79,7 @@ ecdsa==0.18.0
# sshpubkeys
exceptiongroup==1.1.1
# via pytest
faker==18.7.0
faker==18.9.0
# via -r requirements/_test.in
flask==2.3.2
# via
Expand Down Expand Up @@ -139,7 +139,7 @@ markupsafe==2.1.2
# -c requirements/_base.txt
# jinja2
# werkzeug
moto==4.1.9
moto==4.1.10
# via -r requirements/_test.in
mpmath==1.3.0
# via sympy
Expand All @@ -154,7 +154,7 @@ openapi-schema-validator==0.2.3
# via openapi-spec-validator
openapi-spec-validator==0.4.0
# via moto
packaging==23.0
packaging==23.1
# via
# -c requirements/_base.txt
# docker
Expand All @@ -176,7 +176,7 @@ pyasn1==0.5.0
# rsa
pycparser==2.21
# via cffi
pydantic==1.10.2
pydantic==1.10.8
# via
# -c requirements/_base.txt
# aws-sam-translator
Expand Down Expand Up @@ -205,7 +205,7 @@ pytest-aiohttp==1.0.4
# via -r requirements/_test.in
pytest-asyncio==0.21.0
# via pytest-aiohttp
pytest-cov==4.0.0
pytest-cov==4.1.0
# via -r requirements/_test.in
pytest-icdiff==0.6
# via -r requirements/_test.in
Expand Down Expand Up @@ -238,6 +238,7 @@ regex==2023.5.5
# via cfn-lint
requests==2.31.0
# via
# -c requirements/_base.txt
# docker
# moto
# responses
Expand Down Expand Up @@ -269,21 +270,21 @@ tomli==2.0.1
# via
# coverage
# pytest
types-pyyaml==6.0.12.9
types-pyyaml==6.0.12.10
# via responses
typing-extensions==4.5.0
typing-extensions==4.6.1
# via
# -c requirements/_base.txt
# aws-sam-translator
# pydantic
urllib3==1.26.14
urllib3==1.26.16
# via
# -c requirements/_base.txt
# botocore
# docker
# requests
# responses
websocket-client==1.5.1
websocket-client==1.5.2
# via docker
werkzeug==2.3.4
# via
Expand All @@ -295,7 +296,7 @@ wrapt==1.15.0
# aws-xray-sdk
xmltodict==0.13.0
# via moto
yarl==1.8.2
yarl==1.9.2
# via
# -c requirements/_base.txt
# aiohttp
Expand Down
Loading