Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add back cloud profiler #26700

Merged
merged 3 commits into from
May 15, 2023
Merged

Add back cloud profiler #26700

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8ebfc7c
Add back cloud profiler
tvalentyn May 15, 2023
caa8cba
Update base image requirements.
tvalentyn May 15, 2023
3bc55c8
Add cloud profiler.
tvalentyn May 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,8 @@
* If a main session fails to load, the pipeline will now fail at worker startup. ([#25401](https://github.com/apache/beam/issues/25401)).
* Python pipeline options will now ignore unparsed command line flags prefixed with a single dash. ([#25943](https://github.com/apache/beam/issues/25943)).
* The SmallestPerKey combiner now requires keyword-only arguments for specifying optional parameters, such as `key` and `reverse`. ([#25888](https://github.com/apache/beam/issues/25888)).
* The google-cloud-profiler dependency was accidentally removed from Beam's Python Docker
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(minor) does this better fix "known issues" section?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good call.

Image [#26998](https://github.com/apache/beam/issues/26698). [Dataflow Docker images](https://cloud.google.com/dataflow/docs/concepts/sdk-worker-dependencies) still preinstall this dependency.

## Deprecations

Expand Down
2 changes: 2 additions & 0 deletions sdks/python/container/base_image_requirements_manual.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ cython<1
future
# TODO: Uncomment python version once guppy supports it.
# https://github.com/apache/beam/issues/25984
# TODO(https://github.com/GoogleCloudPlatform/cloud-profiler-python/issues/127): Unpin python version.
google-cloud-profiler;python_version<"3.11"
guppy3;python_version<"3.11" # Memory profiler
mmh3 # Optimizes execution of some Beam codepaths. TODO: Make it Beam's dependency.
nltk # Commonly used for natural language processing.
Expand Down
84 changes: 43 additions & 41 deletions sdks/python/container/py310/base_image_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,132 +23,134 @@

absl-py==1.4.0
astunparse==1.6.3
attrs==22.2.0
beautifulsoup4==4.12.0
attrs==23.1.0
beautifulsoup4==4.12.2
bs4==0.0.1
cachetools==4.2.4
certifi==2022.12.7
cachetools==5.3.0
certifi==2023.5.7
cffi==1.15.1
charset-normalizer==3.1.0
click==8.1.3
cloudpickle==2.2.1
crcmod==1.7
cryptography==40.0.1
cryptography==40.0.2
Cython==0.29.34
deprecation==2.1.0
dill==0.3.1.1
dnspython==2.3.0
docker==6.0.1
docker==6.1.2
docopt==0.6.2
exceptiongroup==1.1.1
execnet==1.9.0
fastavro==1.7.3
fastavro==1.7.4
fasteners==0.18
flatbuffers==23.3.3
flatbuffers==23.5.9
freezegun==1.2.2
future==0.18.3
gast==0.4.0
google-api-core==2.11.0
google-api-python-client==2.86.0
google-apitools==0.5.31
google-auth==2.17.1
google-auth==2.18.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==1.0.0
google-cloud-bigquery==3.9.0
google-cloud-bigquery==3.10.0
google-cloud-bigquery-storage==2.19.1
google-cloud-bigtable==2.17.0
google-cloud-core==2.3.2
google-cloud-datastore==2.15.1
google-cloud-datastore==2.15.2
google-cloud-dlp==3.12.1
google-cloud-language==2.9.1
google-cloud-pubsub==2.15.2
google-cloud-pubsublite==1.8.1
google-cloud-profiler==4.0.0
google-cloud-pubsub==2.17.0
google-cloud-pubsublite==1.8.2
google-cloud-recommendations-ai==0.10.3
google-cloud-spanner==3.30.0
google-cloud-spanner==3.33.0
google-cloud-videointelligence==2.11.1
google-cloud-vision==3.4.1
google-crc32c==1.5.0
google-pasta==0.2.0
google-resumable-media==2.4.1
google-resumable-media==2.5.0
googleapis-common-protos==1.59.0
greenlet==2.0.2
grpc-google-iam-v1==0.12.6
grpcio==1.53.0
grpcio-status==1.53.0
guppy3==3.1.2
grpcio==1.54.2
grpcio-status==1.54.2
guppy3==3.1.3
h5py==3.8.0
hdfs==2.7.0
httplib2==0.21.0
hypothesis==6.70.2
httplib2==0.22.0
hypothesis==6.75.3
idna==3.4
iniconfig==2.0.0
jax==0.4.8
jax==0.4.10
joblib==1.2.0
keras==2.12.0
libclang==16.0.0
Markdown==3.4.3
MarkupSafe==2.1.2
ml-dtypes==0.0.4
ml-dtypes==0.1.0
mmh3==3.1.0
mock==5.0.1
mock==5.0.2
nltk==3.8.1
nose==1.3.7
numpy==1.23.5
oauth2client==4.1.3
oauthlib==3.2.2
objsize==0.6.1
opt-einsum==3.3.0
orjson==3.8.9
orjson==3.8.12
overrides==6.5.0
packaging==23.0
packaging==23.1
pandas==1.5.3
parameterized==0.8.1
parameterized==0.9.0
pluggy==1.0.0
proto-plus==1.22.2
protobuf==4.22.1
protobuf==4.22.5
psycopg2-binary==2.9.6
pyarrow==11.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser==2.21
pydot==1.4.2
PyHamcrest==1.10.1
PyHamcrest==2.0.4
pymongo==4.3.3
PyMySQL==1.0.3
pyparsing==3.0.9
pytest==7.2.2
pytest==7.3.1
pytest-timeout==2.1.0
pytest-xdist==3.2.1
pytest-xdist==3.3.0
python-dateutil==2.8.2
python-snappy==0.6.1
pytz==2023.3
PyYAML==6.0
regex==2023.3.23
requests==2.28.2
regex==2023.5.5
requests==2.30.0
requests-mock==1.10.0
requests-oauthlib==1.3.1
rsa==4.9
scikit-learn==1.2.2
scipy==1.10.1
six==1.16.0
sortedcontainers==2.4.0
soupsieve==2.4
SQLAlchemy==1.4.47
soupsieve==2.4.1
SQLAlchemy==1.4.48
sqlparse==0.4.4
tenacity==8.2.2
tensorboard==2.12.1
tensorboard==2.12.3
tensorboard-data-server==0.7.0
tensorboard-plugin-wit==1.8.1
tensorflow==2.12.0
tensorflow-estimator==2.12.0
tensorflow-io-gcs-filesystem==0.32.0
termcolor==2.2.0
termcolor==2.3.0
testcontainers==3.7.1
threadpoolctl==3.1.0
tomli==2.0.1
tqdm==4.65.0
typing_extensions==4.5.0
uritemplate==4.1.1
urllib3==1.26.15
websocket-client==1.5.1
Werkzeug==2.2.3
Werkzeug==2.3.4
wrapt==1.14.1
zstandard==0.20.0
zstandard==0.21.0
79 changes: 39 additions & 40 deletions sdks/python/container/py311/base_image_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,128 +23,127 @@

absl-py==1.4.0
astunparse==1.6.3
attrs==22.2.0
beautifulsoup4==4.12.0
attrs==23.1.0
beautifulsoup4==4.12.2
bs4==0.0.1
cachetools==4.2.4
certifi==2022.12.7
cachetools==5.3.0
certifi==2023.5.7
cffi==1.15.1
charset-normalizer==3.1.0
click==8.1.3
cloudpickle==2.2.1
crcmod==1.7
cryptography==40.0.1
cryptography==40.0.2
Cython==0.29.34
deprecation==2.1.0
dill==0.3.1.1
dnspython==2.3.0
docker==6.0.1
docker==6.1.2
docopt==0.6.2
execnet==1.9.0
fastavro==1.7.3
fastavro==1.7.4
fasteners==0.18
flatbuffers==23.3.3
flatbuffers==23.5.9
freezegun==1.2.2
future==0.18.3
gast==0.4.0
google-api-core==2.11.0
google-apitools==0.5.31
google-auth==2.17.1
google-auth==2.18.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==1.0.0
google-cloud-bigquery==3.9.0
google-cloud-bigquery==3.10.0
google-cloud-bigquery-storage==2.19.1
google-cloud-bigtable==2.17.0
google-cloud-core==2.3.2
google-cloud-datastore==2.15.1
google-cloud-datastore==2.15.2
google-cloud-dlp==3.12.1
google-cloud-language==2.9.1
google-cloud-pubsub==2.15.2
google-cloud-pubsublite==1.8.1
google-cloud-pubsub==2.17.0
google-cloud-pubsublite==1.8.2
google-cloud-recommendations-ai==0.10.3
google-cloud-spanner==3.30.0
google-cloud-spanner==3.33.0
google-cloud-videointelligence==2.11.1
google-cloud-vision==3.4.1
google-crc32c==1.5.0
google-pasta==0.2.0
google-resumable-media==2.4.1
google-resumable-media==2.5.0
googleapis-common-protos==1.59.0
greenlet==2.0.2
grpc-google-iam-v1==0.12.6
grpcio==1.53.0
grpcio-status==1.53.0
grpcio==1.54.2
grpcio-status==1.54.2
h5py==3.8.0
hdfs==2.7.0
httplib2==0.21.0
hypothesis==6.70.2
httplib2==0.22.0
hypothesis==6.75.3
idna==3.4
iniconfig==2.0.0
jax==0.4.8
jax==0.4.10
joblib==1.2.0
keras==2.12.0
libclang==16.0.0
Markdown==3.4.3
MarkupSafe==2.1.2
ml-dtypes==0.0.4
ml-dtypes==0.1.0
mmh3==3.1.0
mock==5.0.1
mock==5.0.2
nltk==3.8.1
nose==1.3.7
numpy==1.23.5
oauth2client==4.1.3
oauthlib==3.2.2
objsize==0.6.1
opt-einsum==3.3.0
orjson==3.8.9
orjson==3.8.12
overrides==6.5.0
packaging==23.0
packaging==23.1
pandas==1.5.3
parameterized==0.8.1
parameterized==0.9.0
pluggy==1.0.0
proto-plus==1.22.2
protobuf==4.22.1
protobuf==4.22.5
psycopg2-binary==2.9.6
pyarrow==11.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycparser==2.21
pydot==1.4.2
PyHamcrest==1.10.1
PyHamcrest==2.0.4
pymongo==4.3.3
PyMySQL==1.0.3
pyparsing==3.0.9
pytest==7.2.2
pytest==7.3.1
pytest-timeout==2.1.0
pytest-xdist==3.2.1
pytest-xdist==3.3.0
python-dateutil==2.8.2
pytz==2023.3
PyYAML==6.0
regex==2023.3.23
requests==2.28.2
regex==2023.5.5
requests==2.30.0
requests-mock==1.10.0
requests-oauthlib==1.3.1
rsa==4.9
scikit-learn==1.2.2
scipy==1.10.1
six==1.16.0
sortedcontainers==2.4.0
soupsieve==2.4
SQLAlchemy==1.4.47
soupsieve==2.4.1
SQLAlchemy==1.4.48
sqlparse==0.4.4
tenacity==8.2.2
tensorboard==2.12.1
tensorboard==2.12.3
tensorboard-data-server==0.7.0
tensorboard-plugin-wit==1.8.1
tensorflow==2.12.0
tensorflow-estimator==2.12.0
tensorflow-io-gcs-filesystem==0.32.0
termcolor==2.2.0
termcolor==2.3.0
testcontainers==3.7.1
threadpoolctl==3.1.0
tqdm==4.65.0
typing_extensions==4.5.0
urllib3==1.26.15
websocket-client==1.5.1
Werkzeug==2.2.3
Werkzeug==2.3.4
wrapt==1.14.1
zstandard==0.20.0
zstandard==0.21.0
Loading