Add missing Python 3.11 dependencies. #29609

tvalentyn · 2023-12-05T00:59:34Z

Add two dependencies that were previously excluded for Python 3.11.
Modify the scripts to install python-snappy linux headers so that dependencies can be regenerated.

fixes: #25985

github-actions · 2023-12-05T01:55:40Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @damccorm for label python.
R: @Abacn for label build.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

AnandInguva · 2023-12-05T02:51:22Z

sdks/python/container/run_generate_requirements.sh

@@ -30,6 +30,16 @@
 # You will need Python interpreters for all versions supported by Beam, see:
 # https://s.apache.org/beam-python-dev-wiki

+if ! [[ "$(expr substr $(uname -s) 1 5)" == "Linux" ]]; then
+  echo This script needs to be executed in a Linux environment.


optional: update wiki page since devs mostly update it on their own systems (which is Mac).

I believe the script already fails on Mac due to google-cloud-profiler or some other dependency not installable on Mac, that is why I added this. Can current version of the script (before this PR) run on Mac?

I believe Wiki already recommends a linux machine.

I believe Wiki already recommends a linux machine.

looks like not. I definitely remember script failing on MacOS before.

I've run this successfully from my (non-M1) mac before

thanks, added instructions for MacOS. @AnandInguva please help verify that is sufficient. You can check out the pr locally by smth like:

git fetch origin pull/29609/head:pr-29609 git checkout pr-29609

Thanks!

Do you need anything else from me here? asking in case if I missed something. As of now, this is not running on M1 Mac.

You could try to see what it takes to make it succeed on a Mac but don't spend too much time if you can't get it to work.

Sounds good. I will take a look.

i tried few things from stackoverflow and python-snappy github page but no luck so far.

AnandInguva · 2023-12-05T02:53:38Z

.github/workflows/python_dependency_tests.yml

@@ -38,6 +38,8 @@ jobs:
        uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.params.py_ver }}
+      - name: Install headers required by some python packages


we might need to install this in the tox tests? for tests that might be using snappy? I don't think we install it tox anyway

yes, it is a gap in beam test coverage. this codepath might be covered by google-internal tests that use TFRecordIo.

codecov · 2023-12-05T19:04:26Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (c9c89fe) 37.88% compared to head (e6627d2) 37.88%.
Report is 150 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #29609   +/-   ##
=======================================
  Coverage   37.88%   37.88%           
=======================================
  Files         690      690           
  Lines      101253   101263   +10     
=======================================
+ Hits        38358    38364    +6     
- Misses      61300    61303    +3     
- Partials     1595     1596    +1

Flag	Coverage Δ
go	`53.39% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

AnandInguva · 2023-12-05T20:17:56Z

sdks/python/container/run_generate_requirements.sh

+elif [[ "$(expr substr $(uname -s) 1 6)" == "Darwin" ]]; then
+  if ! brew list snappy >/dev/null ; then
+    echo You must install snappy to run this script. Run:
+    echo brew install snappy


Generating requirements from MacOS M1/M2 will remove snappy from the base_image_requirements.txt right?

It should not. MacOS users would have to install snappy package and rerun the script.

you could try running the script on macos.

Yes, let me try running it on M1 Mac.

thanks, please let me know if you can get it to work. I briefly asked another coworker, seems like it's not enough - maybe restarting shell is required?

We also tried setting CPPFLAGS="-I/usr/local/include -L/usr/local/lib" pip install python-snappy

I ran > Task :sdks:python:container:py311:generatePythonRequirements FAILED
It failed for me.

Building wheel for python-snappy (setup.py): started Building wheel for python-snappy (setup.py): finished with status 'error' error: subprocess-exited-with-error × python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [25 lines of output] running bdist_wheel running build running build_py creating build Running setup.py clean for python-snappy creating build/lib.macosx-14.1-arm64-cpython-311 creating build/lib.macosx-14.1-arm64-cpython-311/snappy copying src/snappy/snappy.py -> build/lib.macosx-14.1-arm64-cpython-311/snappy copying src/snappy/snappy_cffi.py -> build/lib.macosx-14.1-arm64-cpython-311/snappy copying src/snappy/__init__.py -> build/lib.macosx-14.1-arm64-cpython-311/snappy copying src/snappy/hadoop_snappy.py -> build/lib.macosx-14.1-arm64-cpython-311/snappy copying src/snappy/snappy_formats.py -> build/lib.macosx-14.1-arm64-cpython-311/snappy copying src/snappy/__main__.py -> build/lib.macosx-14.1-arm64-cpython-311/snappy copying src/snappy/snappy_cffi_builder.py -> build/lib.macosx-14.1-arm64-cpython-311/snappy running build_ext building 'snappy._snappy' extension creating build/temp.macosx-14.1-arm64-cpython-311 creating build/temp.macosx-14.1-arm64-cpython-311/src creating build/temp.macosx-14.1-arm64-cpython-311/src/snappy clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Users/anandinguva/Desktop/projects/beam/build/python311_requirements_gen/include -I/Users/anandinguva/.pyenv/versions/3.11.5/include/python3.11 -c src/snappy/crc32c.c -o build/temp.macosx-14.1-arm64-cpython-311/src/snappy/crc32c.o clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Users/anandinguva/Desktop/projects/beam/build/python311_requirements_gen/include -I/Users/anandinguva/.pyenv/versions/3.11.5/include/python3.11 -c src/snappy/snappymodule.cc -o build/temp.macosx-14.1-arm64-cpython-311/src/snappy/snappymodule.o src/snappy/snappymodule.cc:33:10: fatal error: 'snappy-c.h' file not found #include <snappy-c.h> ^~~~~~~~~~~~ 1 error generated. error: command '/usr/bin/clang' failed with exit code 1 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for python-snappy Successfully built bs4 future google-cloud-profiler Failed to build python-snappy ERROR: Could not build wheels for python-snappy, which is required to install pyproject.toml-based projects

tvalentyn · 2023-12-07T01:44:45Z

let's decouple snappy upgrade from googlecloudprofiler, sending: #29651

github-actions · 2023-12-20T12:13:26Z

Reminder, please take a look at this pr: @damccorm @Abacn

Add missing Python 3.11 dependencies.

8178190

github-actions bot added python build docker labels Dec 5, 2023

Also install headers in the python dependency test suite.

6891007

tvalentyn requested a review from AnandInguva December 5, 2023 01:04

github-actions bot added the Next Action: Reviewers label Dec 5, 2023

AnandInguva reviewed Dec 5, 2023

View reviewed changes

AnandInguva approved these changes Dec 5, 2023

View reviewed changes

Add snappy instructions for MacOS

e6627d2

AnandInguva reviewed Dec 5, 2023

View reviewed changes

tvalentyn marked this pull request as draft December 7, 2023 18:54

github-actions bot added the slow-review label Dec 20, 2023

damccorm added Next Action: Author and removed Next Action: Reviewers slow-review labels Dec 20, 2023

tvalentyn closed this Dec 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add missing Python 3.11 dependencies. #29609

Add missing Python 3.11 dependencies. #29609

tvalentyn commented Dec 5, 2023 •

edited

Loading

github-actions bot commented Dec 5, 2023

AnandInguva Dec 5, 2023

tvalentyn Dec 5, 2023 •

edited

Loading

tvalentyn Dec 5, 2023

tvalentyn Dec 5, 2023

damccorm Dec 5, 2023

tvalentyn Dec 5, 2023

AnandInguva Dec 11, 2023

tvalentyn Dec 11, 2023

AnandInguva Dec 11, 2023

AnandInguva Dec 12, 2023

AnandInguva Dec 5, 2023 •

edited

Loading

tvalentyn Dec 5, 2023

codecov bot commented Dec 5, 2023 •

edited

Loading

AnandInguva Dec 5, 2023

tvalentyn Dec 5, 2023

tvalentyn Dec 5, 2023

AnandInguva Dec 6, 2023

tvalentyn Dec 7, 2023

AnandInguva Dec 7, 2023 •

edited

Loading

tvalentyn commented Dec 7, 2023

github-actions bot commented Dec 20, 2023

Add missing Python 3.11 dependencies. #29609

Add missing Python 3.11 dependencies. #29609

Conversation

tvalentyn commented Dec 5, 2023 • edited Loading

github-actions bot commented Dec 5, 2023

Choose a reason for hiding this comment

tvalentyn Dec 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnandInguva Dec 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 5, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnandInguva Dec 7, 2023 • edited Loading

Choose a reason for hiding this comment

tvalentyn commented Dec 7, 2023

github-actions bot commented Dec 20, 2023

tvalentyn commented Dec 5, 2023 •

edited

Loading

tvalentyn Dec 5, 2023 •

edited

Loading

AnandInguva Dec 5, 2023 •

edited

Loading

codecov bot commented Dec 5, 2023 •

edited

Loading

AnandInguva Dec 7, 2023 •

edited

Loading