Vendors dill #23870

ryanthompson591 · 2022-10-27T16:11:56Z

Pulls a copy of the dill library into the beam sdk. This version is the same as the current version (0.3.1.1.).

The main reason to do this is to keep the version of dill on the worker and runner the same.

More detailed information in design doc.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2022-10-27T17:00:13Z

Codecov Report

Merging #23870 (4046a62) into master (90fffc0) will decrease coverage by 1.26%.
The diff coverage is 27.08%.

@@            Coverage Diff             @@
##           master   #23870      +/-   ##
==========================================
- Coverage   73.46%   72.21%   -1.26%     
==========================================
  Files         714      726      +12     
  Lines       96497    99155    +2658     
==========================================
+ Hits        70889    71600     +711     
- Misses      24286    26233    +1947     
  Partials     1322     1322

Flag	Coverage Δ
python	`81.01% <27.08%> (-2.16%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/python/apache_beam/coders/coders.py	`87.48% <0.00%> (ø)`
sdks/python/apache_beam/vendor/dill/__diff.py	`0.00% <0.00%> (ø)`
sdks/python/apache_beam/vendor/dill/_objects.py	`0.00% <0.00%> (ø)`
sdks/python/apache_beam/vendor/dill/source.py	`7.21% <7.21%> (ø)`
sdks/python/apache_beam/vendor/dill/detect.py	`10.50% <10.50%> (ø)`
sdks/python/apache_beam/vendor/dill/temp.py	`15.84% <15.84%> (ø)`
sdks/python/apache_beam/vendor/dill/pointers.py	`16.66% <16.66%> (ø)`
sdks/python/apache_beam/vendor/dill/__init__.py	`48.14% <48.14%> (ø)`
sdks/python/apache_beam/vendor/dill/_dill.py	`53.70% <53.70%> (ø)`
sdks/python/apache_beam/vendor/dill/settings.py	`71.42% <71.42%> (ø)`
... and 9 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2022-10-27T23:39:06Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.6.0 to 1.6.1. - [Release notes](https://github.com/spf13/cobra/releases) - [Commits](spf13/cobra@v1.6.0...v1.6.1) --- updated-dependencies: - dependency-name: github.com/spf13/cobra dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…e#23823) Bumps [cloud.google.com/go/pubsub](https://github.com/googleapis/google-cloud-go) from 1.25.1 to 1.26.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@pubsub/v1.25.1...pubsub/v1.26.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/pubsub dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…apache#23858) Co-authored-by: Ryan Thompson <ryanthompson@google.com>

* Include pkg.go.dev in release vote email * Use RELEASE_VERSION consistently.

* Add Go usage instructions to download page. * whitepsace

* Use Akvelon editor (apache#23415) * Upgrade Akvelon editor (apache#23415)

* tmp * fix mock * datastore_p1 * mock_repo * auth_store * nit * workflow * vars * nit * README

…hConverter` (apache#23455) * Add arrow_type_compatibility A library for relating Beam schemas to/from Arrow schemas, and Beam Rows to/from Arrow RecordBatches (as a BatchConverter). * Add microbenchmark for PyarrowBatchConverter.produce_batch * Test schema options at more levels * fixup! Add arrow_type_compatibility * fixup! Add microbenchmark for PyarrowBatchConverter.produce_batch * Address review comments

… using Python external transforms

…le (apache#23548) * Migrate BINARY, VARBINARY, CHAR, VARCHAR jdbc logical types to portable * Move jdbc logical type to portable logical types in Java * Create portable logical types in Python * Support value_from_runner_api and value_to_runner_api in Python SchemaTransform (currently only support atomic type values) Fix nullable/test/leftovers * Fix typos * Add standard coder test * Fix RowCoderImpl cannot encode bytes column in cython compiled * Set coder_impl.is_compiled=True when running on compiled stream module * Add docstring, add todo and warnings for unsupported

…e Streams connector to test transaction tags filtering in the Change Stream records (apache#23284) * Added Change Stream filter by transaction tag IT test * Improved filter check using both contains and not contains clauses * Ran ./gradlew spotlessApply Co-authored-by: Andrew Galad <agalad@google.com>

* Use --release 8 for builds targeting Java 8 This ensures cross compilation works correctly when building on JDK 11 and targeting Java 8. * Update buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy Co-authored-by: Lukasz Cwik <lcwik@google.com> * Review feedback * Update buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy Co-authored-by: Lukasz Cwik <lcwik@google.com> * Adjust javaVersion configuration Co-authored-by: Lukasz Cwik <lcwik@google.com>

ryanthompson591 · 2022-11-11T16:50:17Z

Run Python 3.7 PostCommit

ryanthompson591 · 2022-11-11T16:50:28Z

Run Python 3.8 PostCommit

ryanthompson591 · 2022-11-11T17:26:17Z

R: @tvalentyn

Can you give this a first pass. There is still more testing to do, but here is the first version.

github-actions · 2022-11-11T17:27:22Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

tvalentyn

Mostly looks good. small nits. a few questions;

Can we add a readme for what it takes to upgrade to newer version?
Are postcommit failures related?
Did you run TGP w/ this change? (Can discuss offline).

sdks/python/apache_beam/vendor/__init__.py

sdks/python/scripts/run_pylint.sh

sdks/python/setup.py

build.gradle.kts

ryanthompson591 · 2022-11-18T14:55:04Z

Run Python 3.8 PostCommit

eddie-scio · 2022-12-09T19:13:52Z

Hello friends, just checking in if there are any blockers here!

tvalentyn · 2022-12-09T19:26:56Z

Ryan is currently OOO for a bit and therefore not actively working on it.

tvalentyn · 2022-12-09T19:27:46Z

I'll see if I can pick it up in between of other efforts to push forward, but no promises at this time. Sorry.

eddie-scio · 2022-12-09T19:29:21Z

Ah, thanks for the update! That's already enormously helpful so we can adjust our blocked efforts accordingly. Appreciate the transparency.

ryanthompson591 · 2022-12-09T19:57:22Z

If valentine can't pick this up, I will get to this first week of January.

…

On Fri, Dec 9, 2022, 2:29 PM Eddie Zhou ***@***.***> wrote: Ah, thanks for the update! That's already enormously helpful so we can adjust our blocked efforts accordingly. Appreciate the transparency. — Reply to this email directly, view it on GitHub <#23870 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABS23CXITMVTWTCKMSZ4VTDWMOCBXANCNFSM6AAAAAARQHIAJU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

eddie-scio · 2023-01-11T04:50:14Z

Hey folks, happy new year! Just wanted to check in on this effort and if we can push through.

eddie-scio · 2023-01-23T23:31:08Z

Hey @ryanthompson591 @tvalentyn don't want to be that guy, but just checking in on any ETA here, or if there's any way we can help unblock

tvalentyn · 2023-01-24T01:57:44Z

Hi, thanks for the ping. Unfortunately this effort currently doesn't have an owner and I can't provide an ETA at this time. Can you remind me of the nature of the blocking issue that you have? Perhaps I could suggest a workaround.

eddie-scio · 2023-01-24T19:06:47Z

Really appreciate the transparency! Also x-linking the issue: #22893

Here's our dependency chain:

We want to upgrade to Spacy V3 to leverage their new v3 models.
We also use allennlp==2.9.3, which has spacy<3.3,>=2.1.0.
There's a bug leading to large latency issues in Spacy V3. The fix was only released in spacy==3.3.0.
allennlp==2.10.0 has spacy>=2.1.0,<3.4 which gets around (4), but it also has dill>=0.3.4, which leads us here, since Beam still has dill<0.3.2.

Since allennlp is now in maintenance mode, we're unlikely to see any changes there, so the only other avenue I see is to remove our allennlp dependency fully, which is quite high lift on our end. And since we saw this dill version cap from Beam was pretty old, we figured you guys were going to work on this anyways soon.

tvalentyn · 2023-03-21T23:13:41Z

we plan to lift the bound before next release, i am taking a look.

github-actions · 2023-06-07T12:43:09Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

github-actions · 2023-06-15T12:41:58Z

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

janheinrichmerker · 2024-06-27T15:15:16Z

Any chance that the conflict between the dill versions of apache-beam and datasets (see #24458 and huggingface/datasets#5613) will be resolved?

tvalentyn · 2024-06-27T16:53:39Z

Eventually yes but it is not straightforward. #22893 has the discussion and suggestions (you can force-install a newer version of dill in submission and runtime environment in the meantime).

ryanthompson592 added 2 commits October 27, 2022 12:05

import dill as a vendored version

0707498

added vendored dill library

5ee1025

changed imports

87d738b

github-actions bot added the python label Oct 27, 2022

aromanenko-dev and others added 23 commits October 28, 2022 09:29

[23832] Remove ParquetIO.withSplit (closes apache#23832) (apache#23833)

089fa2a

[Go SDK] Add tests to the metrics package (apache#23769)

36f5417

Updated documentation to point to notebooks instead of having samples (…

6f2e9ad

…apache#23858) Co-authored-by: Ryan Thompson <ryanthompson@google.com>

Post 2.42.0 Updates to release guide (apache#23672)

1808e1a

* Include pkg.go.dev in release vote email * Use RELEASE_VERSION consistently.

Add Go usage instructions to download page. (apache#23698)

b1198a3

* Add Go usage instructions to download page. * whitepsace

Deactivate Dask Runner code coverage tests as workaround (apache#23841)

4cf9fbb

Use Akvelon editor (apache#23415) (apache#23825)

37d7687

* Use Akvelon editor (apache#23415) * Upgrade Akvelon editor (apache#23415)

[Tour Of Beam] User authorization part 1 (apache#23782)

6772f2d

* tmp * fix mock * datastore_p1 * mock_repo * auth_store * nit * workflow * vars * nit * README

[BEAM-23815] Fix Neo4j tests. (apache#23862)

6cf26b6

[BEAM-23836] Updating documentation for cross-language Java pipelines…

ea73f7a

… using Python external transforms

update

36d4bc9

update

c93eb06

update

b406192

Reduce log spam of Py37PostCommit (apache#23829)

ee5f58c

Actually use the DatsetService that will be auto-closed (apache#23873)

e69bd6e

[23832] Update CHANGES.md

dfb5ce9

added init file

6688dc5

github-actions bot added the build label Oct 28, 2022

github-actions bot removed the build label Nov 11, 2022

tvalentyn reviewed Nov 15, 2022

View reviewed changes

sdks/python/apache_beam/vendor/__init__.py Outdated Show resolved Hide resolved

sdks/python/scripts/run_pylint.sh Outdated Show resolved Hide resolved

sdks/python/setup.py Outdated Show resolved Hide resolved

build.gradle.kts Outdated Show resolved Hide resolved

added changes based on feedback

39e9162

github-actions bot added build docker labels Nov 17, 2022

updated ordering

82aaa11

ryanthompson591 added 2 commits November 18, 2022 10:06

Merge branch 'master' into vendored_dill

b3a2f43

Merge branch 'apache:master' into vendored_dill

4046a62

tvalentyn mentioned this pull request Dec 1, 2022

[Bug]: Python 3.10 installing apache-beam==2.43.0 with multiprocess>=0.70.12 has incompatibility for dill #24458

Open

github-actions bot added the stale label Jun 7, 2023

github-actions bot closed this Jun 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vendors dill #23870

Vendors dill #23870

ryanthompson591 commented Oct 27, 2022 •

edited

Loading

codecov bot commented Oct 27, 2022 •

edited

Loading

github-actions bot commented Oct 27, 2022

ryanthompson591 commented Nov 11, 2022

ryanthompson591 commented Nov 11, 2022

ryanthompson591 commented Nov 11, 2022

github-actions bot commented Nov 11, 2022

tvalentyn left a comment

ryanthompson591 commented Nov 18, 2022

eddie-scio commented Dec 9, 2022

tvalentyn commented Dec 9, 2022 •

edited

Loading

tvalentyn commented Dec 9, 2022 •

edited

Loading

eddie-scio commented Dec 9, 2022

ryanthompson591 commented Dec 9, 2022 via email

eddie-scio commented Jan 11, 2023

eddie-scio commented Jan 23, 2023

tvalentyn commented Jan 24, 2023

eddie-scio commented Jan 24, 2023

tvalentyn commented Mar 21, 2023

github-actions bot commented Jun 7, 2023

github-actions bot commented Jun 15, 2023

janheinrichmerker commented Jun 27, 2024

tvalentyn commented Jun 27, 2024

Vendors dill #23870

Vendors dill #23870

Conversation

ryanthompson591 commented Oct 27, 2022 • edited Loading

More detailed information in design doc.

GitHub Actions Tests Status (on master branch)

codecov bot commented Oct 27, 2022 • edited Loading

Codecov Report

github-actions bot commented Oct 27, 2022

ryanthompson591 commented Nov 11, 2022

ryanthompson591 commented Nov 11, 2022

ryanthompson591 commented Nov 11, 2022

github-actions bot commented Nov 11, 2022

tvalentyn left a comment

Choose a reason for hiding this comment

ryanthompson591 commented Nov 18, 2022

eddie-scio commented Dec 9, 2022

tvalentyn commented Dec 9, 2022 • edited Loading

tvalentyn commented Dec 9, 2022 • edited Loading

eddie-scio commented Dec 9, 2022

ryanthompson591 commented Dec 9, 2022 via email

eddie-scio commented Jan 11, 2023

eddie-scio commented Jan 23, 2023

tvalentyn commented Jan 24, 2023

eddie-scio commented Jan 24, 2023

tvalentyn commented Mar 21, 2023

github-actions bot commented Jun 7, 2023

github-actions bot commented Jun 15, 2023

janheinrichmerker commented Jun 27, 2024

tvalentyn commented Jun 27, 2024

ryanthompson591 commented Oct 27, 2022 •

edited

Loading

codecov bot commented Oct 27, 2022 •

edited

Loading

tvalentyn commented Dec 9, 2022 •

edited

Loading

tvalentyn commented Dec 9, 2022 •

edited

Loading