Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modernize type hints #31755

Merged
merged 29 commits into from
Jul 10, 2024
Merged

Modernize type hints #31755

merged 29 commits into from
Jul 10, 2024

Conversation

robertwb
Copy link
Contributor

@robertwb robertwb commented Jul 2, 2024

Convert from Python2 style comment type hints to true Python 3 annotations.

This was done with com2ann.

This PR only covers those files that have not been edited recently, and is split by top-level directory to reduce the chance and cost of conflicts.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@robertwb
Copy link
Contributor Author

robertwb commented Jul 2, 2024

R: @tvalentyn

Copy link
Contributor

github-actions bot commented Jul 2, 2024

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

@robertwb robertwb force-pushed the type-hints branch 3 times, most recently from 580b599 to a827861 Compare July 3, 2024 00:30
Copy link

codecov bot commented Jul 4, 2024

Codecov Report

Attention: Patch coverage is 99.37965% with 5 lines in your changes missing coverage. Please review.

Project coverage is 81.21%. Comparing base (5a09095) to head (9764359).
Report is 27 commits behind head on master.

Files Patch % Lines
...dks/python/apache_beam/transforms/external_java.py 0.00% 3 Missing ⚠️
...eam/runners/portability/fn_api_runner/fn_runner.py 97.82% 1 Missing ⚠️
sdks/python/apache_beam/transforms/userstate.py 97.43% 1 Missing ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master   #31755       +/-   ##
=============================================
+ Coverage     71.18%   81.21%   +10.02%     
=============================================
  Files          1058      397      -661     
  Lines        134254    67717    -66537     
  Branches       3257        0     -3257     
=============================================
- Hits          95571    54997    -40574     
+ Misses        35536    12720    -22816     
+ Partials       3147        0     -3147     
Flag Coverage Δ
go ?
java ?
python 81.21% <99.37%> (-0.09%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tvalentyn
Copy link
Contributor

Run Python 3.8 PostCommit

@tvalentyn
Copy link
Contributor

postcommits will be here: https://github.com/apache/beam/actions/runs/9848030460

there were linter failures as well - have you also tried importing this and running internal tests?

@robertwb
Copy link
Contributor Author

robertwb commented Jul 9, 2024

Good call. Internal tests look good on import.

@robertwb
Copy link
Contributor Author

robertwb commented Jul 9, 2024

Fixed the lint issue.

Copy link
Contributor

@tvalentyn tvalentyn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at manual commits and spot-checked the autogenerated content, but there is a lot of changes and I could have missed something. What is the worst thing that can go wrong here? Inferring incorrect coder because there is a typo in a typehint? Do we need to be more careful and split it up into smaller PRs ?

@@ -309,8 +310,8 @@ def is_alive():
return not (shutdown_requested or evaluation_context.shutdown_requested)

# The shared queue that allows the producer and consumer to communicate.
channel: Queue[Union[test_stream.Event,
_EndOfStream]] = Queue() # noqa: F821
channel: 'Queue[Union[test_stream.Event, _EndOfStream]]' = ( # noqa: F821
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does the entire hint need to be quoted or only '_EndOfStream' ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also test_stream.Event. Given the noqa, it seems there were issues with this before.

@robertwb
Copy link
Contributor Author

robertwb commented Jul 9, 2024

Yes, looking at the manual changes and spot checking the auto generated should be sufficient.

The worst thing is failure to import because of some circular/forward definition. I think I fixed all of those (even if things aren't imported by default, the docbuilder import most modules.) As you mention, some coders are inferred from type hints, but that should fail quickly because of actually being incorrect (and I didn't run into any of this in all the tests).

@robertwb
Copy link
Contributor Author

robertwb commented Jul 9, 2024

Also, if there are issues we should be able to revert individual commits rather than roll the whole thing back.

I'll wait 'till the release is cut before merging this to give it extra time to bake.

@robertwb robertwb merged commit 3696140 into apache:master Jul 10, 2024
90 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants