Turn Pydantic into an optional dependency #37320
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:providers
area:serialization
area:system-tests
kind:documentation
provider:google
potiuk
requested review from
Taragolis,
ephraimbuddy,
mhenc,
eladkal,
vincbeck,
hussein-awala and
jedcunningham
potiuk
force-pushed
the
make-pydantic-optional-dependency
branch
4 times, most recently
from
312f55d
to
ac41af9
Compare
potiuk
changed the title
potiuk
force-pushed
the
make-pydantic-optional-dependency
branch
from
ac41af9
to
676a4ec
Compare
potiuk
commented
Feb 11, 2024
potiuk
force-pushed
the
make-pydantic-optional-dependency
branch
from
676a4ec
to
88c435a
Compare
potiuk
commented
Feb 11, 2024
potiuk
force-pushed
the
make-pydantic-optional-dependency
branch
3 times, most recently
from
0f41e34
to
b9b06a1
Compare
potiuk
commented
Feb 12, 2024
potiuk
commented
Feb 12, 2024
potiuk
commented
Feb 12, 2024
...system/providers/google/cloud/storage_transfer/example_cloud_storage_transfer_service_aws.py
Show resolved
Hide resolved
Taragolis
reviewed
Feb 12, 2024
We've been internally using pydantic for internal API and it caused some compatibility issues, because Pydantic is so popular and currently still users of Pydantic are somewhat split between Pydantic 1 and Pydantic 2. The popularity of Pydantic works against us, and since we are not yet using it in "production" (and in the future we will only actually use it for Internal API), it seems that turning Pydantic into an optional dependency is the best way we can proceed. It's as simple as converting all the direct imports into a common util imports that have a fallback mechanism when import is not found. This should enable less conflicts when installing 3rd-party libraries with Airflow. Added test where pydantic is removed. Also made sure that the special cases we have tests for run full suite of tests - non-db and db.
potiuk
force-pushed
the
make-pydantic-optional-dependency
branch
from
3d6128a
to
16aeb19
Compare
potiuk
added a commit
that referenced
this pull request
Feb 13, 2024
We've been internally using pydantic for internal API and it caused some compatibility issues, because Pydantic is so popular and currently still users of Pydantic are somewhat split between Pydantic 1 and Pydantic 2. The popularity of Pydantic works against us, and since we are not yet using it in "production" (and in the future we will only actually use it for Internal API), it seems that turning Pydantic into an optional dependency is the best way we can proceed. It's as simple as converting all the direct imports into a common util imports that have a fallback mechanism when import is not found. This should enable less conflicts when installing 3rd-party libraries with Airflow. Added test where pydantic is removed. Also made sure that the special cases we have tests for run full suite of tests - non-db and db. (cherry picked from commit c3f48ee)
potiuk
added a commit
to potiuk/airflow
that referenced
this pull request
Feb 13, 2024
The changes in apache#37320 introduced flakiness while removing side-effect of configuration test. Cleanup of ProvidersManager might interfere with other tests using it at the same time. This caused some Non-DB test failures with missing keys in ProvidersManager. The fix is to mark the configuration test as DB-test - then it will never be run by NonDB xdist and it will always run sequentially to other tests (and still it will not introduce side-effect as cleanup will always be done between tests.
potiuk
added a commit
that referenced
this pull request
Feb 13, 2024
The changes in #37320 introduced flakiness while removing side-effect of configuration test. Cleanup of ProvidersManager might interfere with other tests using it at the same time. This caused some Non-DB test failures with missing keys in ProvidersManager. The fix is to mark the configuration test as DB-test - then it will never be run by NonDB xdist and it will always run sequentially to other tests (and still it will not introduce side-effect as cleanup will always be done between tests.
This was referenced Feb 16, 2024
ephraimbuddy
added
the
type:misc/internal
ephraimbuddy
pushed a commit
that referenced
this pull request
Feb 20, 2024
The changes in #37320 introduced flakiness while removing side-effect of configuration test. Cleanup of ProvidersManager might interfere with other tests using it at the same time. This caused some Non-DB test failures with missing keys in ProvidersManager. The fix is to mark the configuration test as DB-test - then it will never be run by NonDB xdist and it will always run sequentially to other tests (and still it will not introduce side-effect as cleanup will always be done between tests. (cherry picked from commit cbc9af0)
ephraimbuddy
pushed a commit
that referenced
this pull request
Feb 22, 2024
We've been internally using pydantic for internal API and it caused some compatibility issues, because Pydantic is so popular and currently still users of Pydantic are somewhat split between Pydantic 1 and Pydantic 2. The popularity of Pydantic works against us, and since we are not yet using it in "production" (and in the future we will only actually use it for Internal API), it seems that turning Pydantic into an optional dependency is the best way we can proceed. It's as simple as converting all the direct imports into a common util imports that have a fallback mechanism when import is not found. This should enable less conflicts when installing 3rd-party libraries with Airflow. Added test where pydantic is removed. Also made sure that the special cases we have tests for run full suite of tests - non-db and db. (cherry picked from commit c3f48ee)
ephraimbuddy
pushed a commit
that referenced
this pull request
Feb 22, 2024
The changes in #37320 introduced flakiness while removing side-effect of configuration test. Cleanup of ProvidersManager might interfere with other tests using it at the same time. This caused some Non-DB test failures with missing keys in ProvidersManager. The fix is to mark the configuration test as DB-test - then it will never be run by NonDB xdist and it will always run sequentially to other tests (and still it will not introduce side-effect as cleanup will always be done between tests. (cherry picked from commit cbc9af0)
68 tasks
79 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We've been internally using pydantic for internal API and it caused some compatibility issues, because Pydantic is so popular and currently still users of Pydantic are somewhat split between Pydantic 1 and Pydantic 2. The popularity of Pydantic works against us, and since we are not yet using it in "production" (and in the future we will only actually use it for Internal API), it seems that turning Pydantic into an optional dependency is the best way we can proceed.
It's as simple as converting all the direct imports into a common util imports that have a fallback mechanism when import is not found.
This should enable less conflicts when installing 3rd-party libraries with Airflow.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.