Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move provider dependencies to inside provider folders #24672

Merged
merged 1 commit into from
Jun 29, 2022

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Jun 27, 2022

The setup.py had ALWAUS contained provider dependencies,
but this is really a remnant of Airlfow 1.10 where providers
were not separated out to subfolders of "providers".

This change moves all the provider-specific dependencies
to provider.yaml where they are kept together with all other
provider meta-data.

Later, when we move providers out, we can move them to
provider specific setup.py files (or let provider-specific
setup.py files read them from provider.yaml) but this is
not something we want to do it now.

The dependencies.json is now renamed to provider_dependencies.json
and moved to "airflow" so tha it can be kept as part of the
sources needed for sdist package to provide extras. Pre-commit still
generates the file as needed and it contains now both:

  • cross-provider-deps information which providers depend on each
    other
  • deps - information what regular dependencies are needed for each
    provider

On top of preparing to splitting providers it has the advantage,
that there will be no more case where adding a dependency change
for provider will not run tests for that provider.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragement file, named {pr_number}.significant.rst, in newsfragments.

@potiuk potiuk force-pushed the move-setup-dependencies-to-providers branch from 55d3533 to 9190251 Compare June 28, 2022 09:34
@potiuk
Copy link
Member Author

potiuk commented Jun 28, 2022

Updated. I have also updated the "selective checks" to trigger "upgrade-to-newer-dependencies when the "provider_dependencies.json" file changes by pre-commit.

setup.py Outdated Show resolved Hide resolved
@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Jun 28, 2022
@github-actions
Copy link

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

setup.py Outdated Show resolved Hide resolved
@potiuk potiuk force-pushed the move-setup-dependencies-to-providers branch 2 times, most recently from cd41b3e to 1406e52 Compare June 28, 2022 10:48
@potiuk
Copy link
Member Author

potiuk commented Jun 28, 2022

Yeah @kaxil - I think I reverted all the auto-formatting. Sorry for that, I missed it and it was not at all intentional :)

@potiuk potiuk force-pushed the move-setup-dependencies-to-providers branch 2 times, most recently from a4dea54 to 6ba9a88 Compare June 28, 2022 11:02
@potiuk
Copy link
Member Author

potiuk commented Jun 28, 2022

I also unified names all over the setup.py and related. I changed requirements into dependencies - they were used interchangeably and sometimes one was used sometimes the other. Now we have dependencies everywhere

@potiuk
Copy link
Member Author

potiuk commented Jun 28, 2022

Ok. Looks green @kaxil - any more comments :D ?

@potiuk potiuk force-pushed the move-setup-dependencies-to-providers branch 7 times, most recently from dfda130 to 625d131 Compare June 29, 2022 08:41
.pre-commit-config.yaml Outdated Show resolved Hide resolved
airflow/provider.yaml.schema.json Outdated Show resolved Hide resolved
@kaxil
Copy link
Member

kaxil commented Jun 29, 2022

cc @phanikumv @pankajastro @pankajkoti @bharanidharan14 @rajaths010494 -- worth checking this PR out -- so we can help maintain some of the providers dependencies and know where to look for :)

@potiuk
Copy link
Member Author

potiuk commented Jun 29, 2022

cc @phanikumv @pankajastro @pankajkoti @bharanidharan14 @rajaths010494 -- worth checking this PR out -- so we can help maintain some of the providers dependencies and know where to look for :)

Oh yeah I am going to make a VERY LOUD announcement in slack and devlist once we merge this one and already pinged about it in a few PRs. Thanks @kaxil for cross-promotion 😸

The ``setup.py`` had ALWAUS contained provider dependencies,
but this is really a remnant of Airlfow 1.10 where providers
were not separated out to subfolders of "providers".

This change moves all the provider-specific dependencies
to provider.yaml where they are kept together with all other
provider meta-data.

Later, when we move providers out, we can move them to
provider specific setup.py files (or let provider-specific
setup.py files read them from provider.yaml) but this is
not something we want to do it now.

The dependencies.json is now renamed to provider_dependencies.json
and moved to "airflow" so tha it can be kept as part of the
sources needed for sdist package to provide extras. Pre-commit still
generates the file as needed and it contains now both:

* cross-provider-deps information which providers depend on each
  other
* deps - information what regular dependencies are needed for each
  provider

On top of preparing to splitting providers it has the advantage,
that there will be no more case where adding a dependency change
for provider will not run tests for that provider.
@potiuk potiuk force-pushed the move-setup-dependencies-to-providers branch from 625d131 to 6349b5e Compare June 29, 2022 14:27
@potiuk
Copy link
Member Author

potiuk commented Jun 29, 2022

Wait for success and I am merging :). One step closer to full provider separation.

@potiuk potiuk merged commit 0de31bd into main Jun 29, 2022
@kaxil kaxil deleted the move-setup-dependencies-to-providers branch June 29, 2022 18:04
@ephraimbuddy ephraimbuddy added the type:misc/internal Changelog: Misc changes that should appear in change log label Aug 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:dev-tools area:providers full tests needed We need to run full set of tests for this PR to merge provider:cncf-kubernetes Kubernetes provider related issues type:misc/internal Changelog: Misc changes that should appear in change log
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants