Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow extraPipPackages to only install once (using PVC) #104

Open
thesuperzapper opened this issue Mar 27, 2021 · 0 comments
Open

allow extraPipPackages to only install once (using PVC) #104

thesuperzapper opened this issue Mar 27, 2021 · 0 comments
Labels
kind/enhancement kind - new features or changes status/help-wanted status - this needs contributions

Comments

@thesuperzapper
Copy link
Member

Currently we use an init-container for all the *. extraPipPackages values, which must run every time the Pod starts.

This issue is worst for the airflow.kubernetesPodTemplate.extraPipPackages value, (which is related to the pod_template.yaml generation), as it means that users are installing those packages for literally EVERY TASK in thier DAGs.

I believe it would be possible to optionally allow the usage of a PVC rather than ephemeral volume, which would mean that pip-install only runs one-time per helm upgrade/install through the use of pre-install helm-hook Jobs.
(NOTE: it needs to be PRE-install because the user might install something like python-ldap for their webserver to be able to start)

@thesuperzapper thesuperzapper added the kind/enhancement kind - new features or changes label Mar 27, 2021
@thesuperzapper thesuperzapper pinned this issue Mar 27, 2021
@thesuperzapper thesuperzapper added the status/help-wanted status - this needs contributions label Mar 27, 2021
@stale stale bot added lifecycle/stale lifecycle - this is stale and removed lifecycle/stale lifecycle - this is stale labels May 27, 2021
@thesuperzapper thesuperzapper added the lifecycle/frozen lifecycle - this can't become stale label May 28, 2021
@thesuperzapper thesuperzapper unpinned this issue Jul 2, 2021
@thesuperzapper thesuperzapper changed the title make extraPipPackages only install once allow extraPipPackages to only install once (using PVC) Aug 19, 2021
@thesuperzapper thesuperzapper removed the lifecycle/frozen lifecycle - this can't become stale label Mar 22, 2022
@airflow-helm airflow-helm deleted a comment from stale bot Mar 22, 2022
@thesuperzapper thesuperzapper added this to the airflow-8.8.0 milestone Mar 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement kind - new features or changes status/help-wanted status - this needs contributions
Projects
None yet
Development

No branches or pull requests

1 participant