diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 4df1d906e8b..75d948f0bf9 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -671,10 +671,22 @@ Airflow dependencies .. note:: On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver - does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice - of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4 - ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option - ``--use-deprecated legacy-resolver`` to your pip install command. + might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might + depend on your choice of extras. In order to install Airflow you might need to either downgrade + pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, + you need to add option ``--use-deprecated legacy-resolver`` to your pip install command. + + While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we + set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well. + Due to those constraints, only ``pip`` installation is currently officially supported. + + While they are some successes with using other tools like `poetry `_ or + `pip-tools `_, they do not share the same workflow as + ``pip`` - especially when it comes to constraint vs. requirements management. + Installing via ``Poetry`` or ``pip-tools`` is not currently supported. + + If you wish to install airflow using those tools you should use the constraint files and convert + them to appropriate format and workflow that your tool requires. Extras @@ -904,10 +916,22 @@ Pinned constraint files .. note:: On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver - does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice - of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4 - ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option - ``--use-deprecated legacy-resolver`` to your pip install command. + might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might + depend on your choice of extras. In order to install Airflow you might need to either downgrade + pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, + you need to add option ``--use-deprecated legacy-resolver`` to your pip install command. + + While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we + set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well. + Due to those constraints, only ``pip`` installation is currently officially supported. + + While they are some successes with using other tools like `poetry `_ or + `pip-tools `_, they do not share the same workflow as + ``pip`` - especially when it comes to constraint vs. requirements management. + Installing via ``Poetry`` or ``pip-tools`` is not currently supported. + + If you wish to install airflow using those tools you should use the constraint files and convert + them to appropriate format and workflow that your tool requires. By default when you install ``apache-airflow`` package - the dependencies are as open as possible while diff --git a/CONTRIBUTORS_QUICK_START.rst b/CONTRIBUTORS_QUICK_START.rst index 28f62565396..3d6ca501384 100644 --- a/CONTRIBUTORS_QUICK_START.rst +++ b/CONTRIBUTORS_QUICK_START.rst @@ -31,11 +31,11 @@ Prerequisites Installing Prerequisites on Ubuntu -################################### +################################## Docker Community Edition ------------------------------------ +------------------------ 1. Installing required packages for Docker and setting up docker repo @@ -84,7 +84,7 @@ Note : After adding user to docker group Logout and Login again for group member Docker Compose --------------------------- +-------------- 1. Installing latest version of Docker Compose @@ -108,8 +108,8 @@ Docker Compose -pyenv and setting up virtual-env --------------------------------------------- +Pyenv and setting up virtual-env +-------------------------------- 1. Checking required packages @@ -162,7 +162,28 @@ pyenv and setting up virtual-env Setup Airflow with Breeze and PyCharm -##################################################### +##################################### + + +.. note:: + + On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver + might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might + depend on your choice of extras. In order to install Airflow you might need to either downgrade + pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, + you need to add option ``--use-deprecated legacy-resolver`` to your pip install command. + + While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we + set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well. + Due to those constraints, only ``pip`` installation is currently officially supported. + + While they are some successes with using other tools like `poetry `_ or + `pip-tools `_, they do not share the same workflow as + ``pip`` - especially when it comes to constraint vs. requirements management. + Installing via ``Poetry`` or ``pip-tools`` is not currently supported. + + If you wish to install airflow using those tools you should use the constraint files and convert + them to appropriate format and workflow that your tool requires. Forking and cloning Project @@ -274,7 +295,7 @@ Setting up Breeze source ~/.bashrc Using Breeze ------------------------------------------ +------------ 1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run( In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``) @@ -457,7 +478,7 @@ Following are some of important topics of Breeze documentation: Setting up Debug ------------------------------------------ +---------------- 1. Configuring Airflow database connection @@ -528,11 +549,11 @@ Setting up Debug Starting development -################################### +#################### Creating a branch ------------------------------------ +----------------- 1. Click on branch symbol in the bottom right corner of Pycharm @@ -555,7 +576,7 @@ Creating a branch Testing ------------------------------------ +------- All Tests are inside ./tests directory. @@ -719,7 +740,7 @@ All Tests are inside ./tests directory. Pre-commit ------------------------------------ +---------- Before committing changes to github or raising a pull request, code needs to be checked for certain quality standards such as spell check, code syntax, code formatting, compatibility with Apache License requirements etc. This set of @@ -880,7 +901,7 @@ To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be r Contribution guide ------------------------------------ +------------------ - To know how to contribute to the project visit |CONTRIBUTING.rst| @@ -916,7 +937,7 @@ Contribution guide Raising Pull Request ------------------------------------ +-------------------- 1. Go to your Github account and open your fork project and click on Branches diff --git a/IMAGES.rst b/IMAGES.rst index df5a919758b..707d1fc25f3 100644 --- a/IMAGES.rst +++ b/IMAGES.rst @@ -123,10 +123,22 @@ parameter to Breeze: .. note:: On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver - does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice - of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4 - ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option - ``--use-deprecated legacy-resolver`` to your pip install command. + might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might + depend on your choice of extras. In order to install Airflow you might need to either downgrade + pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, + you need to add option ``--use-deprecated legacy-resolver`` to your pip install command. + + While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we + set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well. + Due to those constraints, only ``pip`` installation is currently officially supported. + + While they are some successes with using other tools like `poetry `_ or + `pip-tools `_, they do not share the same workflow as + ``pip`` - especially when it comes to constraint vs. requirements management. + Installing via ``Poetry`` or ``pip-tools`` is not currently supported. + + If you wish to install airflow using those tools you should use the constraint files and convert + them to appropriate format and workflow that your tool requires. This will build the image using command similar to: diff --git a/LOCAL_VIRTUALENV.rst b/LOCAL_VIRTUALENV.rst index 3303cfbecb3..1e96080f825 100644 --- a/LOCAL_VIRTUALENV.rst +++ b/LOCAL_VIRTUALENV.rst @@ -64,10 +64,22 @@ Extra Packages .. note:: On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver - does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice - of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4 - ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option - ``--use-deprecated legacy-resolver`` to your pip install command. + might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might + depend on your choice of extras. In order to install Airflow you might need to either downgrade + pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, + you need to add option ``--use-deprecated legacy-resolver`` to your pip install command. + + While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we + set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well. + Due to those constraints, only ``pip`` installation is currently officially supported. + + While they are some successes with using other tools like `poetry `_ or + `pip-tools `_, they do not share the same workflow as + ``pip`` - especially when it comes to constraint vs. requirements management. + Installing via ``Poetry`` or ``pip-tools`` is not currently supported. + + If you wish to install airflow using those tools you should use the constraint files and convert + them to appropriate format and workflow that your tool requires. You can also install extra packages (like ``[ssh]``, etc) via @@ -126,10 +138,22 @@ To create and initialize the local virtualenv: .. note:: On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver - does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice - of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4 - ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option - ``--use-deprecated legacy-resolver`` to your pip install command. + might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might + depend on your choice of extras. In order to install Airflow you might need to either downgrade + pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, + you need to add option ``--use-deprecated legacy-resolver`` to your pip install command. + + While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we + set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well. + Due to those constraints, only ``pip`` installation is currently officially supported. + + While they are some successes with using other tools like `poetry `_ or + `pip-tools `_, they do not share the same workflow as + ``pip`` - especially when it comes to constraint vs. requirements management. + Installing via ``Poetry`` or ``pip-tools`` is not currently supported. + + If you wish to install airflow using those tools you should use the constraint files and convert + them to appropriate format and workflow that your tool requires. .. code-block:: bash diff --git a/README.md b/README.md index 1eb8509937b..265074c0206 100644 --- a/README.md +++ b/README.md @@ -139,15 +139,27 @@ constraints files separately per major/minor python version. You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify correct Airflow tag/version/branch and python versions in the URL. + 1. Installing just Airflow: NOTE!!! On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver -does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice -of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4 -`pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3, you need to add option -`--use-deprecated legacy-resolver` to your pip install command. +might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might +depend on your choice of extras. In order to install Airflow reliably, you might need to either downgrade +pip to version 20.2.4 `pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3, +you might need to add option] `--use-deprecated legacy-resolver` to your pip install command. +While `pip 20.3.3` solved most of the `teething` problems of 20.3, this note will remain here until we +set `pip 20.3` as official version in our CI pipeline where we are testing the installation as well. +Due to those constraints, only `pip` installation is currently officially supported. + +While they are some successes with using other tools like [poetry](https://python-poetry.org) or +[pip-tools](https://pypi.org/project/pip-tools), they do not share the same workflow as +`pip` - especially when it comes to constraint vs. requirements management. +Installing via `Poetry` or `pip-tools` is not currently supported. + +If you wish to install airflow using those tools you should use the constraint files and convert +them to appropriate format and workflow that your tool requires. ```bash diff --git a/docs/apache-airflow/installation.rst b/docs/apache-airflow/installation.rst index b16a5ed4b6e..eae8cdb4596 100644 --- a/docs/apache-airflow/installation.rst +++ b/docs/apache-airflow/installation.rst @@ -64,7 +64,7 @@ either downgrade pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, ``--use-deprecated legacy-resolver`` to your pip install command. While they are some successes with using other tools like `poetry `_ or -`pip-tools `_, but they do not share the same workflow as +`pip-tools `_, they do not share the same workflow as ``pip`` - especially when it comes to constraint vs. requirements management. Installing via ``Poetry`` or ``pip-tools`` is not currently supported. If you wish to install airflow using those tools you should use the :ref:`constraint files ` and convert them to appropriate diff --git a/docs/apache-airflow/start.rst b/docs/apache-airflow/start.rst index b8cc59fe9ce..18653219718 100644 --- a/docs/apache-airflow/start.rst +++ b/docs/apache-airflow/start.rst @@ -23,13 +23,26 @@ Quick Start .. note:: On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver - does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice - of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4 - ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option - ``--use-deprecated legacy-resolver`` to your pip install command. + might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might + depend on your choice of extras. In order to install Airflow you might need to either downgrade + pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, + you need to add option ``--use-deprecated legacy-resolver`` to your pip install command. + While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we + set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well. + Due to those constraints, only ``pip`` installation is currently officially supported. -The installation is quick and straightforward. + While they are some successes with using other tools like `poetry `_ or + `pip-tools `_, they do not share the same workflow as + ``pip`` - especially when it comes to constraint vs. requirements management. + Installing via ``Poetry`` or ``pip-tools`` is not currently supported. + + If you wish to install airflow using those tools you should use the constraint files and convert + them to appropriate format and workflow that your tool requires. + + +The installation of Airflow is painless if you are following the instructions below. Airflow uses +constraint files to enable reproducible installation, so using ``pip`` and constraint files is recommended. .. code-block:: bash @@ -38,8 +51,12 @@ The installation is quick and straightforward. # (optional) export AIRFLOW_HOME=~/airflow - # install from pypi using pip - pip install apache-airflow + AIRFLOW_VERSION=2.0.0 + PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)" + # For example: 3.6 + CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt" + # For example: https://raw.githubusercontent.com/apache/airflow/constraints-2.0.0/constraints-3.6.txt + pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}" # initialize the database airflow db init diff --git a/docs/apache-airflow/upgrade-check.rst b/docs/apache-airflow/upgrade-check.rst index cd670061e82..817f58dfeb9 100644 --- a/docs/apache-airflow/upgrade-check.rst +++ b/docs/apache-airflow/upgrade-check.rst @@ -39,10 +39,22 @@ time of writing: 1.10.14) and then to download this package and run the script a .. note:: On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver - does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice - of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4 - ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option - ``--use-deprecated legacy-resolver`` to your pip install command. + might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might + depend on your choice of extras. In order to install Airflow you might need to either downgrade + pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, + you need to add option ``--use-deprecated legacy-resolver`` to your pip install command. + + While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we + set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well. + Due to those constraints, only ``pip`` installation is currently officially supported. + + While they are some successes with using other tools like `poetry `_ or + `pip-tools `_, they do not share the same workflow as + ``pip`` - especially when it comes to constraint vs. requirements management. + Installing via ``Poetry`` or ``pip-tools`` is not currently supported. + + If you wish to install airflow using those tools you should use the constraint files and convert + them to appropriate format and workflow that your tool requires. .. code-block:: bash