Skip to content

Commit

Permalink
Update installation notes to warn against common problems.
Browse files Browse the repository at this point in the history
We have recently seen a number of issues created by users who
tried to install airflow with poetry or pip-tools or who had
successes with using the latest pip 20.3.3. This change aims
to update the 'note' content and make sure installation
instructions are consistent everywhere, so that new users
are warned against using anything else than PIP and that they
are aware about potential problems with 'pip 20.3' and ways
to mitigate the problems.

This responds to the needs of confused users such as
one in apache#13711 (comment)
  • Loading branch information
potiuk committed Jan 17, 2021
1 parent 2f79fb9 commit 6e8418a
Show file tree
Hide file tree
Showing 8 changed files with 172 additions and 50 deletions.
40 changes: 32 additions & 8 deletions CONTRIBUTING.rst
Original file line number Diff line number Diff line change
Expand Up @@ -671,10 +671,22 @@ Airflow dependencies
.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
``pip`` - especially when it comes to constraint vs. requirements management.
Installing via ``Poetry`` or ``pip-tools`` is not currently supported.

If you wish to install airflow using those tools you should use the constraint files and convert
them to appropriate format and workflow that your tool requires.


Extras
Expand Down Expand Up @@ -904,10 +916,22 @@ Pinned constraint files
.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
``pip`` - especially when it comes to constraint vs. requirements management.
Installing via ``Poetry`` or ``pip-tools`` is not currently supported.

If you wish to install airflow using those tools you should use the constraint files and convert
them to appropriate format and workflow that your tool requires.


By default when you install ``apache-airflow`` package - the dependencies are as open as possible while
Expand Down
49 changes: 35 additions & 14 deletions CONTRIBUTORS_QUICK_START.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,11 @@ Prerequisites


Installing Prerequisites on Ubuntu
###################################
##################################


Docker Community Edition
-----------------------------------
------------------------


1. Installing required packages for Docker and setting up docker repo
Expand Down Expand Up @@ -84,7 +84,7 @@ Note : After adding user to docker group Logout and Login again for group member
Docker Compose
--------------------------
--------------

1. Installing latest version of Docker Compose

Expand All @@ -108,8 +108,8 @@ Docker Compose
pyenv and setting up virtual-env
--------------------------------------------
Pyenv and setting up virtual-env
--------------------------------

1. Checking required packages

Expand Down Expand Up @@ -162,7 +162,28 @@ pyenv and setting up virtual-env
Setup Airflow with Breeze and PyCharm
#####################################################
#####################################


.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
``pip`` - especially when it comes to constraint vs. requirements management.
Installing via ``Poetry`` or ``pip-tools`` is not currently supported.

If you wish to install airflow using those tools you should use the constraint files and convert
them to appropriate format and workflow that your tool requires.


Forking and cloning Project
Expand Down Expand Up @@ -274,7 +295,7 @@ Setting up Breeze
source ~/.bashrc
Using Breeze
-----------------------------------------
------------

1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run(
In this case python and backend will be picked up from last execution ``./breeze --python 3.8 --backend mysql``)
Expand Down Expand Up @@ -457,7 +478,7 @@ Following are some of important topics of Breeze documentation:


Setting up Debug
-----------------------------------------
----------------

1. Configuring Airflow database connection

Expand Down Expand Up @@ -528,11 +549,11 @@ Setting up Debug


Starting development
###################################
####################


Creating a branch
-----------------------------------
-----------------

1. Click on branch symbol in the bottom right corner of Pycharm

Expand All @@ -555,7 +576,7 @@ Creating a branch


Testing
-----------------------------------
-------

All Tests are inside ./tests directory.

Expand Down Expand Up @@ -720,7 +741,7 @@ All Tests are inside ./tests directory.


Pre-commit
-----------------------------------
----------

Before committing changes to github or raising a pull request, code needs to be checked for certain quality standards
such as spell check, code syntax, code formatting, compatibility with Apache License requirements etc. This set of
Expand Down Expand Up @@ -881,7 +902,7 @@ To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be r


Contribution guide
-----------------------------------
------------------

- To know how to contribute to the project visit |CONTRIBUTING.rst|

Expand Down Expand Up @@ -917,7 +938,7 @@ Contribution guide


Raising Pull Request
-----------------------------------
--------------------

1. Go to your Github account and open your fork project and click on Branches

Expand Down
20 changes: 16 additions & 4 deletions IMAGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -123,10 +123,22 @@ parameter to Breeze:
.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
``pip`` - especially when it comes to constraint vs. requirements management.
Installing via ``Poetry`` or ``pip-tools`` is not currently supported.

If you wish to install airflow using those tools you should use the constraint files and convert
them to appropriate format and workflow that your tool requires.


This will build the image using command similar to:
Expand Down
40 changes: 32 additions & 8 deletions LOCAL_VIRTUALENV.rst
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,22 @@ Extra Packages
.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
``pip`` - especially when it comes to constraint vs. requirements management.
Installing via ``Poetry`` or ``pip-tools`` is not currently supported.

If you wish to install airflow using those tools you should use the constraint files and convert
them to appropriate format and workflow that your tool requires.


You can also install extra packages (like ``[ssh]``, etc) via
Expand Down Expand Up @@ -126,10 +138,22 @@ To create and initialize the local virtualenv:
.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
``pip`` - especially when it comes to constraint vs. requirements management.
Installing via ``Poetry`` or ``pip-tools`` is not currently supported.

If you wish to install airflow using those tools you should use the constraint files and convert
them to appropriate format and workflow that your tool requires.


.. code-block:: bash
Expand Down
20 changes: 16 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,15 +139,27 @@ constraints files separately per major/minor python version.
You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify
correct Airflow tag/version/branch and python versions in the URL.


1. Installing just Airflow:

NOTE!!!

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
`pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3, you need to add option
`--use-deprecated legacy-resolver` to your pip install command.
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow reliably, you might need to either downgrade
pip to version 20.2.4 `pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3,
you might need to add option] `--use-deprecated legacy-resolver` to your pip install command.
While `pip 20.3.3` solved most of the `teething` problems of 20.3, this note will remain here until we
set `pip 20.3` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only `pip` installation is currently officially supported.

While they are some successes with using other tools like [poetry](https://python-poetry.org) or
[pip-tools](https://pypi.org/project/pip-tools), they do not share the same workflow as
`pip` - especially when it comes to constraint vs. requirements management.
Installing via `Poetry` or `pip-tools` is not currently supported.

If you wish to install airflow using those tools you should use the constraint files and convert
them to appropriate format and workflow that your tool requires.


```bash
Expand Down
2 changes: 1 addition & 1 deletion docs/apache-airflow/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ either downgrade pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or,
``--use-deprecated legacy-resolver`` to your pip install command.

While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, but they do not share the same workflow as
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
``pip`` - especially when it comes to constraint vs. requirements management.
Installing via ``Poetry`` or ``pip-tools`` is not currently supported. If you wish to install airflow
using those tools you should use the :ref:`constraint files <installation:constraints>` and convert them to appropriate
Expand Down
31 changes: 24 additions & 7 deletions docs/apache-airflow/start.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,26 @@ Quick Start
.. note::

On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
``--use-deprecated legacy-resolver`` to your pip install command.
might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
depend on your choice of extras. In order to install Airflow you might need to either downgrade
pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.

While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
Due to those constraints, only ``pip`` installation is currently officially supported.

The installation is quick and straightforward.
While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
`pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
``pip`` - especially when it comes to constraint vs. requirements management.
Installing via ``Poetry`` or ``pip-tools`` is not currently supported.

If you wish to install airflow using those tools you should use the constraint files and convert
them to appropriate format and workflow that your tool requires.


The installation of Airflow is painless if you are following the instructions below. Airflow uses
constraint files to enable reproducible installation, so using ``pip`` and constraint files is recommended.

.. code-block:: bash
Expand All @@ -38,8 +51,12 @@ The installation is quick and straightforward.
# (optional)
export AIRFLOW_HOME=~/airflow
# install from pypi using pip
pip install apache-airflow
AIRFLOW_VERSION=2.0.0
PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
# For example: 3.6
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
# For example: https://raw.githubusercontent.com/apache/airflow/constraints-2.0.0/constraints-3.6.txt
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
# initialize the database
airflow db init
Expand Down
Loading

0 comments on commit 6e8418a

Please sign in to comment.