apache · potiuk · Apr 27, 2021 · Apr 24, 2021 · Apr 25, 2021 · Apr 25, 2021
diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
@@ -553,15 +553,7 @@ Airflow dependencies
 
 .. note::
 
-   On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
-   might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
-   depend on your choice of extras. In order to install Airflow you might need to either downgrade
-   pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
-   you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.
-
-   While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
-   set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
-   Due to those constraints, only ``pip`` installation is currently officially supported.
+   Only ``pip`` installation is currently officially supported.
 
    While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
    `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
@@ -788,15 +780,7 @@ Pinned constraint files
 
 .. note::
 
-   On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
-   might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
-   depend on your choice of extras. In order to install Airflow you might need to either downgrade
-   pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
-   you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.
-
-   While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
-   set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
-   Due to those constraints, only ``pip`` installation is currently officially supported.
+   Only ``pip`` installation is officially supported.
 
    While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
    `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as

diff --git a/CONTRIBUTORS_QUICK_START.rst b/CONTRIBUTORS_QUICK_START.rst
@@ -167,15 +167,7 @@ Setup Airflow with Breeze and PyCharm
 
 .. note::
 
-   On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
-   might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
-   depend on your choice of extras. In order to install Airflow you might need to either downgrade
-   pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
-   you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.
-
-   While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
-   set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
-   Due to those constraints, only ``pip`` installation is currently officially supported.
+   Only ``pip`` installation is currently officially supported.
 
    While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
    `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as

diff --git a/Dockerfile b/Dockerfile
@@ -44,7 +44,7 @@ ARG AIRFLOW_GID="50000"
 
 ARG PYTHON_BASE_IMAGE="python:3.6-slim-buster"
 
-ARG AIRFLOW_PIP_VERSION=20.2.4
+ARG AIRFLOW_PIP_VERSION=21.1
 
 # By default PIP has progress bar but you can disable it.
 ARG PIP_PROGRESS_BAR="on"

diff --git a/Dockerfile.ci b/Dockerfile.ci
@@ -212,7 +212,7 @@ ARG AIRFLOW_PRE_CACHED_PIP_PACKAGES="true"
 # By default in the image, we are installing all providers when installing from sources
 ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
 ARG INSTALL_FROM_PYPI="true"
-ARG AIRFLOW_PIP_VERSION=20.2.4
+ARG AIRFLOW_PIP_VERSION=21.1
 # Setup PIP
 # By default PIP install run without cache to make image smaller
 ARG PIP_NO_CACHE_DIR="true"

diff --git a/IMAGES.rst b/IMAGES.rst
@@ -172,15 +172,7 @@ This will build the image using command similar to:
 
 .. note::
 
-   On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
-   might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
-   depend on your choice of extras. In order to install Airflow you might need to either downgrade
-   pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
-   you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.
-
-   While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
-   set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
-   Due to those constraints, only ``pip`` installation is currently officially supported.
+   Only ``pip`` installation is currently officially supported.
 
    While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
    `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
@@ -632,7 +624,7 @@ The following build arguments (``--build-arg`` in docker build command) can be u
 | ``ADDITIONAL_RUNTIME_APT_ENV``           |                                          | Additional env variables defined         |
 |                                          |                                          | when installing runtime deps             |
 +------------------------------------------+------------------------------------------+------------------------------------------+
-| ``AIRFLOW_PIP_VERSION``                  | ``20.2.4``                               | PIP version used.                        |
+| ``AIRFLOW_PIP_VERSION``                  | ``21.1``                                 | PIP version used.                        |
 +------------------------------------------+------------------------------------------+------------------------------------------+
 | ``PIP_PROGRESS_BAR``                     | ``on``                                   | Progress bar for PIP installation        |
 +------------------------------------------+------------------------------------------+------------------------------------------+

diff --git a/INSTALL b/INSTALL
@@ -28,14 +28,6 @@ java -jar apache-rat.jar -E ./.rat-excludes -d .
 python3 -m venv PATH_TO_YOUR_VENV
 source PATH_TO_YOUR_VENV/bin/activate
 
-NOTE!!
-
-On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
-does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
-of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
-``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
-``--use-deprecated legacy-resolver`` to your pip install command.
-
 # [required] building and installing by pip (preferred)
 pip install .
 

diff --git a/LOCAL_VIRTUALENV.rst b/LOCAL_VIRTUALENV.rst
@@ -63,15 +63,7 @@ Extra Packages
 
 .. note::
 
-   On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
-   might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
-   depend on your choice of extras. In order to install Airflow you might need to either downgrade
-   pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
-   you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.
-
-   While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
-   set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
-   Due to those constraints, only ``pip`` installation is currently officially supported.
+   Only ``pip`` installation is currently officially supported.
 
    While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
    `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as
@@ -137,15 +129,7 @@ To create and initialize the local virtualenv:
 
 .. note::
 
-   On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
-   might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
-   depend on your choice of extras. In order to install Airflow you might need to either downgrade
-   pip to version 20.2.4 ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3,
-   you need to add option ``--use-deprecated legacy-resolver`` to your pip install command.
-
-   While ``pip 20.3.3`` solved most of the ``teething`` problems of 20.3, this note will remain here until we
-   set ``pip 20.3`` as official version in our CI pipeline where we are testing the installation as well.
-   Due to those constraints, only ``pip`` installation is currently officially supported.
+   Only ``pip`` installation is currently officially supported.
 
    While they are some successes with using other tools like `poetry <https://python-poetry.org/>`_ or
    `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as

diff --git a/README.md b/README.md
@@ -149,14 +149,7 @@ correct Airflow tag/version/branch and Python versions in the URL.
 
 NOTE!!!
 
-On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
-might work with Apache Airflow as of 20.3.3, but it might lead to errors in installation. It might
-depend on your choice of extras. In order to install Airflow reliably, you might need to either downgrade
-pip to version 20.2.4 `pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3,
-you might need to add option] `--use-deprecated legacy-resolver` to your pip install command.
-While `pip 20.3.3` solved most of the `teething` problems of 20.3, this note will remain here until we
-set `pip 20.3` as official version in our CI pipeline where we are testing the installation as well.
-Due to those constraints, only `pip` installation is currently officially supported.
+Only `pip` installation is currently officially supported.
 
 While they are some successes with using other tools like [poetry](https://python-poetry.org) or
 [pip-tools](https://pypi.org/project/pip-tools), they do not share the same workflow as

diff --git a/UPDATING.md b/UPDATING.md
@@ -1787,13 +1787,6 @@ you should use `pip install apache-airflow[apache.atlas]`.
 
 NOTE!
 
-On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
-does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
-of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
-`pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3, you need to add option
-`--use-deprecated legacy-resolver` to your pip install command.
-
-
 If you want to install integration for Microsoft Azure, then instead of
 
 ```

diff --git a/airflow/provider.yaml.schema.json b/airflow/provider.yaml.schema.json
@@ -191,6 +191,10 @@
       "items": {
         "type": "string"
       }
+    },
+    "additional-extras": {
+      "type": "object",
+      "description": "Additional extras that the provider should have"
     }
   },
   "additionalProperties": false,

diff --git a/airflow/providers/apache/beam/CHANGELOG.rst b/airflow/providers/apache/beam/CHANGELOG.rst
@@ -19,6 +19,64 @@
 Changelog
 ---------
 
+2.0.0
+.....
+
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+Integration with the ``google`` provider
+````````````````````````````````````````
+
+In 2.0.0 version of the provider we've changed the way of integrating with the ``google`` provider.
+The previous versions of both providers caused conflicts when trying to install them together
+using PIP > 20.2.4. The conflict is not detected by PIP 20.2.4 and below but it was there and
+the version of ``Google BigQuery`` python client was not matching on both sides. As the result, when
+both ``apache.beam`` and ``google`` provider were installed, some features of the ``BigQuery`` operators
+might not work properly. This was cause by ``apache-beam`` client not yet supporting the new google
+python clients when ``apache-beam[gcp]`` extra was used. The ``apache-beam[gcp]`` extra is used
+by ``Dataflow`` operators and while they might work with the newer version of the ``Google BigQuery``
+python client, it is not guaranteed.
+
+This version introduces additional extra requirement for the ``apache.beam`` extra of the ``google`` provider
+and symmetrically the additional requirement for the ``google`` extra of the ``apache.beam`` provider.
+Both ``google`` and ``apache.beam`` provider do not use those extras by default, but you can specify
+them when installing the providers. The consequence of that is that some functionality of the ``Dataflow``
+operators might not be available.
+
+Unfortunately the only ``complete`` solution to the problem is for the ``apache.beam`` to migrate to the
+new (>=2.0.0) Google Python clients.
+
+This is the extra for the ``google`` provider:
+
+.. code-block:: python
+
+        extras_require={
+            ...
+            'apache.beam': ['apache-airflow-providers-apache-beam', 'apache-beam[gcp]'],
+            ....
+        },
+
+And likewise this is the extra for the ``apache.beam`` provider:
+
+.. code-block:: python
+
+        extras_require={'google': ['apache-airflow-providers-google', 'apache-beam[gcp]']},
+
+You can still run this with PIP version <= 20.2.4 and go back to the previous behaviour:
+
+.. code-block:: shell
+
+  pip install apache-airflow-providers-google[apache.beam]
+
+or
+
+.. code-block:: shell
+
+  pip install apache-airflow-providers-apache-beam[google]
+
+But be aware that some ``BigQuery`` operators functionality might not be available in this case.
+
 1.0.1
 .....
 

diff --git a/airflow/providers/apache/beam/README.md b/airflow/providers/apache/beam/README.md
@@ -41,14 +41,6 @@ are in `airflow.providers.apache.beam` python package.
 
 ## Installation
 
-NOTE!
-
-On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
-does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
-of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
-`pip install --upgrade pip==20.2.4` or, in case you use Pip 20.3, you need to add option
-`--use-deprecated legacy-resolver` to your pip install command.
-
 You can install this package on top of an existing airflow 2.* installation via
 `pip install apache-airflow-providers-apache-beam`
 

diff --git a/airflow/providers/apache/beam/provider.yaml b/airflow/providers/apache/beam/provider.yaml
@@ -22,6 +22,7 @@ description: |
     `Apache Beam <https://beam.apache.org/>`__.
 
 versions:
+  - 2.0.0
   - 1.0.1
   - 1.0.0
 
@@ -41,3 +42,6 @@ hooks:
   - integration-name: Apache Beam
     python-modules:
       - airflow.providers.apache.beam.hooks.beam
+
+additional-extras:
+  google: apache-beam[gcp]
diff --git a/airflow/providers/apache/hive/transfers/mssql_to_hive.py b/airflow/providers/apache/hive/transfers/mssql_to_hive.py
@@ -15,7 +15,7 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-
+# pylint: disable=no-member
 """This module contains operator to move data from MSSQL to Hive."""
 
 from collections import OrderedDict

diff --git a/airflow/providers/google/CHANGELOG.rst b/airflow/providers/google/CHANGELOG.rst
@@ -19,6 +19,64 @@
 Changelog
 ---------
 
+3.0.0
+.....
+
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+Integration with the ``apache.beam`` provider
+`````````````````````````````````````````````
+
+In 3.0.0 version of the provider we've changed the way of integrating with the ``apache.beam`` provider.
+The previous versions of both providers caused conflicts when trying to install them together
+using PIP > 20.2.4. The conflict is not detected by PIP 20.2.4 and below but it was there and
+the version of ``Google BigQuery`` python client was not matching on both sides. As the result, when
+both ``apache.beam`` and ``google`` provider were installed, some features of the ``BigQuery`` operators
+might not work properly. This was cause by ``apache-beam`` client not yet supporting the new google
+python clients when ``apache-beam[gcp]`` extra was used. The ``apache-beam[gcp]`` extra is used
+by ``Dataflow`` operators and while they might work with the newer version of the ``Google BigQuery``
+python client, it is not guaranteed.
+
+This version introduces additional extra requirement for the ``apache.beam`` extra of the ``google`` provider
+and symmetrically the additional requirement for the ``google`` extra of the ``apache.beam`` provider.
+Both ``google`` and ``apache.beam`` provider do not use those extras by default, but you can specify
+them when installing the providers. The consequence of that is that some functionality of the ``Dataflow``
+operators might not be available.
+
+Unfortunately the only ``complete`` solution to the problem is for the ``apache.beam`` to migrate to the
+new (>=2.0.0) Google Python clients.
+
+This is the extra for the ``google`` provider:
+
+.. code-block:: python
+
+        extras_require={
+            ...
+            'apache.beam': ['apache-airflow-providers-apache-beam', 'apache-beam[gcp]'],
+            ....
+        },
+
+And likewise this is the extra for the ``apache.beam`` provider:
+
+.. code-block:: python
+
+        extras_require={'google': ['apache-airflow-providers-google', 'apache-beam[gcp]']},
+
+You can still run this with PIP version <= 20.2.4 and go back to the previous behaviour:
+
+.. code-block:: shell
+
+  pip install apache-airflow-providers-google[apache.beam]
+
+or
+
+.. code-block:: shell
+
+  pip install apache-airflow-providers-apache-beam[google]
+
+But be aware that some ``BigQuery`` operators functionality might not be available in this case.
+
 2.2.0
 .....
 

diff --git a/airflow/providers/google/provider.yaml b/airflow/providers/google/provider.yaml
@@ -29,6 +29,7 @@ description: |
       - `Google Workspace <https://workspace.google.pl/>`__ (formerly Google Suite)
 
 versions:
+  - 3.0.0
   - 2.2.0
   - 2.1.0
   - 2.0.0
@@ -742,3 +743,6 @@ extra-links:
   - airflow.providers.google.cloud.operators.bigquery.BigQueryConsoleLink
   - airflow.providers.google.cloud.operators.bigquery.BigQueryConsoleIndexableLink
   - airflow.providers.google.cloud.operators.mlengine.AIPlatformConsoleLink
+
+additional-extras:
+  apache.beam: apache-beam[gcp]
diff --git a/airflow/providers/microsoft/mssql/hooks/mssql.py b/airflow/providers/microsoft/mssql/hooks/mssql.py
@@ -15,6 +15,7 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+# pylint: disable=no-member
 """Microsoft SQLServer hook module"""
 
 import pymssql

diff --git a/dev/provider_packages/PROVIDER_INDEX_TEMPLATE.rst.jinja2 b/dev/provider_packages/PROVIDER_INDEX_TEMPLATE.rst.jinja2
@@ -47,15 +47,6 @@ are in ``{{FULL_PACKAGE_NAME}}`` python package.
 Installation
 ------------
 
-.. note::
-
-    On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
-    does not yet work with Apache Airflow and might lead to errors in installation - depends on your choice
-    of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
-    ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
-    ``--use-deprecated legacy-resolver`` to your pip install command.
-
-
 You can install this package on top of an existing airflow 2.* installation via
 ``pip install {{PACKAGE_PIP_NAME}}``
 {%- if PIP_REQUIREMENTS %}