Deprecate the provider and proxy to upstream Airflow provider (#84)

As part of Astronomer's internal plans and decisions, we've decided to contribute the existing functionality provided by the operators and plugins in this repository to the official Apache Airflow Databricks provider. To achieve this, we submitted the following PRs to the Airflow provider: 1. apache/airflow#39178 2. apache/airflow#39771 3. apache/airflow#40013 4. apache/airflow#40724 5. apache/airflow#39295 All functionality has now been contributed to the Airflow Databricks provider, and ongoing support will be maintained there. As a result, we're deprecating the operators and plugins in this repository. Users are encouraged to transition to the official Apache Airflow Databricks provider as soon as possible. The migration process is straightforward—simply update the import path to point to the Airflow provider and ensure that you install `apache-airflow-providers-databricks>=6.8.0`, which includes all the contributions mentioned above. closes: astronomer/issues-airflow#715
astronomer · Aug 8, 2024 · 3c858a4 · 3c858a4
1 parent d49e88c
commit 3c858a4
Show file tree

Hide file tree

Showing 15 changed files with 192 additions and 3,249 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -78,7 +78,7 @@ jobs:
       fail-fast: false
       matrix:
         python: ['3.8', '3.9', '3.10']
-        airflow: [2.5]
+        airflow: [2.7]
     if: >-
       github.event_name == 'push' ||
       (
@@ -125,7 +125,7 @@ jobs:
       fail-fast: false
       matrix:
         python: ['3.8', '3.9', '3.10']
-        airflow: ['2.3', '2.4', '2.5', '2.6', '2.7', '2.8']
+        airflow: ['2.7', '2.8']
 
     if: >-
       github.event_name == 'push' ||
@@ -172,7 +172,7 @@ jobs:
           path: ./.coverage
     env:
       DATABRICKS_CONN_TOKEN: ${{ secrets.DATABRICKS_CONN_TOKEN }}
-      DATABRICKS_CONN_HOST: {{ secrets.DATABRICKS_CONN_HOST }}
+      DATABRICKS_CONN_HOST: ${{ secrets.DATABRICKS_CONN_HOST }}
       DATABRICKS_CONN: ${{ secrets.AIRFLOW_CONN_DATABRICKS_DEFAULT }}
 
 

diff --git a/CHANGELOG.rst b/CHANGELOG.rst
@@ -1,6 +1,14 @@
 Changelog
 =========
 
+0.3.0rc1 (06-08-24)
+-------------------
+
+Deprecations
+
+* Deprecate the provider and proxy instantiations to upstream official Apache Airflow Databricks provider (PR `#84 <https://github.com/astronomer/astro-provider-databricks/pull/84>`_ by @pankajkoti)
+
+
 0.2.2 (16-04-24)
 ----------------
 

diff --git a/README.md b/README.md
@@ -1,3 +1,19 @@
+## Deprecation Notice
+
+With the release ``0.3.0`` of the ``astro-provider-databricks`` package, this provider stands deprecated and will
+no longer receive updates. We recommend migrating to the official ``apache-airflow-providers-databricks>= 6.8.0`` for the latest features and support.
+For the operators and sensors that are deprecated in this repository, migrating to the official Apache Airflow Databricks Provider
+is as simple as changing the import path in your DAG code as per the below examples.
+
+| Previous import path used (Deprecated now)                                    | Suggested import path to use                                                                                |
+|-------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------|
+| `from astro_databricks.operators.notebook import DatabricksNotebookOperator`  | `from airflow.providers.databricks.operators.databricks import DatabricksNotebookOperator`                  |
+| `from astro_databricks.operators.workflow import DatabricksWorkflowTaskGroup` | `from airflow.providers.databricks.operators.databricks_workflow import DatabricksWorkflowTaskGroup`        |
+| `from astro_databricks.operators.common import DatabricksTaskOperator`        | `from airflow.providers.databricks.operators.databricks import DatabricksTaskOperator`                      |
+| `from astro_databricks.plugins.plugin import AstroDatabricksPlugin`           | `from airflow.providers.airflow.providers.databricks.plugins.databricks_workflow import DatabricksWorkflowPlugin` |
+
+# Archives
+
 <h1 align="center">
   Databricks Workflows in Airflow
 </h1>

diff --git a/docs/index.rst b/docs/index.rst
@@ -1,3 +1,30 @@
+.. warning::
+    All the operators and their functionality within this repository have been deprecated and will not receive further updates.
+    Read more about the deprecation in the `Deprecation Notice` section below.
+
+Deprecation Notice
+------------------
+
+With the release ``0.3.0`` of the ``astro-provider-databricks`` package, this provider stands deprecated and will
+no longer receive updates. We recommend migrating to the official ``apache-airflow-providers-databricks>=6.8.0`` for the latest features and support.
+For the operators and sensors that are deprecated in this repository, migrating to the official Apache Airflow Databricks Provider
+is as simple as changing the import path in your DAG code as per the below examples.
+
+.. list-table:: Import paths to change for migrating to the official Apache Airflow Databricks Provider
+   :header-rows: 1
+
+   * - Previous import path used
+     - Newer import path to use
+   * - from astro_databricks.operators.notebook import DatabricksNotebookOperator
+     - from airflow.providers.databricks.operators.databricks import DatabricksNotebookOperator
+   * - from astro_databricks.operators.workflow import DatabricksWorkflowTaskGroup
+     - from airflow.providers.databricks.operators.databricks_workflow import DatabricksWorkflowTaskGroup
+   * - from astro_databricks.operators.common import DatabricksTaskOperator
+     - from airflow.providers.databricks.operators.databricks import DatabricksTaskOperator
+   * - from astro_databricks.plugins.plugin import AstroDatabricksPlugin
+     - from airflow.providers.airflow.providers.databricks.plugins.databricks_workflow import DatabricksWorkflowPlugin
+
+
 Astro Databricks Provider
 =========================
 

diff --git a/pyproject.toml b/pyproject.toml
@@ -35,12 +35,8 @@ classifiers = [
     "Programming Language :: Python :: 3.10",
 ]
 dependencies = [
-    "apache-airflow>=2.3",
-    "databricks-sql-connector>=2.0.4;python_version>='3.10'",
-    "databricks-cli>=0.17.7",
-    "apache-airflow-providers-databricks>=2.2.0",
-    "mergedeep",
-    "pydantic>=1.10.0",
+    "apache-airflow>=2.7",
+    "apache-airflow-providers-databricks>=6.8.0",
 ]
 
 [project.optional-dependencies]

diff --git a/src/astro_databricks/__init__.py b/src/astro_databricks/__init__.py
@@ -3,7 +3,7 @@
 from astro_databricks.operators.notebook import DatabricksNotebookOperator
 from astro_databricks.operators.workflow import DatabricksWorkflowTaskGroup
 
-__version__ = "0.2.2"
+__version__ = "0.3.0rc1"
 __all__ = [
     "DatabricksNotebookOperator",
     "DatabricksWorkflowTaskGroup",