Add Apache Beam operators #12814

TobKed · 2020-12-04T16:36:45Z

Simple Apache Beam operators to run Python and Java pipelines with DirectRunner (DirectRunner is used by default but other runners are supported as well).

It will allow to test DAGs with pipelines which uses DirectRunner for faster feedback, then using e.g. Dataflow operators.

aaltay

I think this is a great idea. I did not review line by line but it looks like it is in good shape.

One question: How would you differentiate between Beam operator and Dataflow operator? Would it make sense to document somewhere what are the differences and when to use one or the other.

Another point: It would be good to align parameters between Beam and Dataflow operators so users (if they want to) could switch between the two operators easily.

aaltay · 2020-12-04T18:42:17Z

airflow/providers/apache/beam/example_dags/example_beam.py

+GCS_INPUT = os.environ.get('APACHE_BEAM_PYTHON', 'gs://apache-beam-samples/shakespeare/kinglear.txt')
+GCS_TMP = os.environ.get('APACHE_BEAM_GCS_TMP', 'gs://test-dataflow-example/temp/')
+GCS_STAGING = os.environ.get('APACHE_BEAM_GCS_STAGING', 'gs://test-dataflow-example/staging/')
+GCS_OUTPUT = os.environ.get('APACHE_BEAM_GCS_OUTPUT', 'gs://test-dataflow-example/output')


Curiosity question: Does gs://test-dataflow-example/ exist as a publicly writable location?

It is not publicly writable location. This example DAG is used in system tests (tests/providers/apache/beam/operators/test_beam_system.py) which is being run by pytest with flag --system e.g. --system=google and when system test is executed user should provide GCP connection with proper permissions to storage and Dataflow resources. Environmental variables with resources should be adjusted as well if necessary.

aaltay · 2020-12-04T18:43:08Z

airflow/providers/apache/beam/example_dags/example_beam.py

+        },
+        py_options=[],
+        job_name='{{task.task_id}}',
+        py_requirements=['apache-beam[gcp]==2.21.0'],


Use a newer version in examples? 2.25.0 is the latest, 2.26.0 is imminent.

Thanks. I will update to 2.25.0 for now.

Cool. (Btw 2.26 is available as of today.)

Thanks. I update version to the latest one :)

aaltay · 2020-12-04T18:49:55Z

airflow/providers/apache/beam/example_dags/example_beam.py

+    tags=['example'],
+) as dag_native_python:
+
+    start_python_job_local_direct_runner = BeamRunPythonPipelineOperator(


I think this and start_python_job_direct_runner are a bit duplicative. They more or less do the same exact thing except for the location of module to run.

Another role of example dags is bo be part of documentation and cen be visible from docs (example from PR for databricks).

I refactored it slightly so local task is "fully" local and in second task both input and output will be on GCS. My idea was to show working example showing usage different sources/outputs. Also executing system tests will confirm that both tasks works.

aaltay · 2020-12-04T18:50:28Z

airflow/providers/apache/beam/example_dags/example_beam.py

+        py_system_site_packages=False,
+    )
+
+    start_python_job_local_direct_runner >> start_python_job_local_flink_runner


Are these two lines equivalent to [start_python_job_local_direct_runner, start_python_job_direct_runner] >> start_python_job_local_flink_runner

You are right. It will be easier to follow the DAG, i made small refactor of this part.

aaltay · 2020-12-04T18:51:32Z

airflow/providers/apache/beam/hooks/beam.py

+        for readable_fd in reads:
+            self._process_fd(readable_fd)
+
+        self.log.info("Process exited with return code: %s", self._proc.returncode)


Is it possible to capture stdout or stderr, that usually has the real relevant errors.

printing stdout/stderr to logs is handled by _process_fd function two lines above :) I used logs produced by it extensively during debugging of example dag and I can confirm they provide helpful information about errors.

airflow/providers/apache/beam/provider.yaml

aaltay · 2020-12-04T18:54:36Z

tests/providers/apache/beam/hooks/test_beam.py

+        wait_for_done.assert_called_once_with()
+
+
+class TestDataflow(unittest.TestCase):


You can also run a real Beam pipeline with DirectRunner as another test.

I found there is mistake in the title (should be TestBeamRunner) and it is purpose is to test providers.apache.beam.hooks.beam._BeamRunner which is responsible for proper executing command in subprocess, printing stdout/stderr to logs and and handling exit code. In other tests this class is mocked and if runner is Dataflow slightly different command runner will be used: airflow.providers.google.cloud.hooks.dataflow._DataflowRunner.

github-actions · 2020-12-04T20:25:55Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

turbaszek · 2020-12-04T22:54:24Z

airflow/providers/apache/beam/hooks/beam.py

+        ]
+        if variables:
+            cmd.extend(self._options_to_args(variables))
+        _BeamRunner(cmd=cmd).wait_for_done()


Should we allow async mode (op + sensor)? So the running pipeline is not blocking worker slot? I suppose this may be hard as the subprocess may be running in worker other than the sensor is using. However we may consider load balancing task to make sure the sensor will be scheduled on proper worker. See #12519 (comment) for ideas

Although, this is a bigger change than this PR

airflow/providers/apache/beam/provider.yaml

turbaszek · 2020-12-04T23:19:56Z

airflow/providers/apache/beam/operators/beam.py

+                )
+                self.py_file = tmp_gcs_file.name
+
+            if self.runner == "DataflowRunner":


While I agree that logging a warning or even using a UserWarning for DataflowRunner is a good thing, I would say that in case of operator behaviour we should have only one, consistent behaviour. That's what I would at least expect as a user. Mind I'm not beam expert

I am not sure did I understand your comment correctly. AFAIK the behavior from user perspective will be the same, just if user wants to use more detaflow specific features available in DataflowOperator will be informed about it. Could you elaborate more?

Correct me if I'm wrong, but we use two different hooks depending if it's dataflow or not. So at least at this point the behaviour is not identical (unless the hooks implements exactly the same logic but then - why to have two different hooks?). So I would be in favour of using only BeamHook and rising a warning if users use DataflowRunner. WDYT?

github-actions · 2020-12-07T21:04:32Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

github-actions · 2020-12-08T12:50:46Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

github-actions · 2020-12-09T15:55:22Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

github-actions · 2020-12-09T18:39:12Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

github-actions · 2020-12-09T20:45:30Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

github-actions · 2020-12-14T12:57:27Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

github-actions · 2020-12-18T10:53:25Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

github-actions · 2020-12-18T10:54:02Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

github-actions · 2021-01-13T12:33:03Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

TobKed · 2021-01-18T14:27:08Z

PTAL @aaltay @mik-laj @turbaszek
I updated PR with refactor which moves common logic between Dataflow and Beam to Beam.

TobKed · 2021-02-03T07:52:52Z

thanks for all reviewers! I rebased on the latest master and squashed fixups

potiuk · 2021-02-03T11:13:23Z

Transient doc error :(

github-actions · 2021-02-03T12:27:00Z

The Workflow run is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.

github-actions · 2021-02-03T12:34:09Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

github-actions · 2021-02-03T18:07:14Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

github-actions · 2021-02-03T18:08:04Z

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

* Refactor common logic * Tests

Co-authored-by: Tomek Urbaszek <tomasz.urbaszek@polidea.com>

potiuk · 2021-02-03T20:33:43Z

Passing in my fork: https://github.com/potiuk/airflow/runs/1825273695 merging it :)

RosterIn · 2021-02-08T06:45:40Z

airflow/providers/apache/beam/hooks/beam.py

+from typing import Callable, List, Optional
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook


@TobKed This raises warning that spams the logs:

/opt/airflow/airflow/providers/apache/beam/hooks/beam.py:28: DeprecationWarning: This module is deprecated. Please use `airflow.hooks.base`. from airflow.hooks.base_hook import BaseHook

Solved in #14554

(cherry picked from commit 1872d87)

boring-cyborg bot added the provider:Apache label Dec 4, 2020

TobKed mentioned this pull request Dec 4, 2020

Add Apache Beam Operator PolideaInternal/airflow#970

Closed

aaltay suggested changes Dec 4, 2020

View reviewed changes

turbaszek reviewed Dec 4, 2020

View reviewed changes

airflow/providers/apache/beam/provider.yaml Outdated Show resolved Hide resolved

turbaszek reviewed Dec 4, 2020

View reviewed changes

airflow/providers/apache/beam/provider.yaml Outdated Show resolved Hide resolved

turbaszek reviewed Dec 4, 2020

View reviewed changes

TobKed force-pushed the add-apache-beam branch from ed8f776 to 12a8cfe Compare December 7, 2020 15:33

TobKed marked this pull request as ready for review December 7, 2020 15:34

TobKed mentioned this pull request Dec 8, 2020

Add list of prerequisites that new provider package should fulfill #10386

Closed

10 tasks

TobKed force-pushed the add-apache-beam branch from 406afe7 to b55c76b Compare December 9, 2020 13:14

TobKed force-pushed the add-apache-beam branch from b55c76b to bb80626 Compare December 9, 2020 18:08

TobKed force-pushed the add-apache-beam branch from bb80626 to 7580003 Compare December 9, 2020 19:07

aaltay approved these changes Dec 12, 2020

View reviewed changes

TobKed force-pushed the add-apache-beam branch from 7580003 to e341463 Compare December 14, 2020 12:22

TobKed force-pushed the add-apache-beam branch 2 times, most recently from b654368 to 362b362 Compare December 18, 2020 10:08

TobKed force-pushed the add-apache-beam branch from 362b362 to 779e44f Compare January 13, 2021 09:16

TobKed force-pushed the add-apache-beam branch from 779e44f to aa0414d Compare January 18, 2021 14:09

TobKed force-pushed the add-apache-beam branch 2 times, most recently from 1690a98 to 8e1d421 Compare February 3, 2021 07:51

Tobiasz Kędzierski and others added 7 commits February 3, 2021 19:18

Add Apache Beam operators

5f2bd80

Refactor common logic to Apache Beam and Dataflow

33f756e

* Refactor common logic * Tests

Fix backport wheels

1da37cf

Add howto guide

b233f6e

Update provider Readme

d3f8799

Update airflow/providers/apache/beam/hooks/beam.py

54749a1

Co-authored-by: Tomek Urbaszek <tomasz.urbaszek@polidea.com>

Add changelog

1a4844e

TobKed force-pushed the add-apache-beam branch from 8e1d421 to 1a4844e Compare February 3, 2021 18:18

potiuk merged commit 1872d87 into apache:master Feb 3, 2021

RosterIn reviewed Feb 8, 2021

View reviewed changes

potiuk pushed a commit that referenced this pull request Mar 3, 2021

Add Apache Beam operators (#12814)

fd4c91c

(cherry picked from commit 1872d87)

potiuk pushed a commit that referenced this pull request Mar 3, 2021

Add Apache Beam operators (#12814)

f394df3

(cherry picked from commit 1872d87)

potiuk pushed a commit that referenced this pull request Mar 3, 2021

Add Apache Beam operators (#12814)

87e747f

(cherry picked from commit 1872d87)

potiuk mentioned this pull request Apr 4, 2022

Add support for Delta Sharing protocol #22692

Closed

shahar1 mentioned this pull request Mar 3, 2023

Add support for impersonation in GKEStartPodOperator, DataflowCreateJavaJobOperator and DataflowCreatePythonJobOperator #10596

Closed

ferruzzi mentioned this pull request Jul 8, 2023

D205 Support - Provider: Google #32356

Merged

		wait_for_done.assert_called_once_with()


		class TestDataflow(unittest.TestCase):

Add Apache Beam operators #12814

Add Apache Beam operators #12814

Conversation

TobKed commented Dec 4, 2020

aaltay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Dec 4, 2020

turbaszek Dec 4, 2020 • edited Loading

Choose a reason for hiding this comment

turbaszek Dec 4, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Dec 7, 2020

github-actions bot commented Dec 8, 2020

github-actions bot commented Dec 9, 2020

github-actions bot commented Dec 9, 2020

github-actions bot commented Dec 9, 2020

github-actions bot commented Dec 14, 2020

github-actions bot commented Dec 18, 2020

github-actions bot commented Dec 18, 2020

github-actions bot commented Jan 13, 2021

TobKed commented Jan 18, 2021 • edited Loading

TobKed commented Feb 3, 2021

potiuk commented Feb 3, 2021

github-actions bot commented Feb 3, 2021

github-actions bot commented Feb 3, 2021

github-actions bot commented Feb 3, 2021

github-actions bot commented Feb 3, 2021

potiuk commented Feb 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turbaszek Dec 4, 2020 •

edited

Loading

turbaszek Dec 4, 2020 •

edited

Loading

TobKed commented Jan 18, 2021 •

edited

Loading