Airflow XCom does not work

[vlavorini]

Describe your problem

While replicating the given example, I hit a bug:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/task/task_runner/standard_task_runner.py", line 85, in _start_by_fork
    args.func(args, dag=self.dag)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 292, in task_run
    _run_task_by_selected_method(args, dag, ti)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 107, in _run_task_by_selected_method
    _run_raw_task(args, ti)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 184, in _run_raw_task
    error_file=args.error_file,
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1332, in _run_raw_task
    self._execute_task_with_callbacks(context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1458, in _execute_task_with_callbacks
    result = self._execute_task(context, self.task)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1517, in _execute_task
    self.xcom_push(key=XCOM_RETURN_KEY, value=result)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 2262, in xcom_push
    session=session,
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 67, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/vineyard/contrib/airflow/xcom/backend.py", line 88, in set
    value = VineyardXCom.serialize_value(value)
  File "/home/airflow/.local/lib/python3.7/site-packages/vineyard/contrib/airflow/xcom/backend.py", line 161, in serialize_value
    client = vineyard.connect(VineyardXCom.options['ipc_socket'])
TypeError: 'method' object is not subscriptable

Maybe at row 161 there is a couple of missing parenthesis?

[sighingnow]

It should be a regression issue. The @property decorator on @classmethod requires python >= 3.9.

I will fix that ASAP.

[sighingnow]

Hi @vlavorini,

The bug is caused by a incompatible syntax for python <3.9, which has been fixed in the main branch and I have uploaded a revised wheel package to pypi, you could upgrade installation via

pip3 install -U airflow-provider-vineyard -i https://pypi.org/simple

[vlavorini]

Hi @sighingnow , thank you for the speed!

However, the (new) pip package does not reflect the changes you made. I still don't see the class __VineyardXComMeta:

[sighingnow]

Hi @vlavorini Sorry that I have just realized that I have made a mistake in the naming of wheel file when republished to pypi.

I have fixed that and re-cut the release v0.3.4 as a quick bug-fix release. You could upgrade the installation while

pip3 install -U airflow-provider-vineyard -i https://pypi.org/simple

Or, downloading the wheel file and install it manually from https://pypi.org/project/airflow-provider-vineyard/0.3.4/#files

[vlavorini]

still something to be done:

    from .backend import VineyardXCom

  File "/home/airflow/.local/lib/python3.7/site-packages/vineyard/contrib/airflow/xcom/backend.py", line 64, in <module>

    class VineyardXCom(BaseXCom, metaclass=__VineyardXComMeta):

TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

[sighingnow]

Thanks for the feedback!

I will take a try on Python 3.7.

[vlavorini]

For completeness: I'm using the docker-compose version of Airflow 2.2

[sighingnow]

Hi @vlavorini, I have removed the use of such tricks and cut new release (v0.3.5 and v0.3.6) to pypi, where the issue has been fixed.

Thanks for your patience!