Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pin kubernetes to a max version of 11.0.0. #11974

Merged
merged 1 commit into from
Oct 30, 2020
Merged

Pin kubernetes to a max version of 11.0.0. #11974

merged 1 commit into from
Oct 30, 2020

Conversation

jedcunningham
Copy link
Member

12.0.0 introduces TypeError: cannot serialize '_io.TextIOWrapper' object when serializing V1Pod's in executor_config.

[2020-10-28 22:12:16,440] {kubernetes_executor.py:616} INFO - Add task TaskInstanceKey(dag_id='python-memory-eater', task_id='eat', execution_date=datetime.datetime(2020, 10, 28, 22, 10, 14, 732352, tzinfo=Timezone('UTC')), try_number=1) with command ['airflow', 'tasks', 'run', 'python-memory-eater', 'eat', '2020-10-28T22:10:14.732352+00:00', '--local', '--pool', 'default_pool', '--subdir', '/dags/python-memory-eater.py'] with executor_config {'KubernetesExecutor': {'request_memory': '128Mi', 'limit_memory': '128Mi'}}
[2020-10-28 22:12:16,440] {warnings.py:99} WARNING - /some/path/site-packages/airflow/kubernetes/pod_generator.py:200: DeprecationWarning: Using a dictionary for the executor_config is deprecated and will soon be removed.please use a `kubernetes.client.models.V1Pod` class with a "pod_override" key instead.
  category=DeprecationWarning)
[2020-10-28 22:12:16,443] {scheduler_job.py:1326} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
  File "/some/path/site-packages/airflow/jobs/scheduler_job.py", line 1307, in _execute
    self._run_scheduler_loop()
  File "/some/path/site-packages/airflow/jobs/scheduler_job.py", line 1380, in _run_scheduler_loop
    self.executor.heartbeat()
  File "/some/path/site-packages/airflow/executors/base_executor.py", line 151, in heartbeat
    self.trigger_tasks(open_slots)
  File "/some/path/site-packages/airflow/executors/base_executor.py", line 183, in trigger_tasks
    executor_config=ti.executor_config)
  File "/some/path/site-packages/airflow/executors/kubernetes_executor.py", line 622, in execute_async
    self.task_queue.put((key, command, kube_executor_config))
  File "<string>", line 2, in put 
  File "/usr/lib64/python3.6/multiprocessing/managers.py", line 756, in _callmethod
    conn.send((self._id, methodname, args, kwds))
  File "/usr/lib64/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib64/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot serialize '_io.TextIOWrapper' object

This applies to both the deprecated executor_config dict and pod_override.

12.0.0 introduces `TypeError: cannot serialize '_io.TextIOWrapper'
object` when serializing V1Pod's in `executor_config`.
@mik-laj
Copy link
Member

mik-laj commented Oct 30, 2020

Did you also open a bug in the client library project? This looks like a problem the library should fix. V1Pod should be pickable.

@github-actions
Copy link

The PR needs to run all tests because it modifies core of Airflow! Please rebase it to latest master or ask committer to re-run it!

@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Oct 30, 2020
@dimberman dimberman merged commit 9687b3b into apache:master Oct 30, 2020
@jedcunningham
Copy link
Member Author

@mik-laj no, and turns out I can't replicate it with the client library alone... Ongoing conversation here:
https://apache-airflow.slack.com/archives/C0146STM600/p1603982769186300

@jedcunningham jedcunningham deleted the pin_kubernetes branch October 30, 2020 18:30
@jedcunningham
Copy link
Member Author

This did ultimately lead to an issue over there: kubernetes-client/python#1304

szn pushed a commit to szn/airflow that referenced this pull request Nov 1, 2020
12.0.0 introduces `TypeError: cannot serialize '_io.TextIOWrapper'
object` when serializing V1Pod's in `executor_config`.
potiuk pushed a commit that referenced this pull request Nov 14, 2020
12.0.0 introduces `TypeError: cannot serialize '_io.TextIOWrapper'
object` when serializing V1Pod's in `executor_config`.

(cherry picked from commit 9687b3b)
@potiuk potiuk added this to the Airflow 1.10.13 milestone Nov 14, 2020
potiuk pushed a commit that referenced this pull request Nov 16, 2020
12.0.0 introduces `TypeError: cannot serialize '_io.TextIOWrapper'
object` when serializing V1Pod's in `executor_config`.

(cherry picked from commit 9687b3b)
potiuk pushed a commit that referenced this pull request Nov 16, 2020
12.0.0 introduces `TypeError: cannot serialize '_io.TextIOWrapper'
object` when serializing V1Pod's in `executor_config`.

(cherry picked from commit 9687b3b)
kaxil pushed a commit that referenced this pull request Nov 18, 2020
12.0.0 introduces `TypeError: cannot serialize '_io.TextIOWrapper'
object` when serializing V1Pod's in `executor_config`.

(cherry picked from commit 9687b3b)
kaxil pushed a commit to astronomer/airflow that referenced this pull request Nov 27, 2020
12.0.0 introduces `TypeError: cannot serialize '_io.TextIOWrapper'
object` when serializing V1Pod's in `executor_config`.

(cherry picked from commit 9687b3b)
(cherry picked from commit 509a097)
kaxil added a commit to astronomer/ap-airflow that referenced this pull request Dec 21, 2020
apache/airflow#11974 -- The new version of Kubernetes raises error when it was not raising an error previously
kaxil added a commit to astronomer/ap-airflow that referenced this pull request Dec 21, 2020
apache/airflow#11974 -- The new version of Kubernetes raises error when it was not raising an error previously
kaxil pushed a commit to astronomer/airflow that referenced this pull request Dec 21, 2020
12.0.0 introduces `TypeError: cannot serialize '_io.TextIOWrapper'
object` when serializing V1Pod's in `executor_config`.

(cherry picked from commit 9687b3b)
(cherry picked from commit 509a097)
kaxil pushed a commit to astronomer/airflow that referenced this pull request Dec 21, 2020
12.0.0 introduces `TypeError: cannot serialize '_io.TextIOWrapper'
object` when serializing V1Pod's in `executor_config`.

(cherry picked from commit 9687b3b)
(cherry picked from commit 509a097)
(cherry picked from commit d441d0e)
cfei18 pushed a commit to cfei18/incubator-airflow that referenced this pull request Mar 5, 2021
12.0.0 introduces `TypeError: cannot serialize '_io.TextIOWrapper'
object` when serializing V1Pod's in `executor_config`.

(cherry picked from commit 9687b3b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
full tests needed We need to run full set of tests for this PR to merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants