Skip to content

Commit

Permalink
Workaround kubernetes python client deadlock issue
Browse files Browse the repository at this point in the history
The kubernetes python client has a bug [1] which results in frequent
deadlocks while being cleaned up, which causes armada to hang at the
end of execution.

This patchset works around that issue by mocking out the associated
thread pools, since they are only needed for async kubernetes api
calls, which armada does not use.

[1]: kubernetes-client/python#411

Change-Id: I71fbfbe355347ae2ddd02ffd26d881368320246b
  • Loading branch information
seaneagan committed Nov 26, 2018
1 parent a64d435 commit 673b1ed
Showing 1 changed file with 8 additions and 0 deletions.
8 changes: 8 additions & 0 deletions armada/handlers/k8s.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,9 @@
from kubernetes import client
from kubernetes import config
from kubernetes import watch
from kubernetes.client import api_client
from kubernetes.client.rest import ApiException
from unittest.mock import Mock
from oslo_config import cfg
from oslo_log import log as logging

Expand All @@ -27,6 +29,12 @@
CONF = cfg.CONF
LOG = logging.getLogger(__name__)

# TODO: Remove after this bug is fixed and we have uplifted to a fixed version:
# https://github.com/kubernetes-client/python/issues/411
# Avoid creating thread pools in kubernetes api_client.
_dummy_pool = Mock()
api_client.ThreadPool = lambda *args, **kwargs: _dummy_pool


class K8s(object):
'''
Expand Down

0 comments on commit 673b1ed

Please sign in to comment.