Unused threads and thread pools in kubernetes.watch.Watch._api_client

[bgagnon]

TL;DR every Watch object consumes 16 threads for no apparent reason

kubernetes.watch.Watch creates an APIClient in its constructor:

    def __init__(self, return_type=None):
        self._raw_return_type = return_type
        self._stop = False
        self._api_client = client.ApiClient()
        self.resource_version = 0

Which is only used for deserialize purposes, not for making calls:

            js['object'] = self._api_client.deserialize(obj, return_type)

The problem is that ApiClient is a rather expensive object that creates a new ThreadPool and RESTClientObject every time:

    def __init__(self, configuration=None, header_name=None, header_value=None, cookie=None):
        if configuration is None:
            configuration = Configuration()
        self.configuration = configuration

        self.pool = ThreadPool()
        self.rest_client = RESTClientObject(configuration)

The RESTClientObject is default-constructed (no configuration), which implies:

        self.connection_pool_maxsize = multiprocessing.cpu_count() * 5

and

class RESTClientObject(object):

    def __init__(self, configuration, pools_size=4, maxsize=None):

[bgagnon]

Workaround that kills some threads right after the Watch object is constructed:

watch = kubernetes.watch.Watch()
watch._api_client.pool.close()
watch._api_client.rest_client.pool_manager.clear()

[minrk]

This is a bug in the upstream swagger-codegen project fixed by swagger-api/swagger-codegen#8061 . Once that is merged, re-rendering with the updated swagger-codegen should fix it.

[bidyut1990]

Is there any update to this? I am also trying to use many watches simultaneously and its hogging a lot of memory. Any advise to make a client(capable of setting up watches and digesting the events) really scalable in the most efficient manner.

[tomplus]

@bidyut1990 which version do you use? It should be fixed in v9.0.0.

If you need to handle many watches in your application you may be interested asynchronous fork of this library kubernetes-asyncio.

[bidyut1990]

@tomplus - I am using 9.0.0 only but i did not realize it's been fixed. Anyways, i ll also take a look at the async based package.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[yliaog]

@bidyut1990 could you please verify if the fix works?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.