IncompleteRead while watching

[max-rocket-internet]

What happened (please include outputs or screenshots):

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 603, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 362, in _error_catcher
    yield
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 668, in read_chunked
    self._update_chunk_length()
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 607, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "controller.py", line 21, in <module>
    for event in stream:
  File "/usr/local/lib/python3.7/site-packages/kubernetes/watch/watch.py", line 134, in stream
    for line in iter_resp_lines(resp):
  File "/usr/local/lib/python3.7/site-packages/kubernetes/watch/watch.py", line 47, in iter_resp_lines
    for seg in resp.read_chunked(decode_content=False):
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 696, in read_chunked
    self._original_response.close()
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/urllib3/response.py", line 380, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

What you expected to happen:

The watcher to always recover and continue watching for events.

How to reproduce it (as minimally and precisely as possible):

Tricky. I've only seen it once and it was random. My application just watches a CRD for changes.

Environment:

• Kubernetes version (kubectl version): v1.12.10-eks
• Python version (python --version) 3.7.0
• Python client version (pip list | grep kubernetes) 9.0.0

Duplicate issues:

Identical issues that were close by the bot: #728 #540

[micw523]

There should be a PR in progress: kubernetes-client/python-base#133

[jinchihe]

Hey @micw523 How to work around this? Seems the PR still be not merged. Thanks.

[micw523]

Hi @jinchihe , @roycaihw might have a better idea about this.

[jinchihe]

@roycaihw is there way to work around the issue? except shorten waiting time. Thanks a lot!

[jinchihe]

I worked around this via @retrying.retry. :-) but still I think we need the enhancement, thanks.

[max-rocket-internet]

I worked around this via @retrying.retry

How does that work?

[jinchihe]

@max-rocket-internet I added the @retrying.retry(wait_fixed=1000, stop_max_attempt_number=20) for my watching function :-)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pitwegner]

/reopen
/remove-lifecycle rotten
This issue is still present

[k8s-ci-robot]

@pitwegner: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen
/remove-lifecycle rotten
This issue is still present

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Guanyan1996]

This issue is still present