Unmarshalling fails in log stream (read_namespaced_pod_log) obtained via watch

[bluenote10]

What happened (please include outputs or screenshots):

I'm using the following code to monitor log outputs of a pod:

from kubernetes import client, config, watch

config.load_kube_config()
k_api = client.CoreV1Api()

w = watch.Watch()
for msg in w.stream(k_api.read_namespaced_pod_log, name="mypod", namespace="mynamespace"):
    print(msg)

This works in general. However if the process running in the pod prints a line that consists of a plain integer (e.g. echo 42), the message unmarshalling crashes:

TypeError                                 Traceback (most recent call last)
/path/to/script.py in <module>()
      1 w = watch.Watch()
----> 2 for msg in w.stream(k_api.read_namespaced_pod_log, name="mypod", namespace="mynamespace"):
      3     print(msg)

/home/myuser/.virtualenvs/ml/local/lib/python2.7/site-packages/kubernetes/watch/watch.pyc in stream(self, func, *args, **kwargs)
    143             try:
    144                 for line in iter_resp_lines(resp):
--> 145                     yield self.unmarshal_event(line, return_type)
    146                     if self._stop:
    147                         break

/home/myuser/.virtualenvs/ml/local/lib/python2.7/site-packages/kubernetes/watch/watch.pyc in unmarshal_event(self, data, return_type)
     91         except ValueError:
     92             return data
---> 93         js['raw_object'] = js['object']
     94         if return_type:
     95             obj = SimpleNamespace(data=json.dumps(js['raw_object']))

TypeError: 'int' object has no attribute '__getitem__'

What you expected to happen:

It should not crash depending on the line content.

How to reproduce it (as minimally and precisely as possible):

• Use the code above to fetch logs.
• echo 42 in a pod.

Anything else we need to know?:

• The problem only occurs when accessing logs as a stream. Fetching the entire logs with a plain read_namespaced_pod_log works even if lines contain pure integeres.
• Possibly related to Failed to get a log stream when log line is json type #982 ?

Environment:

• Kubernetes version (kubectl version):

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:23:26Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

• OS: Ubuntu 18.04
• Python version: Python 2.7.12
• Python client version: 10.0.1

[roycaihw]

/assign

[bluenote10]

I think I have found another problem with reading log message streams: empty lines.

If I do something like echo -e "\n" in a POD, the code above errors with:

  File "myscript.py", line X, in some_func
    for msg in w.stream(k_api.read_namespaced_pod_log, name="mypod", namespace="mynamespace"):
  File "<VENV-PATH>/local/lib/python2.7/site-packages/kubernetes/watch/watch.py", line 144, in stream
    for line in iter_resp_lines(resp):
  File "<VENV-PATH>/local/lib/python2.7/site-packages/kubernetes/watch/watch.py", line 48, in iter_resp_lines
    for seg in resp.read_chunked(decode_content=False):
  File "<VENV-PATH>/local/lib/python2.7/site-packages/urllib3/response.py", line 731, in read_chunked
    "Response is not chunked. "
urllib3.exceptions.ResponseNotChunked: Response is not chunked. Header 'transfer-encoding: chunked' is missing.

Should I open a separate issue, or does it make sense to fix that in one go?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[bluenote10]

/remove-lifecycle stale

The issue still exists and is pretty big showstopper in using the streaming API.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[mitar]

/remove-lifecycle rotten

[mitar]

@bluenote10 I do not think bot recognized your command because of backticks.

[roycaihw]

/unassign

@jiahuif has a reproduce and WIP fix in kubernetes-client/python-base#174