Inconsistent exceptions when adding empty byte strings

[machawk1]

I am encountering an issue when adding strings of 0 length using add_bytes(bytes('')) (in oduwsdl/ipwb#485).

Replication:

> import ipfsapi
> IPFS_API = ipfsapi.Client('localhost', 5001)
> IPFS_API.add_bytes('lorem')
u'QmeHxdVgh5BVLL7S7AvjrXqum8977XcPcfyHYUkoCG624N'
> IPFS_API.add_bytes('')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/ipfsapi/utils.py", line 150, in wrapper
    res = cmd(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/ipfsapi/client.py", line 2087, in add_bytes
    data=body, headers=headers, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/ipfsapi/http.py", line 37, in wrapper
    return func(self, *args, **merged)
  File "/usr/local/lib/python2.7/site-packages/ipfsapi/http.py", line 170, in request
    files, headers, data)
  File "/usr/local/lib/python2.7/site-packages/ipfsapi/http.py", line 100, in _request
    files=files, headers=headers, data=data)
  File "/usr/local/lib/python2.7/site-packages/ipfsapi/http.py", line 77, in _do_request
    six.raise_from(exceptions.ConnectionError(error), error)
  File "/Users/machawk1/Library/Python/2.7/lib/python/site-packages/six.py", line 737, in raise_from
    raise value
ipfsapi.exceptions.ConnectionError: ConnectionError: [Errno 32] Broken pipe
> IPFS_API.add_bytes(bytes(''))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/ipfsapi/utils.py", line 150, in wrapper
    res = cmd(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/ipfsapi/client.py", line 2087, in add_bytes
    data=body, headers=headers, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/ipfsapi/http.py", line 37, in wrapper
    return func(self, *args, **merged)
  File "/usr/local/lib/python2.7/site-packages/ipfsapi/http.py", line 170, in request
    files, headers, data)
  File "/usr/local/lib/python2.7/site-packages/ipfsapi/http.py", line 100, in _request
    files=files, headers=headers, data=data)
  File "/usr/local/lib/python2.7/site-packages/ipfsapi/http.py", line 77, in _do_request
    six.raise_from(exceptions.ConnectionError(error), error)
  File "/Users/machawk1/Library/Python/2.7/lib/python/site-packages/six.py", line 737, in raise_from
    raise value
ipfsapi.exceptions.ConnectionError: ConnectionError: [Errno 32] Broken pipe
>>> 

...but sometimes from this same line with the same empty string I get:

> IPFS_API.add_bytes(bytes(''))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/ipfsapi/utils.py", line 151, in wrapper
    return res[self.field]
TypeError: list indices must be integers, not str

Is it in violation to add an empty string "as bytes" to IPFS using add_bytes()? Any clue why I would be getting multiple errors from the same line (replicable in interactive Python, too)?

ipfsapi 0.4.3 (originally experienced in 0.4.2a)

[ibnesayeed]

I can confirm that in Python 3.7.0 attempting to store an empty bytearray throws a TypeError exception.

$ ipfs daemon --init &
$ python
Python 3.7.0 (default, Aug  4 2018, 02:33:39) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ipfsapi
>>> api = ipfsapi.connect('127.0.0.1', 5001)
>>> api.add_bytes(b'foo')
'QmcJw6x4bQr7oFnVnF6i8SLcJvhXjaxWvj54FYXmZ4Ct6p'
>>> api.add_bytes(b'')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/ipfsapi/utils.py", line 151, in wrapper
    return res[self.field]
TypeError: list indices must be integers or slices, not str

[machawk1]

Thanks for the sanity check @ibnesayeed, as I thought that maybe Py2 was the scapegoat. Are you able to see ipfsapi.exceptions.ConnectionError: ConnectionError: [Errno 32] Broken pipe? I ran the same command (api.add_bytes(b'')) multiple consecutive times and sometimes got the broken pipe exception and other times the TypeError you confirmed.

[ibnesayeed]

No, I never got ConnectionError, though I tried it a few times in succession, it has always been TypeError for me.

[ntninja]

Thanks for the feedback everybody! Turns out the root cause of this issue is likely the same as that of #104: Our broken HTTP multipart streaming backend.

I come to this conclusion, because I found that the failing function received the following result from the deamon:

[
  {'Name': 'bytes', 'Hash': 'QmaRwA91m9Rdfaq9u3FH1fdMVxw1wFPjKL38czkWMxh3KB', 'Size': '8'},
  {'Message': 'multipart: NextPart: EOF', 'Code': 0, 'Type': 'error'}
]

However the function only expects:

{'Name': 'bytes', 'Hash': 'QmaRwA91m9Rdfaq9u3FH1fdMVxw1wFPjKL38czkWMxh3KB', 'Size': '8'}

from which it would then take the string-index 'Hash', explaining the weird error message. (The reason that there is no array here is because array results with exactly one item are automatically unpacked – a behaviour we're stuck with for the time being due to back-compat issues.)

The connection error is likely a result of the HTTP connection ending up in an undefined state after having received broken multipart data (the second message).

[ntninja]

Turns out it's actually a bug with go-IPFS: ipfs/kubo#5168
I'll be pushing a mitigation soon.

[ntninja]

I don't get this error anymore with go-IPFS v0.4.18 – can somebody confirm that this is fixed?

[ibnesayeed]

I can confirm that it is working fine with go-IPFS v0.4.18. This ticket can perhaps be closed now.

$ ipfs --version
ipfs version 0.4.18
$ python
Python 2.7.15 (default, Oct 16 2018, 07:58:03) 
[GCC 6.3.0 20170516] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ipfsapi
>>> api = ipfsapi.connect('127.0.0.1', 5001)
>>> api.add_bytes(b'')
u'QmaRwA91m9Rdfaq9u3FH1fdMVxw1wFPjKL38czkWMxh3KB'
>>> api.cat('QmaRwA91m9Rdfaq9u3FH1fdMVxw1wFPjKL38czkWMxh3KB')
[]
>>> exit()
$ 

[ibnesayeed]

Tested in both Python 2 and 3 with success.

[ntninja]

Thank you! 👍