lxd 3.18 breaks integration tests (on my machine) #379

ajkavanagh · 2019-10-21T10:12:19Z

$ ./integration/run-integration-tests-in-lxd 
...
..
.
+ lxc config set core.https_address '[::]'
+ [[ xenial == \b\i\o\n\i\c ]]
+ echo 'Run 16.04 (xenial) integration tests'
Run 16.04 (xenial) integration tests
+ tox -e integration
integration create: /opt/pylxd/.tox/integration
integration installdeps: -r/opt/pylxd/requirements.txt, -r/opt/pylxd/test-requirements.txt
integration develop-inst: /opt/pylxd
integration installed: attrs==19.3.0,certifi==2019.9.11,cffi==1.13.0,chardet==3.0.4,coverage==4.5.4,cryptography==2.8,ddt==1.2.1,entrypoints==0.3,flake8==3.7.8,funcsigs==1.0.2,idna==2.8,mccabe==0.6.1,mock==3.0.5,mock-services==0.3.1,nose==1.3.7,pbr==5.4.3,pkg-resources==0.0.0,pycodestyle==2.5.0,pycparser==2.19,pyflakes==2.1.1,-e git+git@github.com:lxc/pylxd.git@f79a8fe7934f7498fe9c14d45019002a1127e4e8#egg=pylxd,python-dateutil==2.8.0,requests==2.22.0,requests-mock==1.1.0,requests-toolbelt==0.9.1,requests-unixsocket==0.2.0,six==1.12.0,urllib3==1.25.6,ws4py==0.5.1
integration runtests: PYTHONHASHSEED='3195312664'
integration runtests: commands[0] | nosetests integration
.S.E....SSS...............SSSSSSS......SSSSSSSSSS
======================================================================
ERROR: A command is executed on the container.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/pylxd/integration/test_containers.py", line 152, in test_execute
    result = self.container.execute(['echo', 'test'])
  File "/opt/pylxd/pylxd/models/container.py", line 440, in execute
    manager.close_all()
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/manager.py", line 345, in close_all
    ws.close(code=code, reason=message)
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/client/__init__.py", line 205, in close
    self._write(self.stream.close(code=code, reason=reason).single(mask=True))
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/websocket.py", line 285, in _write
    self.sock.sendall(b)
BrokenPipeError: [Errno 32] Broken pipe

----------------------------------------------------------------------
Ran 49 tests in 39.257s

FAILED (SKIP=21, errors=1)
ERROR: InvocationError: '/opt/pylxd/.tox/integration/bin/nosetests integration'
__________________________________________________________ summary ___________________________________________________________
ERROR:   integration: commands failed

The text was updated successfully, but these errors were encountered:

ajkavanagh · 2020-04-12T13:39:50Z

Also, still broken with LXD 4.0.0:

======================================================================
ERROR: A command is executed on the container.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/pylxd/integration/test_containers.py", line 152, in test_execute
    result = self.container.execute(['echo', 'test'])
  File "/opt/pylxd/pylxd/models/container.py", line 440, in execute
    manager.close_all()
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/manager.py", line 345, in close_all
    ws.close(code=code, reason=message)
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/client/__init__.py", line 205, in close
    self._write(self.stream.close(code=code, reason=reason).single(mask=True))
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/websocket.py", line 285, in _write
    self.sock.sendall(b)
BrokenPipeError: [Errno 32] Broken pipe

----------------------------------------------------------------------
Ran 49 tests in 39.891s

FAILED (SKIP=21, errors=1)
ERROR: InvocationError: '/opt/pylxd/.tox/integration/bin/nosetests integration'
_____________________________________________________________________ summary ______________________________________________________________________
ERROR:   integration: commands failed
alex@alex-x99:~/Projects/Canonical/git/github.com/lxc/pylxd$ lxc version
Client version: 4.0.0
Server version: 4.0.0

ltrager · 2020-04-17T22:09:15Z

The integration tests only fail for me when testing Xenial. If I comment out Xenial and only test Bionic everything passes. Running the tests on Focal with LXD 4.0.0 installed in the Snap the following fail

+ echo 'Run 16.04 (xenial) integration tests'
Run 16.04 (xenial) integration tests
+ tox -e integration
integration create: /opt/pylxd/.tox/integration
integration installdeps: -r/opt/pylxd/requirements.txt, -r/opt/pylxd/test-requirements.txt
integration develop-inst: /opt/pylxd
integration installed: attrs==19.3.0,certifi==2020.4.5.1,cffi==1.14.0,chardet==3.0.4,coverage==5.1,cryptography==2.9,ddt==1.3.1,entrypoints==0.3,flake8==3.7.9,funcsigs==1.0.2,idna==2.9,mccabe==0.6.1,mock==3.0.5,mock-services==0.3.1,nose==1.3.7,pbr==5.4.5,pkg-resources==0.0.0,pycodestyle==2.5.0,pycparser==2.20,pyflakes==2.1.1,-e git+https://github.com/lxc/pylxd.git@691916fbc0092a9866f4a918d2c6bee046f9e718#egg=pylxd,python-dateutil==2.8.1,requests==2.23.0,requests-mock==1.1.0,requests-toolbelt==0.9.1,requests-unixsocket==0.2.0,six==1.14.0,urllib3==1.25.9,ws4py==0.5.1
integration runtests: PYTHONHASHSEED='559647950'
integration runtests: commands[0] | nosetests integration
.S.EE..ESSS...............SSSSSSS......SSSSSSSSSS
======================================================================
ERROR: A command is executed on the container.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/pylxd/integration/test_containers.py", line 152, in test_execute
    result = self.container.execute(['echo', 'test'])
  File "/opt/pylxd/pylxd/models/container.py", line 440, in execute
    manager.close_all()
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/manager.py", line 345, in close_all
    ws.close(code=code, reason=message)
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/client/__init__.py", line 205, in close
    self._write(self.stream.close(code=code, reason=reason).single(mask=True))
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/websocket.py", line 285, in _write
    self.sock.sendall(b)
BrokenPipeError: [Errno 32] Broken pipe

======================================================================
ERROR: A command is executed and force output to ascii
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/pylxd/integration/test_containers.py", line 191, in test_execute_force_decode
    encoding='latin1')
  File "/opt/pylxd/pylxd/models/container.py", line 440, in execute
    manager.close_all()
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/manager.py", line 345, in close_all
    ws.close(code=code, reason=message)
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/client/__init__.py", line 205, in close
    self._write(self.stream.close(code=code, reason=reason).single(mask=True))
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/websocket.py", line 285, in _write
    self.sock.sendall(b)
BrokenPipeError: [Errno 32] Broken pipe

======================================================================
ERROR: A command receives data from stdin and write to stdout handler
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/pylxd/integration/test_containers.py", line 208, in test_execute_pipes
    ['cat', '-'], stdin_payload=test_msg, stdout_handler=stdout_handler
  File "/opt/pylxd/pylxd/models/container.py", line 440, in execute
    manager.close_all()
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/manager.py", line 345, in close_all
    ws.close(code=code, reason=message)
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/client/__init__.py", line 205, in close
    self._write(self.stream.close(code=code, reason=reason).single(mask=True))
  File "/opt/pylxd/.tox/integration/lib/python3.5/site-packages/ws4py/websocket.py", line 285, in _write
    self.sock.sendall(b)
BrokenPipeError: [Errno 32] Broken pipe

----------------------------------------------------------------------
Ran 49 tests in 33.218s

ajkavanagh · 2020-04-23T13:50:44Z

So I did a git bisect on the master branch to see where the error was introduced, and the winner was ... me!

4c59b13409e22471bc0a96947a91d3b6b38b90f3 is the first bad commit
commit 4c59b13409e22471bc0a96947a91d3b6b38b90f3
Author: Alex Kavanagh <alex.kavanagh@canonical.com>
Date:   Fri May 17 16:00:00 2019 +0100

    Fix execute command missing output with pauses in output text
    
    For the container execute command, if the command executed has pauses in
    the output text, the websocket will send binary messages.  These were
    interpreted as the stream being completed, and pylxd closed the
    websocket, thus losing the rest of the output.
    
    Delving into the code indicated some very strange behaviour across the
    threads, so as part of solving the problem, this has been cleaned up as
    well.
    
    Closes: #362
    Signed-off-by: Alex Kavanagh <alex.kavanagh@canonical.com>

:040000 040000 ecb737fb114f36e3d7d45bbe2e71efd0ac71672e 7bb9580b5da98680d9bf527f93fb212e71add2a4 M	pylxd

I'll stare at it and see what might be wrong.

It appears that the manager.close_all() can cause a broken pipe error. At this stage, that means that the pipe is closed and thus it's actually okay to just continue and ignore the error (as per stdin) Fixes: canonical#379

It appears that the manager.close_all() can cause a broken pipe error. At this stage, that means that the pipe is closed and thus it's actually okay to just continue and ignore the error (as per stdin) Fixes: canonical#379 Signed-off-by: Alex Kavanagh <alex.kavanagh@canonical.com>

ajkavanagh mentioned this issue Oct 21, 2019

added image creation from public lxd server #378

Open

This was referenced Oct 28, 2019

Problem with container.execute() timeout/connection reset #376

Open

container.execute() not working #380

Closed

ltrager mentioned this issue Apr 18, 2020

Add support for the instances and virtual-machines endpoints. #387

Merged

ajkavanagh mentioned this issue Apr 23, 2020

Fix broken pipe error on xenial integration tests #394

Merged

ajkavanagh self-assigned this Apr 23, 2020

ajkavanagh added the bug label Apr 23, 2020

ChrisMacNaughton closed this as completed in #394 Apr 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lxd 3.18 breaks integration tests (on my machine) #379

lxd 3.18 breaks integration tests (on my machine) #379

ajkavanagh commented Oct 21, 2019

ajkavanagh commented Apr 12, 2020

ltrager commented Apr 17, 2020

ajkavanagh commented Apr 23, 2020

lxd 3.18 breaks integration tests (on my machine) #379

lxd 3.18 breaks integration tests (on my machine) #379

Comments

ajkavanagh commented Oct 21, 2019

ajkavanagh commented Apr 12, 2020

ltrager commented Apr 17, 2020

ajkavanagh commented Apr 23, 2020