Test failure in runtest.py SerializationTest.testPassingArgumentsByValue.

[robertnishihara]

I just saw this test failure, which caused runtest.py to hang.

Cloudpickle just pushed a new release 0.4.0, so it's probably related to that.

testPassingArgumentsByValue (__main__.SerializationTest) ... 15979:M 10 Aug 04:33:05.891 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
15979:M 10 Aug 04:33:05.891 # Server started, Redis version 3.9.102
15979:M 10 Aug 04:33:05.891 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
15979:M 10 Aug 04:33:05.891 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
Waiting for redis server at 127.0.0.1:17525 to respond...
15983:M 10 Aug 04:33:05.999 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
15983:M 10 Aug 04:33:05.999 # Server started, Redis version 3.9.102
15983:M 10 Aug 04:33:05.999 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
15983:M 10 Aug 04:33:05.999 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
Waiting for redis server at 127.0.0.1:55522 to respond...
Allowing the Plasma store to use up to 3.13729GB of memory.
Starting local scheduler with 2 CPUs, 0 GPUs
[INFO] (/home/travis/build/ray-project/ray/src/local_scheduler/local_scheduler.cc:226) Started worker with pid 15997
Failed to start the UI, you may need to run 'pip install jupyter'.
[INFO] (/home/travis/build/ray-project/ray/src/local_scheduler/local_scheduler.cc:226) Started worker with pid 16004
WARNING: Serializing objects of type <type 'exceptions.Exception'> by expanding them as dictionaries of their fields. This behavior may be incorrect in some cases.
WARNING: Serializing objects of type <class '__main__.CustomError'> by expanding them as dictionaries of their fields. This behavior may be incorrect in some cases.
WARNING: Serializing objects of type <class '__main__.Point'> by expanding them as dictionaries of their fields. This behavior may be incorrect in some cases.
Traceback (most recent call last):
  File "/home/travis/.local/lib/python2.7/site-packages/ray-0.1.2-py2.7-linux-x86_64.egg/ray/worker.py", line 1504, in fetch_and_execute_function_to_run
    function = pickle.loads(serialized_function)
  File "/home/travis/miniconda/lib/python2.7/pickle.py", line 1388, in loads
    return Unpickler(file).load()
  File "/home/travis/miniconda/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/home/travis/miniconda/lib/python2.7/pickle.py", line 1139, in load_reduce
    value = func(*args)
  File "/home/travis/miniconda/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 1053, in _rehydrate_skeleton_class
    setattr(skeleton_class, attrname, attr)
AttributeError: attribute '__dict__' of 'type' objects is not writable


  You can inspect errors by running

      ray.error_info()

  If this driver is hanging, start a new one with

      ray.init(redis_address="127.0.0.1:17525")
  
Traceback (most recent call last):
  File "/home/travis/.local/lib/python2.7/site-packages/ray-0.1.2-py2.7-linux-x86_64.egg/ray/worker.py", line 1504, in fetch_and_execute_function_to_run
    function = pickle.loads(serialized_function)
  File "/home/travis/miniconda/lib/python2.7/pickle.py", line 1388, in loads
    return Unpickler(file).load()
  File "/home/travis/miniconda/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/home/travis/miniconda/lib/python2.7/pickle.py", line 1139, in load_reduce
    value = func(*args)
  File "/home/travis/miniconda/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 1053, in _rehydrate_skeleton_class
    setattr(skeleton_class, attrname, attr)
AttributeError: attribute '__dict__' of 'type' objects is not writable


  You can inspect errors by running

      ray.error_info()

  If this driver is hanging, start a new one with

      ray.init(redis_address="127.0.0.1:17525")
  
Traceback (most recent call last):
  File "/home/travis/.local/lib/python2.7/site-packages/ray-0.1.2-py2.7-linux-x86_64.egg/ray/worker.py", line 1504, in fetch_and_execute_function_to_run
    function = pickle.loads(serialized_function)
  File "/home/travis/miniconda/lib/python2.7/pickle.py", line 1388, in loads
    return Unpickler(file).load()
  File "/home/travis/miniconda/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/home/travis/miniconda/lib/python2.7/pickle.py", line 1139, in load_reduce
    value = func(*args)
  File "/home/travis/miniconda/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 1053, in _rehydrate_skeleton_class
    setattr(skeleton_class, attrname, attr)
AttributeError: attribute '__dict__' of 'type' objects is not writable


  You can inspect errors by running

      ray.error_info()

  If this driver is hanging, start a new one with

      ray.init(redis_address="127.0.0.1:17525")
  
This worker or driver is waiting to receive a class definition so that it can deserialize an object from the object store. This may be fine, or it may be a bug.

  You can inspect errors by running

      ray.error_info()

  If this driver is hanging, start a new one with

      ray.init(redis_address="127.0.0.1:17525")
  


No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

The build has been terminated

The full log is at https://s3.amazonaws.com/archive.travis-ci.org/jobs/262938813/log.txt?X-Amz-Expires=30&X-Amz-Date=20170810T055744Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJRYRXRSVGNKPKO5A/20170810/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=d8d5b253085265e40f0f253ad2a74cb5642433975100730fad6001e523f77181

[robertnishihara]

I can reproduce this on Ubuntu 16 with Anaconda Python 2.7 by running the following.

python test/runtest.py SerializationTest.testPassingArgumentsByValue

[robertnishihara]

The problem can be reproduced more easily (without Ray) as follows.

from collections import namedtuple
import cloudpickle

Point = namedtuple("Point", ["x", "y"])

s = cloudpickle.dumps(Point)
cloudpickle.loads(s)

This gives the error

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-c865fee50a64> in <module>()
      5 
      6 s = cloudpickle.dumps(Point)
----> 7 cloudpickle.loads(s)

/home/ubuntu/anaconda2/lib/python2.7/pickle.pyc in loads(str)
   1386 def loads(str):
   1387     file = StringIO(str)
-> 1388     return Unpickler(file).load()
   1389 
   1390 # Doctest

/home/ubuntu/anaconda2/lib/python2.7/pickle.pyc in load(self)
    862             while 1:
    863                 key = read(1)
--> 864                 dispatch[key](self)
    865         except _Stop, stopinst:
    866             return stopinst.value

/home/ubuntu/anaconda2/lib/python2.7/pickle.pyc in load_reduce(self)
   1137         args = stack.pop()
   1138         func = stack[-1]
-> 1139         value = func(*args)
   1140         stack[-1] = value
   1141     dispatch[REDUCE] = load_reduce

/home/ubuntu/anaconda2/lib/python2.7/site-packages/cloudpickle/cloudpickle.pyc in _rehydrate_skeleton_class(skeleton_class, class_dict)
   1051     """
   1052     for attrname, attr in class_dict.items():
-> 1053         setattr(skeleton_class, attrname, attr)
   1054     return skeleton_class
   1055 

AttributeError: attribute '__dict__' of 'type' objects is not writable

This is a cloudpickle regression being tracked in cloudpipe/cloudpickle#111.

[robertnishihara]

#834 pins cloudpickle to 0.3.0 in the Travis tests, which should fix the test failures.

[robertnishihara]

Closing for now, since the test failure was fixed by #834, though people with the latest cloudpickle will still run into this problem.