Dask_cudf failure: ValueError: Cell is empty

[pseudotensor]

Same specs and dataset as here: #6232

Script:

def fun():
    import dill

    from dask.distributed import Client
    from dask_cuda import LocalCUDACluster
    import dask_cudf
    import xgboost as xgb


    with LocalCUDACluster() as cluster:
        with Client(cluster) as client:
            X_dask_path = "creditcard.csv"
            target = "default payment next month"
            X = dask_cudf.read_csv(X_dask_path)
            y = X[target]
            X.drop(target, axis=1)

            params = {'tree_method': 'gpu_hist'}
            dask_model = xgb.dask.DaskXGBClassifier(**params)
            dask_model.client = client
            dask_model.fit(X, y)

if __name__ == '__main__':
    fun()

Leads to:

/home/jon/minicondadai/lib/python3.6/site-packages/distributed/client.py:3479: RuntimeWarning: coroutine 'Client._update_scheduler_info' was never awaited
  self.sync(self._update_scheduler_info)
Traceback (most recent call last):
  File "/home/jon/minicondadai/lib/python3.6/site-packages/distributed/worker.py", line 3316, in dumps_function
    result = cache_dumps[func]
  File "/home/jon/minicondadai/lib/python3.6/site-packages/distributed/utils.py", line 1547, in __getitem__
    value = super().__getitem__(key)
  File "/home/jon/minicondadai/lib/python3.6/collections/__init__.py", line 991, in __getitem__
    raise KeyError(key)
KeyError: <function _train_async.<locals>.dispatched_train at 0x7f9c0a5a2e18>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jon/minicondadai/lib/python3.6/site-packages/distributed/protocol/pickle.py", line 40, in dumps
    result = pickle.dumps(x, **dump_kwargs)
AttributeError: Can't pickle local object '_train_async.<locals>.dispatched_train'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "check_dask_cudf2.py", line 24, in <module>
    fun()
  File "check_dask_cudf2.py", line 21, in fun
    dask_model.fit(X, y)
  File "/home/jon/minicondadai/lib/python3.6/site-packages/xgboost/dask.py", line 1080, in fit
    eval_set, sample_weight_eval_set, verbose)
  File "/home/jon/minicondadai/lib/python3.6/site-packages/distributed/client.py", line 824, in sync
    self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
  File "/home/jon/minicondadai/lib/python3.6/site-packages/distributed/utils.py", line 339, in sync
    raise exc.with_traceback(tb)
  File "/home/jon/minicondadai/lib/python3.6/site-packages/distributed/utils.py", line 323, in f
    result[0] = yield future
  File "/home/jon/minicondadai/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/home/jon/minicondadai/lib/python3.6/site-packages/xgboost/dask.py", line 1067, in _fit_async
    evals=evals, verbose_eval=verbose)
  File "/home/jon/minicondadai/lib/python3.6/site-packages/xgboost/dask.py", line 632, in _train_async
    workers=workers)
  File "/home/jon/minicondadai/lib/python3.6/site-packages/distributed/client.py", line 1779, in map
    actors=actor,
  File "/home/jon/minicondadai/lib/python3.6/site-packages/distributed/client.py", line 2590, in _graph_to_futures
    "tasks": valmap(dumps_task, dsk3),
  File "cytoolz/dicttoolz.pyx", line 181, in cytoolz.dicttoolz.valmap
  File "cytoolz/dicttoolz.pyx", line 206, in cytoolz.dicttoolz.valmap
  File "/home/jon/minicondadai/lib/python3.6/site-packages/distributed/worker.py", line 3354, in dumps_task
    return {"function": dumps_function(task[0]), "args": warn_dumps(task[1:])}
  File "/home/jon/minicondadai/lib/python3.6/site-packages/distributed/worker.py", line 3318, in dumps_function
    result = pickle.dumps(func)
  File "/home/jon/minicondadai/lib/python3.6/site-packages/distributed/protocol/pickle.py", line 51, in dumps
    result = cloudpickle.dumps(x, **dump_kwargs)
  File "/home/jon/minicondadai/lib/python3.6/site-packages/cloudpickle/cloudpickle_fast.py", line 102, in dumps
    cp.dump(obj)
  File "/home/jon/minicondadai/lib/python3.6/site-packages/cloudpickle/cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
  File "/home/jon/minicondadai/lib/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/home/jon/minicondadai/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/jon/minicondadai/lib/python3.6/site-packages/cloudpickle/cloudpickle_fast.py", line 745, in save_function
    *self._dynamic_function_reduce(obj), obj=obj
  File "/home/jon/minicondadai/lib/python3.6/site-packages/cloudpickle/cloudpickle_fast.py", line 682, in _save_reduce_pickle5
    dictitems=dictitems, obj=obj
  File "/home/jon/minicondadai/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/home/jon/minicondadai/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/jon/minicondadai/lib/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/home/jon/minicondadai/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/jon/minicondadai/lib/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/home/jon/minicondadai/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/jon/minicondadai/lib/python3.6/site-packages/dill/_dill.py", line 1169, in save_cell
    f = obj.cell_contents
ValueError: Cell is empty

This only happens if one has already imported dill. However, this is fairly critical since any use of dill outside xgboost would be normal.

This may be more of a dask issue, but I didn't try to reproduce in dask itself. I did not find anything like this problem on github for dask or xgboost or in google.

However, I'm cross-posting to: dask/dask#6737

[hcho3]

@pseudotensor Thanks for taking time to test XGBoost and file bug reports. Well-written reports with reproducible examples are highly valuable to us. We will try to reproduce the bug and get back to you.

[trivialfis]

Looks like a dill issue to me. ;-) Not sure what's it doing under the hood.

[jsignell]

Looks like it could be this one: uqfoundation/dill#383

[trivialfis]

@jsignell Should have found the correct issue. Will close this one if no objection.