xgboost_synthetic: AttributeError: module 'tornado.ioloop' has no attribute '_Selectable'

[jlewi]

I just tried running xgboost_synthetic example @ 628babc

and I get the following error

AttributeError                            Traceback (most recent call last)
<ipython-input-7-25daace4167a> in <module>
      6 import kfp.compiler as compiler
      7 from kubernetes import client as k8s_client
----> 8 from kubeflow import fairing
      9 from kubeflow.fairing.builders import append
     10 from kubeflow.fairing.deployers import job

/opt/conda/lib/python3.6/site-packages/kubeflow/fairing/__init__.py in <module>
      1 import os
----> 2 from .ml_tasks.tasks import TrainJob, PredictionEndpoint
      3 
      4 if os.getenv('FAIRING_RUNTIME', None) is not None:
      5     from .runtime_config import config

/opt/conda/lib/python3.6/site-packages/kubeflow/fairing/ml_tasks/tasks.py in <module>
      2 import json
      3 import numpy as np
----> 4 from ..backends import KubernetesBackend
      5 from .utils import guess_preprocessor
      6 

/opt/conda/lib/python3.6/site-packages/kubeflow/fairing/backends/__init__.py in <module>
----> 1 from .backends import *

/opt/conda/lib/python3.6/site-packages/kubeflow/fairing/backends/backends.py in <module>
     18 from ..cloud import gcp
     19 from ..cloud import docker
---> 20 from ..ml_tasks import utils as ml_tasks_utils
     21 from ..constants import constants
     22 from ..kubernetes.manager import KubeManager

/opt/conda/lib/python3.6/site-packages/kubeflow/fairing/ml_tasks/utils.py in <module>
      2 from docker.errors import DockerException
      3 from ..functions.function_shim import get_execution_obj_type, ObjectType
----> 4 from ..preprocessors.function import FunctionPreProcessor
      5 from ..preprocessors.base import BasePreProcessor
      6 from ..preprocessors.full_notebook import FullNotebookPreProcessor

/opt/conda/lib/python3.6/site-packages/kubeflow/fairing/preprocessors/function.py in <module>
      9 from .base import BasePreProcessor
     10 from ..functions.function_shim import get_execution_obj_type, ObjectType
---> 11 from ..notebook import notebook_util
     12 
     13 logger = logging.getLogger(__name__)

/opt/conda/lib/python3.6/site-packages/kubeflow/fairing/notebook/notebook_util.py in <module>
      3 import ipykernel
      4 import re
----> 5 from notebook.notebookapp import list_running_servers
      6 import requests
      7 from requests.compat import urljoin

/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py in <module>
     60     raise ImportError(_("The Jupyter Notebook requires tornado >= 4.0, but you have %s") % tornado.version)
     61 
---> 62 from tornado import httpserver
     63 from tornado import web
     64 from tornado.httputil import url_concat

/opt/conda/lib/python3.6/site-packages/tornado/httpserver.py in <module>
     30 
     31 from tornado.escape import native_str
---> 32 from tornado.http1connection import HTTP1ServerConnection, HTTP1ConnectionParameters
     33 from tornado import httputil
     34 from tornado import iostream

/opt/conda/lib/python3.6/site-packages/tornado/http1connection.py in <module>
     32 from tornado import gen
     33 from tornado import httputil
---> 34 from tornado import iostream
     35 from tornado.log import gen_log, app_log
     36 from tornado.util import GzipDecompressor

/opt/conda/lib/python3.6/site-packages/tornado/iostream.py in <module>
    227 
    228 
--> 229 class BaseIOStream(object):
    230     """A utility class to write to and read from a non-blocking file or socket.
    231 

/opt/conda/lib/python3.6/site-packages/tornado/iostream.py in BaseIOStream()
    303         self._closed = False
    304 
--> 305     def fileno(self) -> Union[int, ioloop._Selectable]:
    306         """Returns the file descriptor for this stream."""
    307         raise NotImplementedError()

AttributeError: module 'tornado.ioloop' has no attribute '_Selectable'

[issue-label-bot]

Issue-Label Bot is automatically applying the label kind/bug to this issue, with a confidence of 0.95. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[jlewi]

Looks related:
jupyter/notebook#3407

[jlewi]

I think I have pyzmq 18 installed

Requirement already satisfied: pyzmq in /opt/conda/lib/python3.6/site-packages (18.0.1)

[jlewi]

When I restarted the kernel the imports seemed to work.
So maybe it was just an issue of restarting the kernel after running notebook_setup which installs a bunch of pip packages.

[jinchihe]

@jlewi That's caused by the tornado old version. I already added in the tornado>=6.0.3 before:

examples/xgboost_synthetic/requirements.txt

Line 12 in 628babc

tornado>=6.0.3

The requirements.txt is installed in util.notebook_setup() while executing the notebook. I hit that while running automation, so installed the requirements.txt before starting the notebook in CI/CD.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jtfogarty]

/area example
/priority p2

[k8s-ci-robot]

@jtfogarty: The label(s) area/jupyter cannot be applied, because the repository doesn't have them

In response to this:

/area jupyter
/priority p2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jlewi]

Should be fixed and we have E2E testing now.