Cannot configure ports if 'transport' is 'tcp'

[yves-surrel]

Changing the shell_port, iopub_port etc. in jupyter_console_config.py does not work, neither does using jupyter console --iopub:xxxx in the command line.

After investigation, it comes from the cache_ports attribute of KernelManager:

    def _default_cache_ports(self) -> bool:
        return self.transport == "tcp"

in `manager.py' and

            if km.cache_ports and not self.ports_cached:
                lpc = LocalPortCache.instance()
                km.shell_port = lpc.find_available_port(km.ip)
                km.iopub_port = lpc.find_available_port(km.ip)
                km.stdin_port = lpc.find_available_port(km.ip)
                km.hb_port = lpc.find_available_port(km.ip)
                km.control_port = lpc.find_available_port(km.ip)
                self.ports_cached = True

in async def pre_launch in local_provisioner.py.

Maybe related to #492

Jupiter_client v 7.3.5

[kevin-bates]

Hi @yves-surrel - it looks like you've identified a possible issue. I'm not familiar with jupyter console but see the consoleapp.py file in the repo - although there is no corresponding "console-script" in the set of scripts in pyproject.toml (or has there been in the past), so I'm not sure where jupyter console is coming from. Could you clarify how that application is installed/exposed?

I'm curious, assuming the KernelManager can be configured in this application (as looks to be the case in consoleapp.py), whether you can disable the port caching on the KernelManager prior to your invocation, and whether or not that moves you forward. Perhaps try on the CLI: --KernelManager.cache_ports = False or in the file: c.KernelManager.cache_ports = False? Thanks.

cc: @martinRenou for possible insights

[yves-surrel]

Setting --KernelManager.cache_ports = False does not change anything, and it's the same behavior in jupyter qtconsole, which is no surprise as cache_ports is not Config-able, as stated in #492. BTW, have you an idea why this 'port caching' mechanism does exist for transport = 'tcp'?

[kevin-bates]

@yves-surrel - sorry about that, didn't realize that cache_ports was not configurable and see that I had that very question on #492 that was never answered. 😄 To test this hypothesis, you'd need to modify the boolean default.

BTW, have you an idea why this 'port caching' mechanism does exist for transport = 'tcp'?

Yes. There's an inherent race condition (primarily in local kernels) between the time that the ports are determined and actually used by the kernel (process) in which another application can use any of the ports, which leads to "port in use" failures starting the kernel.

The cache_ports functionality essentially removes any jupyter_client-based application from the set of applications that can inject themselves into this race condition since those applications are "good citizens" by first recording their ports in a "reservation" cache.

Until the kernel determines the ports and conveys that back to the server, this issue will exist.

(Note that some remote kernels have a "launcher" or "nanny" process that creates the ports and sends that information back to the launching server, thereby decreasing the window in which the race the condition can occur from seconds to milliseconds.)

---

Regarding this "jupyter console" application, could you please describe how it's installed/configured? I figured it was the consoleapp.py, but I suspect that's not the case.

[blink1073]

The console app/entry point is provided by https://github.com/jupyter/jupyter_console

[yves-surrel]

you'd need to modify the boolean default

Actually, I replaced

def _default_cache_ports(self) -> bool:
        return self.transport == "tcp"

by

def _default_cache_ports(self) -> bool:
        return self.transport == "tcp" and self.shell_port != 60000

in order not to break anything else and to be able to configure ports (starting with shell_port=60000 of course). Like that, it works, but it's dirty, of course. The simplest would be to have config=True in the definition of cache_ports, as you suggested.

[kevin-bates]

Thanks for the jupyter_console tip @blink1073.

Looking at jupyter_console, and given that it derives from JupyterConsoleApp, it does seem like making KernelManager.cache_ports configurable is the way forward here.

@martinRenou or @SylvainCorlay - is there a reason cache_ports should not be configurable (with a default value of True)?