Slow and blocking startup of jupyter-ai server extension

[krassowski]

Description

jupyter-ai is the slowest extensions of the ones that I am shipping in various deployments. Even worse, it is blocking the jupyter-server from startup. It would be lovely if we could reduce the startup time.

Reproduce

1. Install jupyter-ai
2. Run jupyter server extension list to get detailed timings, on my laptop it is:

Package jupyter_ai took 2.9546s to import

as compared to all other extensions being ready instantaneously:

 Package jupyterlab took 0.0398s to import
 Package nbclassic took 0.0014s to import
 Package jupyter_server_fileid took 0.0000s to import
 Package jupyter_server_ydoc took 0.0655s to import
 Package jupyter_lsp took 0.0290s to import

3. Run jupyter lab or jupyter server
4. See that jupyter-ai blocks jupyter-server startup for 2-3 seconds
5. See that it prints lots of messages about dependencies being not installed even when we block other providers (so it needlessly attempts some imports)

Expected behavior

1. Async startup sequence is used:
   • either using a bespoke solution as in jupyter-lsp (see Use app's HTTP settings, handle feature settings race condition, CI/build updates, lazy server discovery jupyter-lsp/jupyterlab-lsp#882)
   • using new jupyter-server async hook once merged Add async start hook to ExtensionApp API jupyter-server/jupyter_server#1417
2. No imports are performed for blocked providers

Context

• Operating System and version: Ubuntu
• JupyterLab version: 4.3.1