Starting user server fails with 500 Internal Server Error after removing profileList

[nocnokneo]

Bug description

I started with a working multi-profile installation with a few profiles like:

      profileList:
      - display_name: "Standard Environment"
        default: true
      - display_name: "Power User Environment"
        kubespawner_override:
          mem_guarantee: "20G"
          mem_limit: "20G"

I then deleted the entire profileList (no longer want users to have the option of different environments). Now when I try to login and start a singleuser server I get "500 : Internal Server Error". I tested with another user account who had never previously logged in to the multi-profile deployment that worked.

From the logs (see below) it's clear that JupyterHub is remembering the standard-environment slug corresponding to the the first profile I had previously in the multi-profile configuration.

How to reproduce

See explanation above.

Expected behaviour

Expected that users who had logged in previously would still be able to start their singleuser server.

Your personal set up

Z2JH in AWS. Kubernetes version 1.28.

kubectl logs hub-...

Loading /usr/local/etc/jupyterhub/secret/values.yaml
No config at /usr/local/etc/jupyterhub/existing-secret/values.yaml
[I 2023-12-08 18:18:31.931 JupyterHub app:2859] Running JupyterHub version 4.0.2
[I 2023-12-08 18:18:31.931 JupyterHub app:2889] Using Authenticator: cytohub.authenticator.CytoTronicsOAuthenticator
[I 2023-12-08 18:18:31.932 JupyterHub app:2889] Using Spawner: kubespawner.spawner.KubeSpawner-6.2.0
[I 2023-12-08 18:18:31.932 JupyterHub app:2889] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.0.2
[I 2023-12-08 18:18:32.172 JupyterHub app:1984] Not using allowed_users. Any authenticated user will be allowed.
[I 2023-12-08 18:18:32.241 JupyterHub app:2928] Initialized 0 spawners in 0.003 seconds
[I 2023-12-08 18:18:32.248 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2023-12-08 18:18:32.250 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.seven_days
[I 2023-12-08 18:18:32.251 JupyterHub metrics:278] Found 1 active users in the last ActiveUserPeriods.thirty_days
[I 2023-12-08 18:18:32.251 JupyterHub app:3142] Not starting proxy
[I 2023-12-08 18:18:32.259 JupyterHub app:3178] Hub API listening on http://:8081/hub/
[I 2023-12-08 18:18:32.259 JupyterHub app:3180] Private Hub API connect url http://hub:8081/hub/
[I 2023-12-08 18:18:32.259 JupyterHub app:3189] Starting managed service jupyterhub-idle-culler
[I 2023-12-08 18:18:32.259 JupyterHub service:385] Starting service 'jupyterhub-idle-culler': ['python3', '-m', 'jupyterhub_idle_culler', '--url=http://localhost:8081/hub/api', '--timeout=18000', '--cull-every=300', '--concurrency=10']
[I 2023-12-08 18:18:32.261 JupyterHub service:133] Spawning python3 -m jupyterhub_idle_culler --url=http://localhost:8081/hub/api --timeout=18000 --cull-every=300 --concurrency=10
[I 2023-12-08 18:18:32.264 JupyterHub app:3247] JupyterHub is now running, internal Hub API at http://hub:8081/hub/
[I 2023-12-08 18:18:32.640 JupyterHub log:191] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 10.41ms
[I 2023-12-08 18:18:32.652 JupyterHub log:191] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 9.11ms
[I 2023-12-08 18:18:42.379 JupyterHub log:191] 200 GET /hub/home (taylor braun-jones@10.1.0.64) 65.63ms
[I 2023-12-08 18:18:43.642 JupyterHub provider:659] Creating oauth client jupyterhub-user-taylor%20braun-jones
[E 2023-12-08 18:18:43.668 JupyterHub user:884] Unhandled error starting taylor braun-jones's server: No such profile: standard-environment. Options include:
    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 798, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2667, in _start
        await self.load_user_options()
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 3327, in load_user_options
        self._validate_user_options(profile_list)
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 3132, in _validate_user_options
        profile = self._get_profile(profile_slug, profile_list)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 3176, in _get_profile
        raise ValueError(
    ValueError: No such profile: standard-environment. Options include: 
    
[I 2023-12-08 18:18:43.722 JupyterHub reflector:282] watching for pods with label selector='component=singleuser-server' in namespace computeapi
[E 2023-12-08 18:18:43.724 JupyterHub pages:313] Error starting server taylor braun-jones: No such profile: standard-environment. Options include:
    Traceback (most recent call last):
    NoneType: None
    
[W 2023-12-08 18:18:43.724 JupyterHub web:1869] 500 GET /hub/spawn/taylor%20braun-jones (10.1.0.64): Unhandled error starting server taylor braun-jones
[E 2023-12-08 18:18:43.754 JupyterHub log:183] {
      "Cookie": "_xsrf=[secret]; jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]",
      "Accept-Language": "en-US,en;q=0.9,fr-CH;q=0.8,fr;q=0.7,es-MX;q=0.6,es;q=0.5",
      "Accept-Encoding": "gzip, deflate, br",
      "Referer": "https://voilahub.test.cytotronics.io/hub/home",
      "Sec-Fetch-Dest": "document",
      "Sec-Fetch-User": "?1",
      "Sec-Fetch-Mode": "navigate",
      "Sec-Fetch-Site": "same-origin",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
      "Dnt": "1",
      "Upgrade-Insecure-Requests": "1",
      "Sec-Ch-Ua-Platform": "\"Linux\"",
      "Sec-Ch-Ua-Mobile": "?0",
      "Sec-Ch-Ua": "\"Not_A Brand\";v=\"8\", \"Chromium\";v=\"120\", \"Google Chrome\";v=\"120\"",
      "X-Scheme": "https",
      "X-Forwarded-Scheme": "https",
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Port": "443,80",
      "X-Forwarded-Host": "voilahub.test.cytotronics.io",
      "X-Forwarded-For": "10.1.0.64,::ffff:10.1.20.18",
      "X-Real-Ip": "10.1.0.64",
      "X-Request-Id": "178455466202853493bca02ad07e86d7",
      "Host": "voilahub.test.cytotronics.io",
      "Connection": "keep-alive"
    }
[E 2023-12-08 18:18:43.754 JupyterHub log:191] 500 GET /hub/spawn/taylor%20braun-jones (taylor braun-jones@10.1.0.64) 141.01ms

...

[I 2023-12-08 18:58:21.798 JupyterHub log:191] 200 POST /hub/api/users/labuser/activity (labuser@10.1.12.212) 11.40ms
[I 2023-12-08 18:58:32.690 JupyterHub log:191] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 8.89ms
[I 2023-12-08 18:58:32.695 JupyterHub log:191] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 4.62ms
[I 2023-12-08 19:03:23.666 JupyterHub log:191] 200 POST /hub/api/users/labuser/activity (labuser@10.1.12.212) 11.24ms
[I 2023-12-08 19:03:32.667 JupyterHub log:191] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 7.86ms
[I 2023-12-08 19:03:32.673 JupyterHub log:191] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 4.42ms
[I 2023-12-08 19:08:32.724 JupyterHub log:191] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 8.23ms
[I 2023-12-08 19:08:32.737 JupyterHub log:191] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 11.57ms
[I 2023-12-08 19:08:45.733 JupyterHub log:191] 200 POST /hub/api/users/labuser/activity (labuser@10.1.12.212) 9.49ms

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[nocnokneo]

Forgot to note that I've also tried profileList: [] and profileList: null with the same 500 error result and same log messages.

[nocnokneo]

Also tried restarting all the JupyterHub deployments - same result.

kubectl rollout restart deployment hub
kubectl rollout restart deployment user-scheduler
kubectl rollout restart deployment proxy

Exec'ing into the hub container confirms the user-specific leftover profile configuration {"profile": "standard-environment"}:

jovyan@hub-c5445f467-7vvpb:/srv/jupyterhub$ sqlite3 /srv/jupyterhub/jupyterhub.sqlite
SQLite version 3.34.1 2021-01-20 14:10:07
Enter ".help" for usage hints.
sqlite> SELECT * FROM spawners ;
1|1||{"pod_name": "jupyter-taylor-20braun-2djones", "namespace": "computeapi", "dns_name": "jupyter-taylor-20braun-2djones.computeapi.svc.cluster.local"}|||2023-12-08 18:55:21.032237|{"profile": "standard-environment"}|
2|5|1|{"pod_name": "jupyter-labuser", "namespace": "computeapi", "dns_name": "jupyter-labuser.computeapi.svc.cluster.local"}||2023-12-08 18:48:19.144022|2023-12-08 19:36:27.935000||jupyterhub-user-labuser
3|2|||||||
4|3|||||||
5|4|||||||

[nocnokneo]

Workaround is:

UPDATE spawners SET user_options = NULL;

then restart the hub deployment: kubectl rollout restart deployment hub