Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spawners crash when deploying after cluster reboot #196

Open
mkossatz opened this issue May 12, 2021 · 2 comments
Open

Spawners crash when deploying after cluster reboot #196

mkossatz opened this issue May 12, 2021 · 2 comments

Comments

@mkossatz
Copy link

We set up this workshop via RHPDS for a customer, then put the cluster to sleep so when can start it for the meeting the next day. Upon restarting the cluster, the Homeroom was available but upon clicking the individual language-specific lab links, these crashed. We manually re-created the project to fix it.

Having restarted it a second time so that the customer can continue to work on it, most labs worked and just the Java one didn't. We didn't look further into it the first time it happened and the details below are specific to the Java lab crashing upon cluster restart but we assume it's the same problem.

This is the Pod that crashes:
lab-getting-started-java-spawner-2-6jfk4

Logs:

+ . /opt/app-root/src/jupyterhub_config.sh
++ CONFIGURATION_TYPE=terminal-server
++ '[' -f /opt/app-root/src/configs/terminal-server.sh ']'
++ . /opt/app-root/src/configs/terminal-server.sh
+++ KUBERNETES_SERVER_URL=https://172.30.0.1:443
+++ OAUTH_METADATA_URL=https://172.30.0.1:443/.well-known/oauth-authorization-server
++++ curl -ks https://172.30.0.1:443/.well-known/oauth-authorization-server
++++ jq -r .issuer
+++ OAUTH_ISSUER_ADDRESS=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ export OPENSHIFT_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ OPENSHIFT_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ export OPENSHIFT_REST_API_URL=https://172.30.0.1:443
+++ OPENSHIFT_REST_API_URL=https://172.30.0.1:443
+++ export OPENSHIFT_AUTH_API_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ OPENSHIFT_AUTH_API_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
++ '[' -f /opt/app-root/configs/jupyterhub_config.sh ']'
++ . /opt/app-root/configs/jupyterhub_config.sh
+ trap 'kill -TERM $PID' TERM INT
+ PID=32
+ wait 32
+ jupyterhub -f /opt/app-root/src/jupyterhub_config.py
INFO: Workshop name is 'lab-getting-started-java'.
INFO: Application name is 'lab-getting-started-java'.
INFO: Configuration type is 'terminal-server'.
INFO: Homeroom link is ''.
INFO: Homeroom name is 'homeroom'.
INFO: Service account name is 'lab-getting-started-java-spawner'.
INFO: Namespace is 'lab'.
INFO: Full service account name is 'system:serviceaccount:lab:lab-getting-started-java-spawner'.
/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.30.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
INFO: Homeroom link set to http://homeroom-lab.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com.
[I 2021-05-12 10:49:19.642 JupyterHub app:2120] Using Authenticator: oauthenticator.openshift.OpenShiftOAuthenticator-0.9.0
[I 2021-05-12 10:49:19.643 JupyterHub app:2120] Using Spawner: kubespawner.spawner.KubeSpawner
[I 2021-05-12 10:49:19.644 JupyterHub app:1257] Loading cookie_secret from /opt/app-root/data/cookie_secret
[I 2021-05-12 10:49:19.660 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2021-05-12 10:49:19.662 JupyterHub app:1532] No admin users, admin interface will be unavailable.
[W 2021-05-12 10:49:19.662 JupyterHub app:1534] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-05-12 10:49:19.662 JupyterHub app:1563] Not using whitelist. Any authenticated user will be allowed.
[I 2021-05-12 10:49:19.729 JupyterHub reflector:199] watching for pods with label selector='component=singleuser-server' in namespace lab
[I 2021-05-12 10:49:19.788 JupyterHub reflector:199] watching for events with field selector='involvedObject.kind=Pod' in namespace lab
[W 2021-05-12 10:49:22.852 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:25.924 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:28.996 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:32.068 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:35.140 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:38.212 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:41.284 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
++ kill -TERM 32
+ trap - TERM INT
+ wait 32
/opt/app-root/etc/run.sh: line 18:    32 Terminated              jupyterhub -f /opt/app-root/src/jupyterhub_config.py
+ STATUS=143
+ exit 143
@mkossatz mkossatz changed the title Spawners crash when starting after cluster reboot Spawners crash when deploying after cluster reboot May 12, 2021
@mkossatz
Copy link
Author

mkossatz commented May 14, 2021 •
edited
Loading

We have that problem continuously after cluster reboots. This is now always with the java lab spawner. Oddly, the log lacks the failed connection errors that I thought were the reason for the crash.

+ . /opt/app-root/src/jupyterhub_config.sh
++ CONFIGURATION_TYPE=terminal-server
++ '[' -f /opt/app-root/src/configs/terminal-server.sh ']'
++ . /opt/app-root/src/configs/terminal-server.sh
+++ KUBERNETES_SERVER_URL=https://172.30.0.1:443
+++ OAUTH_METADATA_URL=https://172.30.0.1:443/.well-known/oauth-authorization-server
++++ curl -ks https://172.30.0.1:443/.well-known/oauth-authorization-server
++++ jq -r .issuer
+++ OAUTH_ISSUER_ADDRESS=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ export OPENSHIFT_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ OPENSHIFT_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ export OPENSHIFT_REST_API_URL=https://172.30.0.1:443
+++ OPENSHIFT_REST_API_URL=https://172.30.0.1:443
+++ export OPENSHIFT_AUTH_API_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ OPENSHIFT_AUTH_API_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
++ '[' -f /opt/app-root/configs/jupyterhub_config.sh ']'
++ . /opt/app-root/configs/jupyterhub_config.sh
+ trap 'kill -TERM $PID' TERM INT
+ PID=32
+ wait 32
+ jupyterhub -f /opt/app-root/src/jupyterhub_config.py
INFO: Workshop name is 'lab-getting-started-java'.
INFO: Application name is 'lab-getting-started-java'.
INFO: Configuration type is 'terminal-server'.
INFO: Homeroom link is ''.
INFO: Homeroom name is 'homeroom'.
INFO: Service account name is 'lab-getting-started-java-spawner'.
INFO: Namespace is 'lab'.
INFO: Full service account name is 'system:serviceaccount:lab:lab-getting-started-java-spawner'.
/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.30.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
INFO: Homeroom link set to http://homeroom-lab.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com.
[I 2021-05-14 08:07:44.204 JupyterHub app:2120] Using Authenticator: oauthenticator.openshift.OpenShiftOAuthenticator-0.9.0
[I 2021-05-14 08:07:44.204 JupyterHub app:2120] Using Spawner: kubespawner.spawner.KubeSpawner
[I 2021-05-14 08:07:44.205 JupyterHub app:1257] Loading cookie_secret from /opt/app-root/data/cookie_secret
[I 2021-05-14 08:07:44.222 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2021-05-14 08:07:44.223 JupyterHub app:1532] No admin users, admin interface will be unavailable.
[W 2021-05-14 08:07:44.223 JupyterHub app:1534] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-05-14 08:07:44.223 JupyterHub app:1563] Not using whitelist. Any authenticated user will be allowed.
[I 2021-05-14 08:07:44.302 JupyterHub reflector:199] watching for pods with label selector='component=singleuser-server' in namespace lab
[I 2021-05-14 08:07:44.531 JupyterHub reflector:199] watching for events with field selector='involvedObject.kind=Pod' in namespace lab
++ kill -TERM 32
+ trap - TERM INT
+ wait 32
/opt/app-root/etc/run.sh: line 18:    32 Terminated              jupyterhub -f /opt/app-root/src/jupyterhub_config.py
+ STATUS=143
+ exit 143

@blues-man
Copy link
Contributor

See this #193

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@blues-man @mkossatz