Skip to content

Commit

Permalink
describe how to use jupyter on an HPC as well (#11)
Browse files Browse the repository at this point in the history 
* describe how to use jupyter on an HPC as well

* update for tscc2 which also uses slurm

* clarify some questions that arose

* explain the image

* add multiplexing to docs

* remove somewhat unnecessary controlpersist

* fix broken img in jupyter docs

* move img lower down
  • Loading branch information
@aryarm
aryarm authored Mar 30, 2024
1 parent 8a08691 commit 011ac69
Show file tree
Hide file tree
Showing 2 changed files with 52 additions and 18 deletions.
70 changes: 52 additions & 18 deletions wiki/docs/jupyter_setup.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,12 @@
"\n",
" I recommend installing them into their own environment. Let’s call it `jupyter`.\n",
" ```bash\n",
" conda create -n jupyter -c conda-forge jupyterlab nb_conda_kernels\n",
" conda create -y -n jupyter -c conda-forge jupyterlab nb_conda_kernels\n",
" ```\n",
"\n",
" _But_ if you're **not** concerned about a package in your _base_ environment conflicting with jupyter or its dependencies now or in the future, you can just inall them in your _base_ environment, instead.\n",
" _But_ if you're **not** concerned about a package in your _base_ environment conflicting with jupyter or its dependencies now or in the future, you can just install them in your _base_ environment, instead.\n",
" ```bash\n",
" conda install -n base -c conda-forge jupyterlab nb_conda_kernels\n",
" conda install -y -n base -c conda-forge jupyterlab nb_conda_kernels\n",
" ```\n",
"2. If you installed jupyter into its own environment (as recommended), you’ll need to activate that environment every time you want to use it. Do this now.\n",
" ```bash\n",
Expand All @@ -52,28 +52,30 @@
"**Scenario**: You would like to install R version 4.0 and use it from within Jupyter.\n",
"In order to access your new R installation from within Jupyter, you will also need to install [IRKernel](https://github.com/IRkernel/IRkernel). So, create a new environment called r4 containing both R version 4.0 and IRKernel:\n",
"```bash\n",
"conda create -n r4 -c conda-forge 'r-base=4.0' 'r-irkernel'\n",
"conda create -y -n r4 -c conda-forge 'r-base=4.0' 'r-irkernel'\n",
"```\n",
"\n",
"**Scenario**: A similar strategy can be employed for Python. Let’s say you’d like to install python 3.5 in a new conda environment called py35, and you want to run your python 3.5 installation from within Jupyter.\n",
"To do this, just install both python 3.5 and [IPython kernel](https://ipython.readthedocs.io/en/stable/install/kernel_install.html) in a new environment:\n",
"```bash\n",
"conda create -n py35 -c conda-forge 'python=3.5' ipykernel\n",
"conda create -y -n py35 -c conda-forge 'python=3.5' ipykernel\n",
"```\n",
"\n",
"**Scenario**: Instead, you have an existing environment called py35 with python 3.5, and you’d like to run your python 3.5 installation from within Jupyter.\n",
"Just install [IPython kernel](https://ipython.readthedocs.io/en/stable/install/kernel_install.html) in the existing environment:\n",
"```bash\n",
"conda install -n py35 -c conda-forge ipykernel\n",
"```"
"conda install -y -n py35 -c conda-forge ipykernel\n",
"```\n",
"\n",
"If you want to install a python package to use within your notebooks, you can install it in the `py35`environment, for example. You should never need to install anything in the `jupyter` environment!"
]
},
{
"cell_type": "markdown",
"id": "7c5f9934-f544-4108-bae0-7740ce36b25d",
"metadata": {},
"source": [
"<p style='font-size:x-small'>In general, you will need to install a new kernel in every conda environment that you would like to use with jupyter. But you do not need to install a new copy of jupyter in every environment. The kernels that you install will allow jupyter in your `jupyter` environment to execute each version of Python and R that you install in other environments. And the `nb_conda_kernels` package will automatically detect kernels you’ve installed in those environments on startup.</p>"
"<p style='font-size:small'>In general, you will need to install a new kernel in every conda environment that you would like to use with jupyter. But you do not need to install a new copy of jupyter in every environment. The kernels that you install will allow jupyter in your jupyter environment to execute each version of Python and R that you install in other environments. And the nb_conda_kernels package will automatically detect kernels you’ve installed in those environments on startup. Each kernel should appear when you start up jupyter:</p>"
]
},
{
Expand All @@ -99,16 +101,40 @@
" conda activate jupyter\n",
" ```\n",
"3. On the cluster, start jupyter with the cluster port you used in step #1. And don’t let jupyter try to open a browser:\n",
"```bash\n",
"jupyter lab --no-browser --port 9999\n",
"```\n",
" ```bash\n",
" jupyter lab --no-browser --port 9999\n",
" ```\n",
"4. In a browser running on your own computer, go to http://localhost:8888/lab\n",
"5. Log in using the password you created in step #3 of the setup\n",
"6. Your R and python installations/environments should appear in the Notebook Launcher menu. Click on one to create a notebook. Watch [this video](https://www.youtube.com/watch?v=A5YyoCKxEOU&t=112s) to learn how to write one.\n",
"\n",
"<img src=\"../images/jupyter_envs.png\" width=\"500\"/>\n",
"\n",
"When you’re done, make sure to shut down both your notebook kernel(s) (“Kernel” > “Shut Down All Kernels”) and Jupyter itself (“File” > “Shutdown”) from the menu bar! Otherwise, they might continue running on the cluster forever, especially if you use `screen` sessions."
]
},
{
"cell_type": "markdown",
"id": "a50afbd6-d56e-4d9b-a138-424605df6101",
"metadata": {},
"source": [
"## Usage on an HPC\n",
"If you are working on a high performance computing cluster like TSCC or Expanse, you should perform the following steps between steps 1 and 2 from above."
]
},
{
"cell_type": "markdown",
"id": "49f8bfd9-6c01-4273-924d-d8a8fe9665b2",
"metadata": {},
"source": [
"1. Start an interactive node on your computing cluster\n",
"2. Reverse-forward the port you chose (from step #1 above) from your interactive node back to the login node. Run the following on the interactive node:\n",
" ```bash\n",
" ssh -NfR 9999:localhost:9999 $SLURM_SUBMIT_HOST\n",
" ```\n",
" `$SLURM_SUBMIT_HOST` will automatically determine your login node (ex: login1, login2, etc)."
]
},
{
"cell_type": "markdown",
"id": "cde8fe0a-11a4-4944-bc1f-dfd873160795",
Expand Down Expand Up @@ -148,6 +174,12 @@
" HostName your_server_address\n",
" User your_server_username\n",
" LocalForward 8888 localhost:9999\n",
"```\n",
"If you like to ssh into your cluster multiple times from different terminals, you should also add the following to the top of your `~/.ssh/config`, so that `ssh` will reuse the same connection.\n",
"```\n",
"Host *\n",
" ControlMaster auto\n",
" ControlPath ~/.ssh/ssh_mux_%h_%p_%r\n",
"```"
]
},
Expand Down Expand Up @@ -232,16 +264,17 @@
"id": "c1453140-7dfd-4673-acec-94fdb2bb3396",
"metadata": {},
"source": [
"Assuming you followed the steps to setup the simplified usage, the usage instructions will reduce to just three memorable steps:\n",
"Assuming you followed the steps to setup the simplified usage, the usage instructions will reduce to just three (or four) memorable steps:\n",
"1. ssh into the cluster\n",
" ```bash\n",
" ssh your_short_server_nickname\n",
" ```\n",
"2. Start jupyter on the cluster\n",
"2. If you are working on an HPC, you can login to an interactive node and reverse-forward your port at this stage.\n",
"3. Start jupyter on the cluster\n",
" ```bash\n",
" jup &\n",
" ```\n",
"3. In a browser running on your own computer, go to http://localhost:8888/lab"
"4. In a browser running on your own computer, go to http://localhost:8888/lab"
]
},
{
Expand Down Expand Up @@ -270,16 +303,17 @@
" ```bash\n",
" screen -ls\n",
" ```\n",
"4. Start jupyter on the cluster\n",
"4. If you are working on an HPC, you can login to an interactive node and reverse-forward your port at this stage.\n",
"5. Start jupyter on the cluster\n",
" ```bash\n",
" jup &\n",
" ```\n",
"5. In a browser running on your own computer, go to http://localhost:8888/lab\n",
"6. If your ssh connection dies, everything in your screen session will continue running. Just log back in and reattach the session to continue where you left off.\n",
"6. In a browser running on your own computer, go to http://localhost:8888/lab\n",
"7. If your ssh connection dies, everything in your screen session will continue running. Just log back in and reattach the session to continue where you left off. (Note: If you are working on an HPC, you will need to use the same login node as before!)\n",
" ```bash\n",
" screen -r jupyter\n",
" ```\n",
"7. You can also terminate a session while inside of it\n",
"8. You can also terminate a session while inside of it\n",
" ```bash\n",
" exit\n",
" ```\n",
Expand Down
Binary file added wiki/images/jupyter_envs.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 011ac69

Please sign in to comment.