Skip to content

Commit

Permalink
Merge pull request #16 from aisingapore/0.2.1-fix
Browse files Browse the repository at this point in the history
0.2.1 fixes
  • Loading branch information
Syakyr authored Jan 31, 2024
2 parents c53ba76 + 6c1d633 commit 6bb5044
Show file tree
Hide file tree
Showing 15 changed files with 451 additions and 295 deletions.
14 changes: 8 additions & 6 deletions .github/ISSUE_TEMPLATE/bug.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,27 +6,28 @@ assignees: ["syakyr", "deonchia"]
body:
- type: markdown
attributes:
value: |
value:
Please fill this form if you have any bugs you wish to resolve
when using Kapitan Hull.
- type: checkboxes
attributes:
label: Problem Domain
description: \
description:
Let us know which service(s) this problem has impacted.
You can choose more than one domain depending on your issue.
options:
- label: Docker
- label: Run:ai
- label: MLFlow
- label: Kubernetes
- label: Gitlab
- label: Other issue
validations:
required: true
- type: checkboxes
attributes:
label: OS/Platform(s) Used
description: \
description:
What OS or platform(s) did you encounter this issue? Specify
exact version or distro in the Problem Brief if needed.
options:
Expand All @@ -37,19 +38,20 @@ body:
- label: Coder
- label: Run:ai VSCode
- label: Run:ai JupyterLab
- label: Other platforms/None
validations:
required: true
- type: textarea
attributes:
label: Problem Brief
description: |
description:
Briefly explain what the problem is about in 1 to 3 sentences.
validations:
required: true
- type: textarea
attributes:
label: Steps to Reproduce
description: |
description:
This may be the most grueling step to write about as you may
need to trace back your steps on how you encounter it. If it's
a random chance that you may get the issue, estimate the chance
Expand All @@ -59,7 +61,7 @@ body:
- type: textarea
attributes:
label: Expected Result
description: |
description:
What is the expected result when reproducing the steps?
validations:
required: true
Expand Down
7 changes: 4 additions & 3 deletions .github/ISSUE_TEMPLATE/query.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,27 +5,28 @@ labels: ["triage"]
body:
- type: markdown
attributes:
value: |
value:
Please fill this form if you have any queries/issues with
Kapitan Hull.
- type: checkboxes
attributes:
label: Query Domain
description: \
description:
Let us know which service(s) this query is about.
You can choose more than one domain depending on your query.
options:
- label: Docker
- label: Run:ai
- label: MLFlow
- label: Kubernetes
- label: Gitlab
- label: Other issue
validations:
required: true
- type: textarea
attributes:
label: Query Brief
description: |
description:
Briefly explain what the query is about in 1 to 3 sentences.
validations:
required: true
2 changes: 2 additions & 0 deletions {{cookiecutter.repo_name}}/.gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ test:conda-build:
changes:
- {{cookiecutter.repo_name}}-conda-env.yaml
- if: $CI_PIPELINE_SOURCE == "push"
- if: $CI_PIPELINE_SOURCE == "web"
- if: $CI_COMMIT_TAG
when: never
artifacts:
Expand All @@ -41,6 +42,7 @@ test:pylint-pytest:
- src/**/*
- conf/**/*
- if: $CI_PIPELINE_SOURCE == "push"
- if: $CI_PIPELINE_SOURCE == "web"
- if: $CI_COMMIT_TAG
when: never
needs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,20 @@ An advantage presented by orchestration platforms is that you can
utilise the Kubernetes cluster's resources for your development and
engineering works instead of your own resources.

We can make use of
[Run:ai workspaces](https://docs.run.ai/v2.13/Researcher/user-interface/workspaces/overview/)
to spin up VSCode or JupyterLab servers with which cluster resources
can be dedicated.
We can make use of either [Coder][coder] or
[Run:ai workspaces][runai-wksp] to spin up VSCode or JupyterLab servers
with which cluster resources can be dedicated.

While there exist the option for engineers to set up either a VSCode
or JupyterLab service (or both), the former would suffice. Reason
While there exist the option for engineers to set up either a VSCode or
JupyterLab service (or both), the former would suffice. Reason being
being VSCode is an excellent code editor with integrated terminal
capabilities and it can also work with Jupyter notebooks.
JupyterLab on the other hand, while being the best interface for
Jupyter notebooks, has subpar UX for its terminal and code editor.
That is to be expected however as it is dedicated to the
Jupyter ecosystem.
capabilities and it can also work with Jupyter notebooks. JupyterLab
on the other hand, while being the best interface for Jupyter
notebooks, has subpar UX for its terminal and code editor. That is to
be expected however as it is dedicated to the Jupyter ecosystem.

[coder]: https://coder.com/
[runai-wksp]: https://docs.run.ai/v2.13/Researcher/user-interface/workspaces/overview/

## Workspace Building Blocks

Expand Down Expand Up @@ -177,9 +178,9 @@ either method to gain access to a remote VSCode developer workspace.
### Persistent Workspaces

!!! warning "Attention"
If you have spun up using the Run:ai YAML method, then you can
skip this step since you've already prepared your workspace prior
to spinning it up.
If you have spun up using the Run:ai YAML method, then you can skip
this step since you've already prepared your workspace prior to
spinning it up.

As mentioned, a PVC should be attached to the workspaces to persist
changes to the filesystems. You can use the following command to search
Expand All @@ -194,9 +195,9 @@ for the PVC:
If there's no result, check with the MLOps team about this.

By default, the PVCs would contain a `workspaces` directory with which
you can create a subdirectory for yourself treat it as your own personal
workspace, where all your work and other relevant assets can be
persisted.
you can create a subdirectory for yourself treat it as your own
personal workspace, where all your work and other relevant assets can
be persisted.

=== "VSCode Server Terminal"

Expand All @@ -208,8 +209,8 @@ persisted.
### Git from VSCode

To clone or push to Git repositories within the VSCode integrated
terminal, it is recommended that you first disable VSCode's
Git authentication handler:
terminal, it is recommended that you first disable VSCode's Git
authentication handler:

Git by default is installed in the VSCode server image. One thing to
take note is that as the persistent storage would be accessible by the
Expand Down Expand Up @@ -245,6 +246,16 @@ if you intend to use Jupyter notebooks within the VSCode environment.
- [`ms-toolsai.jupyter`][vsx-jy]: Official extension by Microsoft
for Jupyter support.

!!! warning "Manual Installation"
For some clusters, you may need to install the extensions manually
due to firewall issues. If that is the case, you can download the
extension(s) through your local machine and upload them to the
VSCode terminal. From there, you can make use of the following
command:
```
$ code-server --install-extension /path/to/extension.vsix
```

!!! warning "Attention"
Do head over [here][jy-vscode] on how to enable the usage of
virtual `conda` environments within VSCode.
Expand Down Expand Up @@ -319,10 +330,12 @@ interface for interacting with or editing notebooks.

!!! warning "Attention"
Under the `Environment` block, there is an expandable section
called `More settings`. Under this section, you can provide more
arguments for a container that will be spun up for the
workspace. For the JupyterLab interface to be able to access any
PVC mounted to the container, you should include the following argument: `--NotebookApp.notebook_dir="/path/to/pvc"`.
called `More settings`. Under this section, you can provide
more arguments for a container that will be spun up for the
workspace. For the JupyterLab interface to be able to access
any PVC mounted to the container, you should include the
following argument:
`--NotebookApp.notebook_dir="/path/to/pvc"`.

Once you have selected the blocks, you can proceed to create the
workspace and you will be redirected to the workspaces page. On this
Expand Down Expand Up @@ -427,10 +440,10 @@ server as well as any associated files can be found under
## Using Docker within Kubernetes

!!! caution
Since these development environments are essentially pods
deployed within a Kubernetes cluster, using Docker within the pods
themselves is not feasible by default and while possible,
should be avoided.
Since these development environments are essentially pods deployed
within a Kubernetes cluster, using Docker within the pods
themselves is not feasible by default and while possible, should
be avoided.

??? info "Reference Link(s)"

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ workspace directory:
=== "VSCode Server Terminal"

```bash
(base) $ conda env create -f {{cookiecutter.repo_name}}-conda-env.yaml \
(base) $ conda env create \
-f {{cookiecutter.repo_name}}-conda-env.yaml \
-p /<NAME_OF_DATA_SOURCE>/workspaces/<YOUR_HYPHENATED_NAME>/conda_envs/{{cookiecutter.repo_name}}
```

Expand Down Expand Up @@ -66,33 +67,30 @@ server to detect the `conda` environments that you would have created.
In this case, you should be navigating to the folder
`/<NAME_OF_DATA_SOURCE>/workspaces/<YOUR_HYPHENATED_NAME>/{{cookiecutter.repo_name}}`.

- Install the VSCode extensions
[`ms-python.python`](https://marketplace.visualstudio.com/items?itemName=ms-python.python)
and
[`ms-toolsai.jupyter`](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter).
After installation of these extensions, restart VSCode by using
the shortcut `Ctrl + Shift + P`, entering `Developer: Reload Window` in the
prompt and pressing `Enter` following that.

- Ensure that you have
[`ipykernel`](https://ipython.readthedocs.io/en/stable/install/kernel_install.html)
installed in the `conda` environment that you intend to use.
This template by default lists the library as a dependency under
- Install the VSCode extensions [`ms-python.python`][py-ext] and
[`ms-toolsai.jupyter`][jy-ext]. After installation of these
extensions, restart VSCode by using the shortcut `Ctrl + Shift + P`,
entering `Developer: Reload Window` in the prompt and pressing
`Enter` following that.

- Ensure that you have [`ipykernel`][ipyk] installed in the `conda`
environment that you intend to use. This template by default lists
the library as a dependency under
`{{cookiecutter.repo_name}}-conda-env.yaml`. You can check for the
library like so:

=== "VSCode Server Terminal"

```bash
$ conda activate /<NAME_OF_DATA_SOURCE>/workspaces/<YOUR_HYPHENATED_NAME>/conda_envs/{{cookiecutter.repo_name}}-conda-env
$ conda activate /<NAME_OF_DATA_SOURCE>/workspaces/<YOUR_HYPHENATED_NAME>/conda_envs/{{cookiecutter.repo_name}}
$ conda list | grep "ipykernel"
ipykernel X.X.X pypi_0 pypi
```

- Now enter `Ctrl + Shift + P` again and execute `Python: Select Interpreter`.
Provide the path to the Python executable within the `conda`
environment that you intend to use, something like so:
`path/to/conda_env/bin/python`.
- Now enter `Ctrl + Shift + P` again and execute
`Python: Select Interpreter`. Provide the path to the Python
executable within the `conda` environment that you intend to use,
something like so: `path/to/conda_env/bin/python`.

- Open up any Jupyter notebook and click on the button that says
`Select Kernel` on the top right hand corner. You will be presented
Expand All @@ -102,27 +100,29 @@ server to detect the `conda` environments that you would have created.
- Test out the kernel by running the cells in the sample notebook
provided under `notebooks/sample-pytorch-notebook.ipynb`.

[py-ext]: https://marketplace.visualstudio.com/items?itemName=ms-python.python
[jy-ext]: https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter
[ipyk]: https://ipython.readthedocs.io/en/stable/install/kernel_install.html

## Jupyter Kernel for JupyterLab

The same with the VSCode server, the JupyterLab server
would not by default detect `conda` environments. You would have to
specify to the JupyterLab installation the `ipython` kernel existing
within your `conda` environment.
The same with the VSCode server, the JupyterLab server would not by
default detect `conda` environments. You would have to specify to the
JupyterLab installation the `ipython` kernel existing within your
`conda` environment.

- Open up a
[terminal within JupyterLab](https://jupyterlab.readthedocs.io/en/stable/user/terminal.html).
- Open up a [terminal within JupyterLab][jy-tty].

- Activate the `conda` environment in question and ensure that you have
[`ipykernel`](https://ipython.readthedocs.io/en/stable/install/kernel_install.html)
installed in the `conda` environment that you intend to use.
This template by default lists the library as a dependency under
`{{cookiecutter.repo_name}}-conda-env.yaml`. You can check for the
library like so:
[`ipykernel`][ipyk] installed in the `conda` environment that you
intend to use. This template by default lists the library as a
dependency under `{{cookiecutter.repo_name}}-conda-env.yaml`. You can
check for the library like so:

=== "JupyterLab Terminal"

```bash
$ conda activate /<NAME_OF_DATA_SOURCE>/workspaces/<YOUR_HYPHENATED_NAME>/conda_envs/{{cookiecutter.repo_name}}-conda-env
$ conda activate /<NAME_OF_DATA_SOURCE>/workspaces/<YOUR_HYPHENATED_NAME>/conda_envs/{{cookiecutter.repo_name}}
$ conda list | grep "ipykernel"
ipykernel 6.9.2 pypi_0 pypi
```
Expand All @@ -132,17 +132,16 @@ within your `conda` environment.
=== "JupyterLab Terminal"

```bash
$ ipython kernel install --name "{{cookiecutter.repo_name}}-conda-env" --user
$ ipython kernel install --name "{{cookiecutter.repo_name}}" --user
```

- Refresh the page.

- Open up the sample notebook provided under `notebooks/sample-pytorch-notebook.ipynb`.

- Within each Jupyter notebook, you can select the kernel of
specific `conda` environments that you intend to use by heading to
the toolbar under
`Kernel` -> `Change Kernel...`.
- Within each Jupyter notebook, you can select the kernel of specific
`conda` environments that you intend to use by heading to the toolbar
under `Kernel` -> `Change Kernel...`.

![Run:ai - JupyterLab Server Change Kernel](assets/screenshots/runai-jupyterlab-server-change-kernel.png)

Expand All @@ -151,3 +150,5 @@ within your `conda` environment.
??? info "Reference Link(s)"

- [Jupyter Docs - Kernels (Programming Languages)](https://docs.jupyter.org/en/latest/projects/kernels.html)

[jy-tty]: https://jupyterlab.readthedocs.io/en/stable/user/terminal.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,9 @@
## Sample Data

While you may have your own project data to work with, for the purpose
of following through with this template guide, let's download
the sample data for the
[sample problem statement](./02-preface.md#guides-problem-statement)
at hand within our VSCode server workspace.
of following through with this template guide, let's download the
sample data for the [sample problem statement][prob] at hand within our
VSCode server workspace.

=== "VSCode Server Terminal"

Expand All @@ -31,3 +30,5 @@ at hand within our VSCode server workspace.

In the next section, we will work towards processing this set of raw
data and eventually training an image classification model.

[prob]: ./02-preface.md#guides-problem-statement
Loading

0 comments on commit 6bb5044

Please sign in to comment.