Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOCS: Removing errors from documentation #941

Merged
merged 12 commits into from
Dec 15, 2021
Merged
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/feature-request---.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Please provide these details:
Describe your environment:
- Conda version (use `conda --version`);
- Kubernetes version (use `0kubectl version`):
- Cloud provider (e.g. AWS, GCP, DigitalOcean, etc).
- Cloud provider (for example AWS, GCP, DigitalOcean, etc).
- Operating system and version.

> Feel free to add any **additional context** or screenshots about the feature request here.
2 changes: 1 addition & 1 deletion .github/workflows/kubernetes_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
echo "GITHUB_REF: ${GITHUB_REF}"
- name: 'QHUB_GH_BRANCH set for a branch (not a tag)'
if: ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/heads/') }}
# e.g. QHUB_GH_BRANCH="main"
# for example QHUB_GH_BRANCH="main"
run: |
echo "QHUB_GH_BRANCH=${GITHUB_REF:11}" >> "$GITHUB_ENV"
- name: 'Checkout Infrastructure'
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/markdown.links.config.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@
},
{
"pattern": "myqhubsite.com"
},
{
"pattern": "https://cloud.google.com"
}
]
}
6 changes: 3 additions & 3 deletions docs/source/admin_guide/backup.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Manual Backups
# Manual backups

Your cloud provider may have native ways to backup your Kubernetes cluster and volumes.

Expand All @@ -10,7 +10,7 @@ There are three main locations that you need to backup:
2. The Keycloak user/group database
3. The JupyterHub database (for Dashboard configuration)

## Network File System
## Network file system

This amounts to:

Expand Down Expand Up @@ -162,7 +162,7 @@ gsutil cp gs://<your_bucket_name>/backups/2021-04-23.tar .
Similar instructions, but use Digital Ocean spaces. This guide explains installation of the command-line tool:
https://www.digitalocean.com/community/tutorials/how-to-migrate-from-amazon-s3-to-digitalocean-spaces-with-rclone

## Keycloak User/Group Database
## Keycloak user/group database

QHub provides a simple script to export the important user/group database. Your new QHub cluster will recreate a lot of Keycloak config (including new Keycloak clients which will have new secrets), so only the high-level Group and User info is exported.

Expand Down
4 changes: 2 additions & 2 deletions docs/source/admin_guide/gpu.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ used. This can easily be checked via the command `nvidia-smi`.

```shell
$ nvidia-smi
Thu May 20 18:05:14 2021
Thu May 20 18:05:14 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
Expand All @@ -162,7 +162,7 @@ Thu May 20 18:05:14 2021
```

The important section is `CUDA Version`. In general you should install
a version of cudatoolkit that is less than or equal to the cuda
a version of cudatoolkit that's less than or equal to the cuda
version (but not too old). If you install `cudatoolkit-dev` and
`cudatoolkit` make sure that they are the same version exactly
including minor version. Also in the near future cuda should have
Expand Down
4 changes: 2 additions & 2 deletions docs/source/admin_guide/jupyterhub.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ QHub has the JupyterHub project at its core.

Within the `qhub deploy` step, JupyterHub is installed using the [Zero2JupyterHub Helm package](https://zero-to-jupyterhub.readthedocs.io/).

It is possible to specify Helm overrides (i.e. your own values for selected fields in the JupyterHub deployment's `values.yaml` file) from the `qhub-config.yaml` file. However, be aware that this may conflict with values that are needed to be set in a certain way in order for QHub to operate correctly.
It's possible to specify Helm overrides (i.e. your own values for selected fields in the JupyterHub deployment's `values.yaml` file) from the `qhub-config.yaml` file. However, be aware that this may conflict with values that are needed to be set in a certain way in order for QHub to operate correctly.

To set a Helm override, for example enabling auth state:

Expand All @@ -17,7 +17,7 @@ jupyterhub:
enable_auth_state: true
```

Where it is possible to influence a value using 'native' QHub configuration, you should use that as a preference. For example, you would not set `jupyterhub.overrides.hub.image.name` to use a custom JupyterHub Docker image. Instead you would set `default_images.jupyterhub`.
Where it's possible to influence a value using 'native' QHub configuration, you should use that as a preference. For example, you would not set `jupyterhub.overrides.hub.image.name` to use a custom JupyterHub Docker image. Instead you would set `default_images.jupyterhub`.

There is special behavior for the values `jupyterhub.overrides.hub.extraEnv` and `jupyterhub.overrides.hub.extraConfig`. Setting these would have naturally seen them be overridden in their entirety by QHub's own values, but there is special treatment whereby QHub's values are merged into the list of any values that you might have set as overrides.

Expand Down
4 changes: 2 additions & 2 deletions docs/source/admin_guide/keycloak.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ QHub includes a deployment of [Keycloak](https://www.keycloak.org/documentation.

Within the `qhub deploy` step, Keycloak is installed using the [Helm chart](https://github.com/codecentric/helm-charts/tree/master/charts/keycloak).

It is possible to specify Helm overrides (i.e. your own values for selected fields in the Keycloak deployment's `values.yaml` file) from the `qhub-config.yaml` file. However, be aware that this may conflict with values that are needed to be set in a certain way in order for QHub to operate correctly.
It's possible to specify Helm overrides (i.e. your own values for selected fields in the Keycloak deployment's `values.yaml` file) from the `qhub-config.yaml` file. However, be aware that this may conflict with values that are needed to be set in a certain way in order for QHub to operate correctly.

To set a Helm override, for example:

Expand All @@ -24,6 +24,6 @@ security:
repository: dockerusername/my-qhub-keycloak
```

If you do set `overrides.extraEnv` as above, you must remember to include `PROXY_ADDRESS_FORWARDING=true`. Otherwise, the Keycloak deployment will not work as you will have overridden an important default Helm value that is required by QHub.
If you do set `overrides.extraEnv` as above, you must remember to include `PROXY_ADDRESS_FORWARDING=true`. Otherwise, the Keycloak deployment will not work as you will have overridden an important default Helm value that's required by QHub.

To find out more about using Keycloak in QHub, see [Installation - Login](../installation/login.md)
49 changes: 15 additions & 34 deletions docs/source/admin_guide/prefect.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# Prefect

Prefect is a workflow automation system. QHub integrates Prefect with a
feature flag as follows (in the top level):
Prefect is a workflow automation system. QHub integrates Prefect with a feature flag as follows (in the top level):

```yaml
prefect:
Expand All @@ -16,8 +15,7 @@ prefect:
image: prefecthq/prefect:0.14.22-python3.8
```

There are a bunch of components in getting Prefect working for you, here
is a brief description of them:
There are a bunch of components in getting Prefect working for you, here is a brief description of them:

1. Create a free Prefect cloud account here: https://cloud.prefect.io/
2. Create a Service Account and an API key for the same and add this to the CI secrets as `TF_VAR_prefect_token`:
Expand All @@ -29,32 +27,21 @@ is a brief description of them:
prefect create project 'your-prefect-project-name'
```

The `TF_VAR_prefect_token` API key is set as `PREFECT__CLOUD__AGENT__AUTH_TOKEN`
environment variable in the agent. It is used while deploying Prefect Agent so that
it can connect to Prefect Cloud and query flows.
The `TF_VAR_prefect_token` API key is set as `PREFECT__CLOUD__AGENT__AUTH_TOKEN` environment variable in the agent. It's used while deploying Prefect Agent so that it can connect to Prefect Cloud and query flows.

## Prefect Cloud

Prefect Cloud is a fully hosted, production-ready backend for Prefect Core.
Checkout [prefect documentation](https://docs.prefect.io/orchestration/#prefect-cloud)
to know more.

Prefect Cloud is a fully hosted, production-ready backend for Prefect Core. Checkout [prefect documentation](https://docs.prefect.io/orchestration/#prefect-cloud) to know more.

## Prefect Agent

Prefect Agents is a lightweight processes for orchestrating flow runs. Agents
run inside a user's architecture, and are responsible for starting and monitoring
flow runs.

During operation the agent process queries the Prefect API for any scheduled flow
runs, and allocates resources for them on their respective deployment platforms.
Prefect Agents is a lightweight processes for orchestrating flow runs. Agents run inside a user's architecture, and are responsible for starting and monitoring flow runs. During operation the agent process queries the Prefect API for any scheduled flow runs, and allocates resources for them on their respective deployment platforms.

When you enable prefect via `qhub-config.yml` prefect agent is deployed on the
QHub's kubernetes cluster, which queries the Prefect Cloud for flow runs.
When you enable prefect via `qhub-config.yml` prefect agent is deployed on the QHub's kubernetes cluster, which queries the Prefect Cloud for flow runs.

## Agent configuration overrides
You can override your agent configuration without having to modify the helm files directly. The extra variable `overrides` makes this
possible by changing the default values for the Agent chart according to the settings presented on your qhub-config.yaml file.

You can override your agent configuration without having to modify the helm files directly. The extra variable `overrides` makes this possible by changing the default values for the Agent chart according to the settings presented on your qhub-config.yaml file.

The current variables, originally available in the [Agent helm chart](https://github.com/PrefectHQ/server/blob/master/helm/prefect-server/templates/agent/deployment.yaml) that can be overridden include:

Expand All @@ -68,8 +55,7 @@ The current variables, originally available in the [Agent helm chart](https://gi
- IMAGE_PULL_POLICY
```

For example, if you just want to override the amount of CPU limits for each job, you would need to craft a declarative configuration,
in you qhub-config.yaml file, as follows:
For example, if you just want to override the amount of CPU limits for each job, you would need to craft a declarative configuration, in you qhub-config.yaml file, as follows:

```yaml
prefect:
Expand All @@ -91,7 +77,9 @@ prefect:
envVars:
MY_VAR: "<value>"
```

### Adding secrets to your Agent configuration

Overrides also allow you to define extra secrets to pass through your agent configuration, for example, when using [default secrets](https://docs.prefect.io/core/concepts/secrets.html#default-secrets) to automatically authenticate your flow with the listed service. In the Google cloud case, for `GCP_CREDENTIALS` context secret, you can do it by adding that specific key value pair into your configuration:

```yaml
Expand All @@ -105,9 +93,7 @@ This secret will then be stored as a [kubernetes secret](https://kubernetes.io/d

## Flows

Prefect agent can only orchestrate your flows, you need an actual flow to run via
prefect agent. The API for the same can be found in the [prefect documentation](https://docs.prefect.io/core/concepts/flows.html)
Here is a simple example from their official doc:
Prefect agent can only orchestrate your flows, you need an actual flow to run via prefect agent. The API for the same can be found in the [prefect documentation](https://docs.prefect.io/core/concepts/flows.html) Here is a simple example from their official doc:

```python
from prefect import task, Task, Flow
Expand All @@ -128,10 +114,7 @@ with Flow('My Functional Flow') as flow:

## Storage

The Prefect Storage interface encapsulates logic for storing flows. Each storage
unit is able to store multiple flows (with the constraint of name uniqueness within a given unit).

The API documentation for the same can be found in the [prefect documentation](https://docs.prefect.io/api/latest/storage.html#docker)
The Prefect Storage interface encapsulates logic for storing flows. Each storage unIt's able to store multiple flows (with the constraint of name uniqueness within a given unit). The API documentation for the same can be found in the [prefect documentation](https://docs.prefect.io/api/latest/storage.html#docker)

## Example: Creating, Building and Register Flow

Expand Down Expand Up @@ -227,9 +210,7 @@ if __name__ == "__main__":

## Running your flows

Now that you have Prefect Agent running in QHub Kubernetes cluster, you
can now run your flows from either of the two ways:
Now that you have Prefect Agent running in QHub Kubernetes cluster, you can now run your flows from either of the two ways:

- Triggering manually from the Prefect Cloud dashboard.
- Running them on a schedule by adding a parameter to you flow. You can read
more about it in the [prefect docs.](https://docs.prefect.io/core/tutorial/05-running-on-a-schedule.html#running-on-schedule)
- Running them on a schedule by adding a parameter to you flow. You can read more about it in the [prefect docs.](https://docs.prefect.io/core/tutorial/05-running-on-a-schedule.html#running-on-schedule)
Loading