Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SQL Scheduler docs #59

Merged
merged 6 commits into from
Mar 18, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+326 KB for that one. Acceptable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used oxipng on all of them 🤷🏻‍♂️
From looking at other screenshots they're about average, no? Not sure I can do much more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just wonder why that one is much larger than the others.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
matkuliak marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That file still weighs in with +507 KB. Is it legit?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From looking at other screenshots they're about average, no? Not sure I can do much more.

docs/_assets/img/cluster-sql-console.png from GH-66 is also pretty large, but I also can't get it smaller, even after reducing its physical width.

So, I figure it will be good to go. Thanks.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, thought I was doing something wrong for a sec. But they are somewhat big, it's weird.
I will look into it a bit more. Thank you.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_assets/img/cluster-sql-scheduler-logs.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
122 changes: 122 additions & 0 deletions docs/reference/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ more information on CrateDB Cloud-related terminology.
- [Backups](#overview-cluster-backups)
- [Cluster Cloning](#overview-cluster-cloning)
- [Failed cloning](#overview-cluster-cloning-fail)
- [SQL Scheduler](#overview-sql-scheduler)
- [Scale](#overview-cluster-settings-scale)
- [Manage](#overview-cluster-manage)
- [Community](#overview-community)
Expand Down Expand Up @@ -604,6 +605,127 @@ screen.

![Cloud Console cluster failed cloning](../_assets/img/cluster-clone-failed.png)

(overview-sql-scheduler)=
:::
## SQL Scheduler
:::

The SQL Scheduler is designed to automate routine database tasks by scheduling
SQL queries to run at specific times, in UTC time. This feature
supports creating job descriptions with valid
[cron patterns](https://www.ibm.com/docs/en/db2oc?topic=task-unix-cron-format)
and SQL statements, enabling a wide range of tasks. Users can manage these jobs
through the Cloud UI, adding, removing, editing, activating, and deactivating
them as needed.

### Use Cases

- Deleting old/redundant data to maintain database efficiency.
- Regularly updating or aggregating table data.
- Automating export and import of data.

:::{note}
- The SQL Scheduler is automatically available for all newly deployed clusters.
- For existing clusters, the feature can be enabled on demand.
(Contact support for activation.)
:::

### Accessing and Using the SQL Scheduler

SQL Scheduler can be found in "SQL Scheduler" tab in the left-hand navigation
menu. There are 2 tabs on the SQL Scheduler page:

:::{tab} Overview
<br>

**Overview** shows a list of your existing jobs. In the list you can
activate/deactivate each job with a toggle in the "Active" column. You can
also edit and delete jobs with buttons on the right side of the list.

![SQL Scheduler overview](../_assets/img/cluster-sql-scheduler-overview.png)
:::

:::{tab} Logs
<br>

**Logs** shows a list of *scheduled* job runs, whether they failed or
succeeded, execution time, run time, and the error in case they were
unsuccessful. In case of an error, more details can be viewed showing the
executed query and a stack trace. You can filter the logs by status, or by
specific job.

![SQL Scheduler overview](../_assets/img/cluster-sql-scheduler-logs.png)
:::

### Examples

::::{tab} Cleanup of old files
<br>

Cleanup tasks represent a common use case for these types of automated jobs.
This example deletes records older than 30 days, from a specified table once a
day:

:::{code} sql
DELETE FROM "sample_data"
WHERE
"timestamp_column" < NOW() - INTERVAL '30 days';
:::

How often you run it of course depends on you, but once a day is common for
clean up. This expression runs every day at 2:30 PM UTC:

Schedule: `30 14 * * *`

![SQL Scheduler overview](../_assets/img/cluster-sql-scheduler-example-cleanup.png)
::::

::::{tab} Copying logs into persistent table
<br>

Another useful example might be copying data to another table for archival
purposes. This specifically copies from system logs table into one of our
own tables.

:::{code} sql
CREATE TABLE IF NOT EXISTS "logs"."persistent_jobs_log" (
"classification" OBJECT (DYNAMIC),
"ended" TIMESTAMP WITH TIME ZONE,
"error" TEXT,
"id" TEXT,
"node" OBJECT (DYNAMIC),
"started" TIMESTAMP WITH TIME ZONE,
"stmt" TEXT,
"username" TEXT,
PRIMARY KEY (id)
) CLUSTERED INTO 1 SHARDS;

INSERT INTO
"logs"."persistent_jobs_log"
SELECT
*
FROM
sys.jobs_log
ON CONFLICT ("id") DO NOTHING;
:::

In this example, we schedule the job to run every hour:

Schedule: `0 * * * *`


![SQL Scheduler overview](../_assets/img/cluster-sql-scheduler-example-copying.png)
::::

:::{note}
Limitations and Known Issues:

* Only one job can run at a time; subsequent jobs will be queued until the
current one completes.
* Long-running jobs may block the execution of queued jobs, leading to
potential delays.
:::

(overview-cluster-settings-scale)=
### Scale

Expand Down
Loading