Documentation on TTL in CREATE and ALTER TABLE + Query Tracing

documentation/concept/query-tracing.md

@@ -0,0 +1,48 @@
+---
+title: Query Tracing
+sidebar_label: Query Tracing
+description:
+  Query tracing is a feature that helps you diagnose performance issues with
+  queries by recording each query's execution time in a system table.
+---
+
+Query tracing is a feature that helps you diagnose performance issues with
+queries by recording each query's execution time in a system table called
+`_query_trace`. You can then analyze the data in this table using the full power
+of QuestDB's SQL statements.
+
+Query tracing is disabled by default. You can enable it using the following
+configuration property:
+
+```text
+query.tracing.enabled=true
+```
+
+You don't need to restart the database server for this property to take effect;
+just run the following query to reload the configuration:
+
+```sql
+select reload_config();
+```
+
+This is an example of what the `_query_trace` table may contain:
+
+```sql
+_query_trace;
+```
+
+|             ts              |        query_text         | execution_micros |
+| --------------------------- | ------------------------- | ---------------- |
+| 2025-01-15T08:52:56.600757Z | telemetry_config LIMIT -1 |             1206 |
+| 2025-01-15T08:53:03.815732Z | tables()                  |             1523 |
+| 2025-01-15T08:53:22.971239Z | 'sys.query_trace'         |             5384 |
+
+As a simple performance debugging example, to get the text of all queries that
+took more than 100 ms, run:
+
+```sql
+select query_text from query_trace() where execution_micros > 100_000;
+```
+
+The `_query_trace` table will drop data older than 24 hours in order to limit
+how much storage query tracing uses.

documentation/concept/ttl.md

@@ -0,0 +1,86 @@
+---
+title: Time To Live (TTL)
+sidebar_label: Time To Live (TTL)
+description: Conceptual overview of the time-to-live feature in QuestDB. Use it to limit data size.
+---
+
+If you're interested in storing and analyzing only recent data with QuestDB, you
+can configure a time-to-live (TTL) for the table data. Both the `CREATE TABLE`
+and `ALTER TABLE` commands support the `TTL` clause.
+
+This feature works as follows:
+
+1. The age of the data is measured by the most recent timestamp stored in the table
+2. As you keep inserting time-series data, the age of the oldest data starts
+   exceeding its TTL limit
+3. When **all** the data in a partition becomes stale, the partition as a whole
+   becomes eligible to be dropped
+4. QuestDB detects a stale partition and drops it as a part of the commit
+   operation
+
+To be more precise, the latest timestamp stored in a given partition does not
+matter. Instead, QuestDB considers the entire time period for which a partition
+is responsible. As a result, it will drop the partition only when the end of
+that period falls behind the TTL limit. This is a compromise that favors a low
+overhead of the TTL enforcement procedure.
+
+To demonstrate, assume we have created a table partitioned by hour, with TTL set
+to one hour:
+
+```sql
+CREATE TABLE tango (ts TIMESTAMP) timestamp (ts) PARTITION BY HOUR TTL 1 HOUR;
+-- or:
+CREATE TABLE tango (ts TIMESTAMP) timestamp (ts) PARTITION BY HOUR TTL 1H;
+```
+
+1\. Insert the first row at 8:00 AM. This is the very beginning of the "8 AM"
+partition:
+
+```sql
+INSERT INTO tango VALUES ('2025-01-01T08:00:00');
+```
+
+| ts |
+|----|
+| 2025-01-01 08:00:00.000000 |
+
+2\. Insert the second row one hour later, at 9:00 AM:
+
+```sql
+INSERT INTO tango VALUES ('2025-01-01T09:00:00');
+```
+
+| ts |
+|----|
+| 2025-01-01 08:00:00.000000 |
+| 2025-01-01 09:00:00.000000 |
+
+The 8:00 AM row remains.
+
+3\. Insert one more row at 9:59:59 AM:
+
+```sql
+INSERT INTO tango VALUES ('2025-01-01T09:59:59');
+```
+
+| ts |
+|----|
+| 2025-01-01 08:00:00.000000 |
+| 2025-01-01 09:00:00.000000 |
+| 2025-01-01 09:59:59.000000 |
+
+The 8:00 AM data is still there, because the "8 AM" partition ends at 9:00 AM.
+
+4\. Insert a row at 10:00 AM:
+
+```sql
+INSERT INTO tango VALUES ('2025-01-01T10:00:00');
+```
+
+| ts |
+|----|
+| 2025-01-01 09:00:00.000000 |
+| 2025-01-01 09:59:59.000000 |
+| 2025-01-01 10:00:00.000000 |
+
+Now the whole "8 AM" partition is outside its TTL limit, and has been dropped.

documentation/reference/sql/alter-table-set-ttl.md

@@ -0,0 +1,85 @@
+---
+title: ALTER TABLE SET TTL
+sidebar_label: SET TTL
+description: ALTER TABLE SET TTL SQL keyword reference documentation.
+---
+
+Sets the time-to-live (TTL) period on a table.
+
+Refer to the [section on TTL](/docs/concept/ttl/) for a conceptual overview.
+
+## Syntax
+
+![Flow chart showing the syntax of the ALTER TABLE keyword](/images/docs/diagrams/alterTable.svg)
+
+![Flow chart showing the syntax of ALTER TABLE with SET TTL keyword](/images/docs/diagrams/setTtl.svg)
+
+## Description
+
+To store and analyze only recent data, configure a time-to-live (TTL) period on
+a table using the `ALTER TABLE SET TTL` command.
+
+Follow the `TTL` keyword with a number and a time unit, one of:
+
+- `HOURS`
+- `DAYS`
+- `WEEKS`
+- `MONTHS`
+- `YEARS`
+
+TTL units fall into two categories:
+
+1. Fixed time periods:
+   - `HOURS`
+   - `DAYS`
+   - `WEEKS`
+2. Calendar-based periods:
+   - `MONTHS`
+   - `YEARS`
+
+Fixed-time periods are always exact durations: `1 WEEK` is always 7 days.
+
+Calendar-based periods may vary in length: `1 MONTH` from January 15th goes to
+February 15th and could be between 28 and 31 days.
+
+QuestDB accepts both singular and plural forms:
+
+- `HOUR` or `HOURS`
+- `DAY` or `DAYS`
+- `WEEK` or `WEEKS`
+- `MONTH` or `MONTHS`
+- `YEAR` or `YEARS`
+
+It also supports shorthand notation: `3H` for 3 hours, `2M` for 2 months.
+
+:::note
+
+QuestDB drops data that exceeded its TTL only a whole partition at a time. For
+this reason, the TTL period must be a whole number multiple of the table's
+partition size.
+
+For example:
+
+- If a table is partitioned by `DAY`, the TTL must be a whole number of days
+  (`24 HOURS`, `2 DAYS` and `3 MONTHS` are all accepted)
+- If a table is partitioned by `MONTH`, the TTL must be in months or years.
+  QuestDB won't accept the `HOUR`, `DAY`, or `WEEK` units
+
+Refer to the [section on TTL in Concepts](/docs/concept/ttl/) for detailed
+information on the behavior of this feature.
+
+:::
+
+## Examples
+
+Set the TTL to 3 weeks:
+
+```sql
+ALTER TABLE weather SET TTL 3 WEEKS;
+```
+
+Set the TTL to 12 hours, using the shorthand syntax for the time unit:
+
+```sql
+ALTER TABLE weather SET TTL 12H;
+```

documentation/reference/sql/create-table.md

@@ -11,7 +11,7 @@ column definitions are used.
 
 To create a table by manually entering parameters and settings:
 
-![Flow chart showing the syntax of the CREATE TABLE keyword](/images/docs/diagrams/createTable.svg)
+![Flow chart showing the syntax of the CREATE TABLE keyword](/images/docs/diagrams/createTableDef.svg)
 
 :::note
 
@@ -53,8 +53,22 @@ CREATE TABLE trades(
 PARTITION BY DAY;
 ```
 
-Next, data deduplication is enabled to discard duplicates for the timestamp and
-ticker columns:
+Now we can add a time-to-live (TTL) period. Once an entire data partition is
+past its TTL, it becomes eligible for automatic removal.
+
+```questdb-sql title="With TTL"
+CREATE TABLE trades(
+  timestamp TIMESTAMP,
+  symbol SYMBOL,
+  price DOUBLE,
+  amount DOUBLE
+  ) TIMESTAMP(timestamp)
+PARTITION BY DAY
+TTL 1 WEEK;
+```
+
+Next, we enable data deduplication. This will discard exact duplicates on the
+timestamp and ticker columns:
 
 ```questdb-sql title="With deduplication, adding ticker as an upsert key."
 CREATE TABLE trades(
@@ -64,6 +78,7 @@ CREATE TABLE trades(
   amount DOUBLE
   ) TIMESTAMP(timestamp)
 PARTITION BY DAY
+TTL 1 WEEK
 DEDUP UPSERT KEYS(timestamp, symbol);
 ```
 
@@ -76,7 +91,8 @@ CREATE TABLE trades(
   price DOUBLE,
   amount DOUBLE
   ) TIMESTAMP(timestamp)
-PARTITION BY DAY;
+PARTITION BY DAY
+TTL 1 WEEK
 DEDUP UPSERT KEYS(timestamp, symbol);
 ```
 
@@ -122,6 +138,63 @@ one of the following:
 The partitioning strategy **cannot be changed** after the table has been
 created.
 
+## Time To Live (TTL)
+
+To store and analyze only recent data, configure a time-to-live (TTL) period on
+a table using the `TTL` clause, placing it right after `PARTITION BY <unit>`.
+You can't set TTL on a non-partitioned table.
+
+Follow the `TTL` keyword with a number and a time unit, one of:
+
+- `HOURS`
+- `DAYS`
+- `WEEKS`
+- `MONTHS`
+- `YEARS`
+
+TTL units fall into two categories:
+
+1. Fixed time periods:
+   - `HOURS`
+   - `DAYS`
+   - `WEEKS`
+2. Calendar-based periods:
+   - `MONTHS`
+   - `YEARS`
+
+Fixed-time periods are always exact durations: `1 WEEK` is always 7 days.
+
+Calendar-based periods may vary in length: `1 MONTH` from January 15th goes to
+February 15th and could be between 28 and 31 days.
+
+QuestDB accepts both singular and plural forms:
+
+- `HOUR` or `HOURS`
+- `DAY` or `DAYS`
+- `WEEK` or `WEEKS`
+- `MONTH` or `MONTHS`
+- `YEAR` or `YEARS`
+
+It also supports shorthand notation: `3H` for 3 hours, `2M` for 2 months.
+
+:::note
+
+QuestDB drops data that exceeded its TTL only a whole partition at a time. For
+this reason, the TTL period must be a whole number multiple of the table's
+partition size.
+
+For example:
+
+- If a table is partitioned by `DAY`, the TTL must be a whole number of days
+  (`24 HOURS`, `2 DAYS` and `3 MONTHS` are all accepted)
+- If a table is partitioned by `MONTH`, the TTL must be in months or years.
+  QuestDB won't accept the `HOUR`, `DAY`, or `WEEK` units
+
+Refer to the [section on TTL in Concepts](/docs/concept/ttl/) for detailed
+information on the behavior of this feature.
+
+:::
+
 ## Deduplication
 
 When [Deduplication](/docs/concept/deduplication) is enabled, QuestDB only

documentation/sidebars.js

@@ -157,6 +157,7 @@ module.exports = {
                 "reference/sql/alter-table-resume-wal",
                 "reference/sql/alter-table-set-param",
                 "reference/sql/alter-table-set-type",
+                "reference/sql/alter-table-set-ttl",
                 "reference/sql/alter-table-squash-partitions",
               ],
             },
@@ -424,7 +425,9 @@ module.exports = {
         },
         "concept/sql-extensions",
         "concept/jit-compiler",
+        "concept/query-tracing",
         "concept/partitions",
+        "concept/ttl",
         "concept/symbol",
         "concept/indexes",
         "concept/interval-scan",