Skip to content

Commit

Permalink
fix: editorial and styles
Browse files Browse the repository at this point in the history
  • Loading branch information
harshini-rangaswamy committed Jul 5, 2024
1 parent 3c79109 commit 5dc5105
Show file tree
Hide file tree
Showing 8 changed files with 348 additions and 273 deletions.
7 changes: 4 additions & 3 deletions docs/products/kafka/concepts/tiered-storage-limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ The main trade-off of tiered storage is the higher latency while accessing and r

## Related pages

- [Tiered storage in Aiven for Apache Kafka® overview](/docs/products/kafka/concepts/kafka-tiered-storage)
- [How tiered storage works in Aiven for Apache Kafka®](/docs/products/kafka/concepts/tiered-storage-how-it-works)
- [Enabled tiered storage for Aiven for Apache Kafka® service](/docs/products/kafka/howto/enable-kafka-tiered-storage)
- [Tiered storage in Aiven for Apache Kafka® overview](/docs/products/kafka/concepts/kafka-tiered-storage)


- [Enabled tiered storage for Aiven for Apache Kafka® service](/docs/products/kafka/howto/enable-kafka-tiered-storage)
170 changes: 75 additions & 95 deletions docs/products/kafka/howto/configure-topic-tiered-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,142 +2,122 @@
title: Enable and configure tiered storage for topics
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import ConsoleLabel from "@site/src/components/ConsoleIcons"

Aiven for Apache Kafka® allows you to configure tiered storage and set retention policies for individual topics. Learn how to configure tiered storage for individual topics and set local retention policies step by step.

## Prerequisite
Aiven for Apache Kafka® allows you to configure tiered storage and set retention policies for individual topics.

- [Tiered storage enabled for the Aiven for Apache Kafka service](/docs/products/kafka/howto/enable-kafka-tiered-storage).
## Prerequisite

## Configure tiered storage for topics via Aiven Console
- [Tiered storage enabled for the Aiven for Apache Kafka service](/docs/products/kafka/howto/enable-kafka-tiered-storage).

1. Access [Aiven console](https://console.aiven.io/), select your
project, and select your Aiven for Apache Kafka service.
1. Click **Topics** from the sidebar.
1. You have the option to either add a new topic with tiered
storage configuration or modify an existing topic to use tiered
storage.
## Important information about tiered storage

### For a new topic
When you activate tiered storage for a service, any new topics created in that service
have tiered storage enabled by default.

1. From the **Topics** page, click **Add topic**.
- **Disable tiered storage for new topics**: You can disable tiered storage by setting
**Remote storage enable** to false during the creation of a new topic in the topic
advanced configurations. Once you activate tiered storage for a topic, you cannot
disable it. Contact Aiven support for assistance.

1. Set the **"Enable advanced configuration?"** option to **Yes** to enable
advanced configurations.
- **Tiered storage for existing topics**: For existing topics, tiered storage is disabled.

1. In the **Topic advanced configuration** drop-down, choose
`remote_storage_enable`. This action reveals the **Remote
storage enabled** drop-down.
## Enable and configure tiered storage for topics

1. Select **True** to activate tiered storage for the topic.
<Tabs groupId="setup">
<TabItem value="Console" label="Console" default>

1. Additionally, you can also set the values for `local_retention_ms`
and `local_retention_bytes` using the respective options from the
drop-down list.
1. Access the [Aiven Console](https://console.aiven.io/), select your project,
and select your Aiven for Apache Kafka service.
1. Click <ConsoleLabel name="topics" />.

:::important
If the values for `local_retention_bytes` and `local_retention_ms`
are not set, they default to -2 or take the configuration from the
service level.
1. On the **Topics** page, either add a new topic with tiered storage or modify an
existing one:
- **Add a new topic**: Click **Add topic**.
- **Modify an existing topic**: Select the topic to modify, then either:
- Click the topic name to open the **Topic info** screen, and click **Modify**.
- Alternatively, in the topic row, click <ConsoleLabel name="actions"/> >
<ConsoleLabel name="Edit topic"/>.

When set to -2, the retention in local storage matches the total
retention. In this scenario, the data segments sent to remote
storage are also retained locally. The remote storage will contain
older data segments than in the local storage only when the total
retention is set to be greater than the local retention.
:::
1. The **Enable advanced configuration?** option is set to **Yes** by default.

1. Click **Add topic** to save your changes and add the topic with
tiered storage.
1. Ensure the **Remote storage enable** option is set to **True** to activate tiered
storage for the topic. To disable it, set it to **False**.

### For an existing topic
1. Optionally, adjust the default values for `local_retention_ms` and `local_retention_bytes`.

1. From the **Topics** page, select the topic for which you wish to
enable tiered storage.
1. Use the ellipsis or open the topic and choose **Modify**.
1. In the **Modify** page, choose `remote_storage_enable` from the
drop-down list, followed by selecting **True** from the **Remote
storage enable** drop-down.
1. Additionally, you can also set the values for `local_retention_ms`
and `local_retention_bytes` using the respective options from the
drop-down list.
:::important
By default, `local_retention_bytes` and `local_retention_ms` are set to `-2` or
inherit the configuration from the service level.

:::important
If the values for `local_retention_bytes` and `local_retention_ms` are
not set, they default to -2 or take the configuration from the service
level.
When set to `-2`, the retention in local storage matches the total retention. In this
scenario, the data segments sent to remote storage are also retained locally. Remote
storage will contain older data segments only if the total retention exceeds the local retention.
:::

When set to -2, the retention in local storage matches the total
retention. In this scenario, the data segments sent to remote storage
are also retained locally. The remote storage will contain older data
segments than in the local storage only when the total retention is set
to be greater than the local retention.
:::
1. Click **Create topic** or **Update** to save your changes and activate tiered storage.

1. Click **Update** to save your changes and activate tiered storage.
</TabItem>
<TabItem value="CLI" label="CLI">

## (Optional) configure client-side parameter
Using the [Aiven CLI](/docs/tools/cli), you can enable tiered storage for specific
Apache Kafka® topics and set retention policies.

For optimal performance and reduced risk of broker interruptions when
using tiered storage, it is recommended to update the client-side
parameter `fetch.max.wait.ms` from its default value of 500 ms to 5000 ms.

## Enable tiered storage for topics via Aiven CLI

Using the [Aiven CLI](/docs/tools/cli),
you can enable tiered storage for specific Apache Kafka® topics and set
retention policies.

### Enable tiered storage for a topic

To enable tiered storage for a Kafka topic, execute the following
command:
To create a topic with tiered storage enabled:

```bash
avn service topic-create \
--project demo-kafka-project \
--service-name demo-kafka-service \
--topic exampleTopic \
--partitions 1 \
--partitions 2 \
--replication 2 \
--remote-storage-enable
```

In this example:

- `demo-kafka-project` is the name of your project.
- `demo-kafka-service` is the name of your Aiven for Apache Kafka®
service.
- `exampleTopic` is the name of the topic you are creating with tiered
storage enabled.
- The topic will have 1 partition and a replication factor of 2.
- `demo-kafka-project` is the name of your project.
- `demo-kafka-service` is the name of your Aiven for Apache Kafka® service.
- `exampleTopic` is the name of the topic you are creating with tiered storage enabled.
- The topic has the partition set to `2` and the replication factor set to `2`

### Configure retention policies for a topic with tiered storage
### Configure retention policies for a topic

After enabling tiered storage, you can configure the retention policies
for local storage:
After enabling tiered storage, you can configure the retention policies for local storage:

```bash
avn service topic-update \
--project demo-kafka-project \
--service-name demo-kafka-service \
--topic exampleTopic \
--local-retention-ms 100 \
--local-retention-bytes 10
--project demo-kafka-project \
--service-name demo-kafka-service \
--topic exampleTopic \
--local-retention-ms 100 \
--local-retention-bytes 10
```

This command sets the local retention time to 100 milliseconds and the
local retention size to 10 bytes for the topic named `exampleTopic` in
the `demo-kafka-service` of the `demo-kafka-project`.
This command sets the local retention time to 100 milliseconds and the local retention
size to 10 bytes for the topic named `exampleTopic` in the `demo-kafka-service` of
the `demo-kafka-project`.

:::important
If the values for `local_retention_bytes` and `local_retention_ms` are
not set, they default to -2 or inherit the configuration from the
service level.

When set to -2, the retention in local storage matches the total
retention. Consequently, data segments sent to remote storage are also
retained locally. The remote storage will contain older data segments
than the local storage, only if the total retention exceeds the local

By default, `local_retention_bytes` and `local_retention_ms` are set to `-2` or
inherit the configuration from the service level.

When set to `-2`, the retention in local storage matches the total retention. In this
scenario, the data segments sent to remote storage are also retained locally. Remote
storage will contain older data segments only if the total retention exceeds the local
retention.
:::

</TabItem>
</Tabs>

## (Optional) configure client-side parameter

For optimal performance and reduced risk of broker interruptions when
using tiered storage, it is recommended to update the client-side
parameter `fetch.max.wait.ms` from its default value of 500 ms to 5000 ms.
93 changes: 65 additions & 28 deletions docs/products/kafka/howto/create-topic.md
Original file line number Diff line number Diff line change
@@ -1,34 +1,71 @@
---
title: Creating an Apache Kafka® topic
title: Create an Apache Kafka® topic
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

When working with Apache Kafka®, while it is possible to configure it to
[automatically create topics when a message is produced to a non-existent topic](create-topics-automatically), it is generally recommended to create topics beforehand,
especially in production environments. This approach offers several
advantages:
When working with Apache Kafka®, while it is possible to configure it to [automatically create topics when a message is produced to a non-existent topic](create-topics-automatically), it is generally recommended to create topics beforehand,
especially in production environments.

- It allows you to define specific topic settings, such as the number
of partitions, replication factor, retention period, and more.
- It helps prevent the generation of incorrect topics due to typos or
This approach offers several advantages:

- You can define specific topic settings, such as the number of partitions,
replication factor, retention period, and more.
- It helps prevent the creation of incorrect topics due to typos or
other mistakes.

## Create an Apache Kafka® topic

To create a topic using the [Aiven
Console](https://console.aiven.io/), follow these steps:

1. Log in to [Aiven Console](https://console.aiven.io/) and select the
Aiven for Apache Kafka® service where you want to create the topic.
2. From the left sidebar, select **Topics**.
3. Select **Create topic** to create a topic and enter a name for
the topic.
4. If required, set the advanced configuration option to **Yes**.
5. In the **Topic advanced configuration** section, you can set
properties such as the replication factor, number of partitions, and
other settings. These settings can be modified later if needed.
6. Select **Create topic**. The new topic will be visible immediately,
but may take a few minutes before you can update its settings.

You can also use the
[topic-create](/docs/tools/cli/service/topic#avn_cli_service_topic_create) command to create a topic using
[Aiven CLI](/docs/tools/cli).
:::note
When tiered storage is activated for your Aiven for Apache Kafka service, all new topics
will have tiered storage enabled by default.
[Learn more about tiered storage](/docs/products/kafka/concepts/kafka-tiered-storage).
:::

## Steps to create an Apache Kafka® topic

<Tabs groupId="setup">
<TabItem value="Console" label="Console" default>

1. Log in to the [Aiven Console](https://console.aiven.io/) and select the Aiven for
Apache Kafka® service to create the topic.
1. Click **Topics** in the sidebar.
1. Click **Create topic** and enter a name for the topic.
1. If required, set the advanced configuration option to **Yes**.
1. In the **Topic advanced configuration** section, set properties such as the
replication factor, number of partitions, and other settings. These settings can be
modified later if needed.
1. Click **Create topic**. The new topic will be visible immediately, but it may take a
few minutes before you can update its settings.

</TabItem>
<TabItem value="CLI" label="CLI">

1. Determine the topic specifications, including the number of partitions,
replication factor, and other settings.

1. Run the following command to create a topic named `exampleTopic`:

```bash
avn service topic-create \
--project demo-kafka-project \
--service-name demo-kafka-service \
--topic exampleTopic \
--partitions 2 \
--replication 2
```

Parameters:

- `avn service topic-create`: Command to create a topic.
- `--project demo-kafka-project`: Specifies the project name.
- `--service-name demo-kafka-service`: Specifies the Aiven for Apache Kafka® service name.
- `--topic exampleTopic`: Specifies the name of the topic to create.
- `--partitions 2`: Specifies the number of partitions for the topic.
- `--replication 2`: Specifies the replication factor for the topic.

</TabItem>
</Tabs>

## Related pages

- [Manage Aiven for Apache Kafka® topics via CLI](/docs/tools/cli/service/topic#avn_cli_service_topic_create)
- [Create Apache Kafka® topics automatically](/docs/products/kafka/howto/create-topics-automatically)
Loading

0 comments on commit 5dc5105

Please sign in to comment.