Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more description for topics and partitions #313

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/create/create-topic.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The parameters in the `Advanced settings` panel are as follows:

| Parameter | Description |
|---|---|
| Partitions | The basic unit of paralellism in a Kafka topic. The more partitions the more messages can be processed concurrently. You can think of partitions in Kafka topics as lanes on a motorway. Each partition acts as an independent lane where messages (like vehicles) flow in an ordered sequence.<br/><br/>Just like lanes on a motorway, partitions enable parallel processing of messages within a Kafka topic. Multiple consumers (or subscribers) can each read from their own partition concurrently, enabling efficient and scalable message processing.<br/><br/>Additionally, partitions provide fault tolerance similar to lanes providing redundancy on a motorway. If one lane (partition) becomes congested or experiences issues, the others can continue operating independently without affecting the overall flow of traffic (messages) on the motorway (topic). **Default = 1** |
| Partitions | The basic unit of paralellism in a Kafka topic. The more partitions the more messages can be processed concurrently. You can think of partitions in Kafka topics as lanes on a motorway. Each partition acts as an independent lane where messages (like vehicles) flow in an ordered sequence.<br/><br/>Just like lanes on a motorway, partitions enable parallel processing of messages within a Kafka topic. Multiple consumers (or subscribers) can each read from their own partition concurrently, enabling efficient and scalable message processing.<br/><br/>Additionally, partitions provide fault tolerance similar to lanes providing redundancy on a motorway. If one lane (partition) becomes congested or experiences issues, the others can continue operating independently without affecting the overall flow of traffic (messages) on the motorway (topic). <br/><br/>Note, you can't reduce the number of partitions, but you can increase them using the edit topic menu in Quix Cloud. If you need to reduce the number of partitions, you'd have to delete the topic and create it again, with the required number of partitions. **Default = 1** |
| Retention time (hours) | Messages are retained in the topic for this amount of time, regardless of whether they have been consumed or not. After the specified time period elapses, Kafka will automatically delete messages older than the retention period. **Default = 24** |
| Retention size (MB) | Messages are retained in the topic until the topic reaches this size. Once the topic size exceeds this threshold, Kafka starts deleting the oldest messages to make space for new ones. **Default = 50** |

Expand Down
18 changes: 18 additions & 0 deletions docs/get-started/streaming.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,22 @@ In streaming, it is important to understand the following terms:
* **Quix producer** - In Quix, a producer publishes data to a Kafka topic. The data is typically thought of as packaged in a message, each message containing one or more events.
* **Quix consumer** - A consumer subscribes to a topic, and typically processes the data it receives from that topic, depending on the use case.

More detail on these is provided in the following sections.

## Topic

Topics provide the conduits for data flow in Kafka-based solutions. When you [create a topic](../create/create-topic.md) you can specify important parameters such as the number of partitions in the topic. For an analogy, if you imagine the topic as a motorway, then partitions are the lanes. Clients can publish messages using a key (also known as a stream ID). The messages are divided among the partitions by the message key. For example, if you had two partitions, and three clients (producers) using different keys such as `server-1-cpu`, `server-2-cpu`, and `server-3-cpu`, then messages from client 1 would most likely be sent to partition 0, and clients 2 and 3 messages would be routed to partition 1. You can see the partitions being used in the Quix Cloud topic explorer, as illustrated by the following screenshot:

![Partitions](../images/messages-topic-partitions.png)

Note that all messages of a particular key (stream ID) will be published to the same partition. You would never have, using this example, `server-1-cpu` messages on partition 1, they are always just on partition 0, and similarly, you'd only have `server-2-cpu` and `server-3-cpu` messages on partition 1. This approach guarantees that messages arrive in order.

## Producer

Producers publish messages to a topic. When you publish a message you can also provide a key. The key is used to divide messages among available partitions. Messages of a specific key are always routed to the same partition.

## Consumer

A consumer subscribes to a topic and then consumes all messages in that topic. You can organize consumers into consumer groups. In a consumer group each consumer will process the messages from partitions allocated to it by Kafka. For example, if you had two partitions in a topic, and two consumers in a consumer group, then the first consumer would process messages from one partition, and the second would consume messages from the other partition. If you had three partitions and two consumers, then one consumer would be allocated one partition, and the other consumer would be allocated two. Using consumer groups enables you to perform horizontal scaling of your processing.

[Read more about Quix](../kb/what-is-quix.md).
Binary file added docs/images/messages-topic-partitions.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading