Merge pull request #315 from quixio/dev

Docs Release 2024-05-002

docs/create/create-topic.md

@@ -21,7 +21,7 @@ The parameters in the `Advanced settings` panel are as follows:
 
 | Parameter | Description |
 |---|---|
-| Partitions | The basic unit of paralellism in a Kafka topic. The more partitions the more messages can be processed concurrently. You can think of partitions in Kafka topics as lanes on a motorway. Each partition acts as an independent lane where messages (like vehicles) flow in an ordered sequence.<br/><br/>Just like lanes on a motorway, partitions enable parallel processing of messages within a Kafka topic. Multiple consumers (or subscribers) can each read from their own partition concurrently, enabling efficient and scalable message processing.<br/><br/>Additionally, partitions provide fault tolerance similar to lanes providing redundancy on a motorway. If one lane (partition) becomes congested or experiences issues, the others can continue operating independently without affecting the overall flow of traffic (messages) on the motorway (topic). **Default = 1** |
+| Partitions | The basic unit of paralellism in a Kafka topic. The more partitions the more messages can be processed concurrently. You can think of partitions in Kafka topics as lanes on a motorway. Each partition acts as an independent lane where messages (like vehicles) flow in an ordered sequence.<br/><br/>Just like lanes on a motorway, partitions enable parallel processing of messages within a Kafka topic. Multiple consumers (or subscribers) can each read from their own partition concurrently, enabling efficient and scalable message processing.<br/><br/>Additionally, partitions provide fault tolerance similar to lanes providing redundancy on a motorway. If one lane (partition) becomes congested or experiences issues, the others can continue operating independently without affecting the overall flow of traffic (messages) on the motorway (topic). <br/><br/>Note, you can't reduce the number of partitions, but you can increase them using the edit topic menu in Quix Cloud. If you need to reduce the number of partitions, you'd have to delete the topic and create it again, with the required number of partitions. **Default = 1** |
 | Retention time (hours) | Messages are retained in the topic for this amount of time, regardless of whether they have been consumed or not. After the specified time period elapses, Kafka will automatically delete messages older than the retention period. **Default = 24** |
 | Retention size (MB) | Messages are retained in the topic until the topic reaches this size. Once the topic size exceeds this threshold, Kafka starts deleting the oldest messages to make space for new ones. **Default = 50** |
 

docs/get-started/streaming.md

@@ -6,4 +6,22 @@ In streaming, it is important to understand the following terms:
 * **Quix producer** - In Quix, a producer publishes data to a Kafka topic. The data is typically thought of as packaged in a message, each message containing one or more events. 
 * **Quix consumer** - A consumer subscribes to a topic, and typically processes the data it receives from that topic, depending on the use case.
 
+More detail on these is provided in the following sections.
+
+## Topic
+
+Topics provide the conduits for data flow in Kafka-based solutions. When you [create a topic](../create/create-topic.md) you can specify important parameters such as the number of partitions in the topic. For an analogy, if you imagine the topic as a motorway, then partitions are the lanes. Clients can publish messages using a key (also known as a stream ID). The messages are divided among the partitions by the message key. For example, if you had two partitions, and three clients (producers) using different keys such as `server-1-cpu`, `server-2-cpu`, and `server-3-cpu`, then messages from client 1 would most likely be sent to partition 0, and  clients 2 and 3 messages would be routed to partition 1. You can see the partitions being used in the Quix Cloud topic explorer, as illustrated by the following screenshot:
+
+![Partitions](../images/messages-topic-partitions.png)
+
+Note that all messages of a particular key (stream ID) will be published to the same partition. You would never have, using this example, `server-1-cpu` messages on partition 1, they are always just on partition 0, and similarly, you'd only have `server-2-cpu` and `server-3-cpu` messages on partition 1. This approach guarantees that messages arrive in order.
+
+## Producer
+
+Producers publish messages to a topic. When you publish a message you can also provide a key. The key is used to divide messages among available partitions. Messages of a specific key are always routed to the same partition.
+
+## Consumer
+
+A consumer subscribes to a topic and then consumes all messages in that topic. You can organize consumers into consumer groups. In a consumer group each consumer will process the messages from partitions allocated to it by Kafka. For example, if you had two partitions in a topic, and two consumers in a consumer group, then the first consumer would process messages from one partition, and the second would consume messages from the other partition. If you had three partitions and two consumers, then one consumer would be allocated one partition, and the other consumer would be allocated two. Using consumer groups enables you to perform horizontal scaling of your processing.
+
 [Read more about Quix](../kb/what-is-quix.md).

mkdocs.yml

@@ -21,11 +21,14 @@ nav:
       - 'get-started/streaming.md'
       - 'get-started/stream-processing.md'
       - 'get-started/stream-processing-pipelines.md'
-    - 'Tutorials':
+    - 'Quickstart':
       - 'Produce data': 'get-started/produce.md'
       - 'Consume data': 'get-started/consume.md'
       - 'Process data': 'get-started/process.md'
-      - 'CLI quickstart': 'get-started/cli-quickstart.md'
+      - 'Build pipelines': 'get-started/cli-quickstart.md'
+# Mostly out of date
+#    - 'Project templates': 'get-started/project-templates.md'
+  - 'Quix Streams': '!import https://github.com/quixio/quix-streams?branch=main'
   - 'Quix CLI':
     - 'Overview': 'cli/overview.md'
     - 'Reference guide': 'cli/cli-reference.md'
@@ -39,10 +42,6 @@ nav:
         - 'Add a destination': 'cli/cli-add-destination.md'
         - 'Sync to Cloud': 'cli/cli-sync-to-cloud.md'
       - 'Using the CLI with a template': 'cli/cli-template.md'
-
-# Mostly out of date
-#    - 'Project templates': 'get-started/project-templates.md'
-  - 'Quix Streams': '!import https://github.com/quixio/quix-streams?branch=main'
   - 'Quix Cloud':
     - 'Overview': 'quix-cloud/overview.md'
     - 'Quickstart': 'quix-cloud/quickstart.md'