Skip to content

Commit

Permalink
Fix some typos and reword a few things
Browse files Browse the repository at this point in the history
  • Loading branch information
weeco committed Jun 3, 2021
1 parent 10ed356 commit 5982e2c
Showing 1 changed file with 31 additions and 26 deletions.
57 changes: 31 additions & 26 deletions docs/end-to-end.md
Original file line number Diff line number Diff line change
@@ -1,54 +1,61 @@
# End-To-End Monitoring

This page describes the end-to-end monitoring feature in kminion, how it works, and what metrics it provides.
This page describes the end-to-end monitoring feature in KMinion, how it works, and what metrics it provides.

## Motivation
> What is the issue? Why did we build this feature?
We can (and do!) monitor metrics like CPU usage, free disk space, or even consumer group lag.
However those metrics don't give us a good idea of the performance characteristics an actual, real-world, client can expect from the cluster.
We can monitor metrics like CPU usage, free disk space, or even consumer group lag.
However, these metrics don't give us a good idea of the performance characteristics an actual, real-world, client
experiences when connected to the cluster.

With the "classic" metrics lots of questions go unanswered:
- Can a client even produce messages to the cluster in general?
- Does the cluster still give a an acceptable performance for producing messages, committing offsets, and consuming messages?
- What if only a subset of the brokers is somehow unable to handle requests?
- Can a client produce messages to the cluster?
- Can clients produce & consume messages as well as commit group offsets with an acceptable latency?
- Is the cluster in a healthy state from a client's perspective?

## Approach & Implementation
> How do we solve those issues? How does the feature work?
The most reliably way to get real-world performance and availability metrics is to just actually run a producer/consumer ourselves - which is exactly what the end-to-end monitoring feature does!
The most reliably way to get real-world performance and availability metrics is to actually run a producer/consumer
ourselves. This is exactly what the end-to-end monitoring feature does!

## High Level Overview
In order to determine if the cluster is fully operational and its performance is within acceptable limits, kminion continously produces and consumes messages to/from the cluster. That way we can measure things like ack-latency, commit-latency, and roundtrip-time.
In order to determine if the cluster is fully operational, and it's performance is within acceptable limits,
KMinion continuously produces and consumes messages to/from the cluster. That way we can measure things like ack-latency,
commit-latency, and roundtrip-time.

Kminion creates and manages its own topic for the end-to-end test messages. The name of the topic can be configured.
KMinion creates and manages its own topic for the end-to-end test messages. The name of the topic can be configured.

**The first step** is to create a message and send it to the cluster.
- Every produced message is added to an internal tracker so we can recognize messages being "lost" (messages that don't arrive back at the consumer within the configured time span).
- Every produced message is added to an internal tracker, so we can recognize messages being "lost".
A message is considered lost if it doesn't arrive back at the consumer within the configured time span.

**The second step** is to continously consume the topic.
- As each message arrives, we calculate its roundtrip time (time from the point the message was created, until kminion received it again)
**The second step** is to continuously consume the topic.
- As each message arrives, we calculate its roundtrip time (time from the point the message was created, until KMinion received it again)
- Consumer group offsets are committed periodically, while also recording the time each commit takes.

### Topic Management
The topic kminion uses is created and managed completely automatically (the topic name can be configured though).
The topic KMinion uses, is created and managed completely automatically (the topic name can be configured though).

Kminion continously verifies the topic and fixes issues automatically:
- Add partitions to the topic so it has at least as many partitions as there are brokers.
- Will reassign partitions to ensure every broker leads at least one partition, and that all partitions' replicas are distributed evenly across the brokers. (Preferably assigns so partitionID and leader brokerID matches)
KMinion continuously checks the topic and fixes issues/imbalances automatically:
- Add partitions to the topic, so it has at least as many partitions as there are brokers.
- Will reassign partitions to ensure every broker leads at least one partition, and that all partitions' replicas
are distributed evenly across the brokers. KMinion tries to assign partitionIDs to brokers that have the same broker id.


### Consumer Group Management
On startup each kminion instance generates a unique identifier (UUID) used to create its own consumer group - using the shared prefix from the config.
On startup each KMinion instance generates a unique identifier (UUID) that is used to create its own consumer group.
It incorporates the shared prefix from the config.

That is neccesary because:
That is necessary because:
- Offsets must not be shared among multiple instances.
- Each instance must always consume **all** partitions of the topic.

The instances' UUID is also embedded in every message, so each instance can easily filter out messages it didn't produce.
That's why it is perfectly fine to run multiple kminion instances against the same cluster, using the same topic.
That's why it is perfectly fine to run multiple KMinion instances against the same cluster, using the same topic.

Kminion also monitors and deletes consumer groups that use its configured prefix.
KMinion also monitors and deletes consumer groups that use it's configured prefix.
That way, when an instance exits/restarts, previous consumer groups will be cleaned up quickly (check happens every 20s).


Expand All @@ -58,7 +65,7 @@ The end-to-end monitoring feature exports the following metrics.
### Counters
| Name | Description |
| --- | --- |
| `kminion_end_to_end_messages_produced_total ` | Messages kminion *tried* to send |
| `kminion_end_to_end_messages_produced_total ` | Messages KMinion *tried* to send |
| `kminion_end_to_end_messages_acked_total ` | Messages actually sent and acknowledged by the cluster |
| `kminion_end_to_end_messages_received_total ` | Number of messages received (only counts those that match, i.e. that this instance actually produced itself) |
| `kminion_end_to_end_commits_total` | Number of successful offset commits |
Expand All @@ -67,11 +74,9 @@ The end-to-end monitoring feature exports the following metrics.
### Histograms
| Name | Description |
| --- | --- |
| `kminion_end_to_end_produce_latency_seconds ` | Duration until a the cluster acknowledged a message. |
| `kminion_end_to_end_produce_latency_seconds ` | Duration until the cluster acknowledged a message. |
| `kminion_end_to_end_commit_latency_seconds` | Duration of offset commits. Has a label for coordinator brokerID that answered the commit request |
| `kminion_end_to_end_roundtrip_latency_seconds ` | Duration from creation of a message, until it was received/consumed agagin. |


| `kminion_end_to_end_roundtrip_latency_seconds ` | Duration from creation of a message, until it was received/consumed again. |

## Config Properties
All config properties related to this feature are located in `minion.endToEnd`.
Expand All @@ -83,7 +88,7 @@ All config properties related to this feature are located in `minion.endToEnd`.
topicManagement:
# You can disable topic management, without disabling the testing feature.
# Only makes sense if you have multiple kminion instances, and for some reason only want one of them to create/configure the topic.
# It is strongly reccommended to leave this enabled.
# It is strongly recommended to leave this enabled.
enabled: true

# Name of the topic kminion uses to send its test messages
Expand Down

0 comments on commit 5982e2c

Please sign in to comment.