This repository has been archived by the owner on Nov 8, 2022. It is now read-only.
Add streaming documentation #1728
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
IzabellaRaulin
suggested changes
Sep 1, 2017
docs/PLUGIN_AUTHORING.md
Outdated
|
|
||
| * collector: gathering metrics | ||
| * processor: transforming metrics | ||
| * publisher: publishing metrics | ||
| * streaming collector: streaming gathered metrics |
There was a problem hiding this comment.
@rashmigottipati, could you rephrase this one. I think a good idea is to highlight that
collector: gathering metrics in provided interval
streaming collector gathering metrics when they available
You can add here also a notice that Snap supports in collection both mechanism of gathering metrics - pull and push.
docs/STREAMING.md
Outdated
| * Supports runtime configuration controlling event throughput | ||
| * Buffer configurations enable dispatching events after a given duration and/or when a given event count has been reached | ||
|
|
||
| Currently there are two plugins that support streaming - [snap relay] (https://github.com/intelsdi-x/snap-relay) and [snap-plugin-collector-rand-streaming] (https://github.com/intelsdi-x/snap-plugin-lib-go/tree/master/examples/snap-plugin-collector-rand-streaming). |
There was a problem hiding this comment.
change
[snap relay] (https://github.com/intelsdi-x/snap-relay)
to
snap relay
docs/STREAMING.md
Outdated
| 2. Start the plugin using flags: | ||
| * Run snap-plugin-collector-rand-streaming plugin | ||
| ``` | ||
| go run main.go --stand-alone --max-collect-duration 10s --max-metrics-buffer 50 |
There was a problem hiding this comment.
I think you can skip go run main.go because there is Makefile and binary can be found in '/build'
There was a problem hiding this comment.
I think it will be better to base on snap-relay as example and show real use case
docs/TASKS.md
Outdated
| schedule: | ||
| type: "streaming" | ||
| ``` | ||
| For more details on streaming, visit [STREAMING.md](STREAMING.md) |
There was a problem hiding this comment.
Could you add a sentence that there is no interval and count - those params will be skipped if they are provided
examples/README.md
Outdated
| @@ -33,3 +33,5 @@ For additional examples of using Snap, checkout the examples in these repositori | |||
| - [snap-plugin-publisher-influxdb](https://github.com/intelsdi-x/snap-plugin-publisher-influxdb) | |||
| - [snap-plugin-publisher-graphite](https://github.com/intelsdi-x/snap-plugin-publisher-graphite) | |||
| - [snap-plugin-publisher-file](https://github.com/intelsdi-x/snap-plugin-publisher-file) | |||
| - [snap-relay](https://github.com/intelsdi-x/snap-relay) (streaming collector plugin) | |||
| - [snap-plugin-collector-rand-streaming](https://github.com/intelsdi-x/snap-plugin-lib-go/blob/master/examples/snap-plugin-collector-rand-streaming) (streaming collector plugin) | |||
There was a problem hiding this comment.
This file contains real plugins, not mocked - so please remove snap-plugin-collector-rand-streaming from the list
examples/README.md
Outdated
| @@ -33,3 +33,5 @@ For additional examples of using Snap, checkout the examples in these repositori | |||
| - [snap-plugin-publisher-influxdb](https://github.com/intelsdi-x/snap-plugin-publisher-influxdb) | |||
| - [snap-plugin-publisher-graphite](https://github.com/intelsdi-x/snap-plugin-publisher-graphite) | |||
| - [snap-plugin-publisher-file](https://github.com/intelsdi-x/snap-plugin-publisher-file) | |||
| - [snap-relay](https://github.com/intelsdi-x/snap-relay) (streaming collector plugin) | |||
There was a problem hiding this comment.
in the end in brackets could you add sth like that:
(streaming collector plugin to retrieve data from collectd or statsd and include it into Snap workflow)
rashmigottipati
force-pushed
the
add-streaming-doc
branch
from
24d6ba1
to
bdb037f
Compare
rashmigottipati
force-pushed
the
add-streaming-doc
branch
from
bdb037f
to
3a9257c
Compare
kjlyon
reviewed
Sep 5, 2017
README.md
Outdated
| * **Collectors** - Collectors consume telemetry data. Collectors are plugins for leveraging existing telemetry solutions (Facter, CollectD, Ohai) as well as specific plugins for consuming Intel telemetry (Node, DCM, NIC, Disk) and can reach into new architectures through additional plugins (see [Plugin Authoring below](#author-a-plugin)). Telemetry data is organized into a dynamically generated catalog of available data points. | ||
| * **Processors** - Extensible workflow injection. Convert telemetry into another data model for consumption by existing systems. Allows encryption of all or part of the telemetry payload before publishing. Inject remote queries into workflow for tokens, filtering, or other external calls. Implement filtering at an agent level reducing injection load on telemetry consumer. | ||
| * **Publishers** - Store telemetry into a wide array of systems. Snap decouples the collection of telemetry from the implementation of where to send it. Snap comes with a large library of publisher plugins that allow exposure to telemetry analytics systems both custom and common. This flexibility allows Snap to be valuable to open source and commercial ecosystems alike by writing a publisher for their architectures. | ||
| * **Streaming Collectors** - Streaming collectors act just like collectors, but rather than waiting for a specified duration before sending the collected data, they send it immediatly over a grpc stream to snaptel. Check out [STREAMING.md](/docs/STREAMING.md) for more details on the differences between collectors and streaming collectors. |
docs/STREAMING.md
Outdated
| # Streaming | ||
|
|
||
|
|
||
| Uses grpc streams to allow the plugin to send data immediately or after a certain period of time instead of on an interval governed by Snap. |
There was a problem hiding this comment.
Can you edit this to start with, Streaming plugins use grpc streams to allow ...
docs/STREAMING.md
Outdated
| Uses grpc streams to allow the plugin to send data immediately or after a certain period of time instead of on an interval governed by Snap. | ||
| Streaming by snap enables - | ||
| * Improved performance by enabling event based data flows | ||
| * Supports runtime configuration controlling event throughput |
There was a problem hiding this comment.
Can you get rid of Supports and start the sentence with Runtime configuration to control event throughput
docs/STREAMING.md
Outdated
| Streaming by snap enables - | ||
| * Improved performance by enabling event based data flows | ||
| * Supports runtime configuration controlling event throughput | ||
| * Buffer configurations enable dispatching events after a given duration and/or when a given event count has been reached |
There was a problem hiding this comment.
Change enable dispatching to which dispatch
rashmigottipati
force-pushed
the
add-streaming-doc
branch
2 times, most recently
from
7484652
to
b22e2aa
Compare
rashmigottipati
force-pushed
the
add-streaming-doc
branch
from
b22e2aa
to
b678619
Compare
IzabellaRaulin
approved these changes
Sep 6, 2017
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add streaming documentation that includes details on exemplary task manifest, configuration options.
@intelsdi-x/snap-maintainers