Add streaming documentation (#1728)

* Add streaming documentation

* Address comments

README.md

@@ -66,10 +66,11 @@ These and other terminology are explained in the [glossary](docs/GLOSSARY.md).
 
 The key features of Snap are:
 
-* **Plugin Architecture**: Snap has a simple and smart modular design. The three types of plugins (collectors, processors, and publishers) allow Snap to mix and match functionality based on user need. All plugins are designed with versioning, signing and deployment at scale in mind. The **open plugin model** allows for loading built-in, community, or proprietary plugins into Snap.
+* **Plugin Architecture**: Snap has a simple and smart modular design. The four types of plugins (collectors, processors, publishers and streaming collectors) allow Snap to mix and match functionality based on user need. All plugins are designed with versioning, signing and deployment at scale in mind. The **open plugin model** allows for loading built-in, community, or proprietary plugins into Snap.
   * **Collectors** - Collectors consume telemetry data. Collectors are plugins for leveraging existing telemetry solutions (Facter, CollectD, Ohai) as well as specific plugins for consuming Intel telemetry (Node, DCM, NIC, Disk) and can reach into new architectures through additional plugins (see [Plugin Authoring below](#author-a-plugin)). Telemetry data is organized into a dynamically generated catalog of available data points.
   * **Processors** - Extensible workflow injection. Convert telemetry into another data model for consumption by existing systems. Allows encryption of all or part of the telemetry payload before publishing. Inject remote queries into workflow for tokens, filtering, or other external calls. Implement filtering at an agent level reducing injection load on telemetry consumer.
   * **Publishers** - Store telemetry into a wide array of systems. Snap decouples the collection of telemetry from the implementation of where to send it. Snap comes with a large library of publisher plugins that allow exposure to telemetry analytics systems both custom and common. This flexibility allows Snap to be valuable to open source and commercial ecosystems alike by writing a publisher for their architectures.
+  * **Streaming Collectors** - Streaming collectors act just like collectors, but rather than waiting for a specified duration before sending the collected data, they send it immediately over a grpc stream to snaptel. Check out [STREAMING.md](/docs/STREAMING.md) for more details on the differences between collectors and streaming collectors. 
 
 * **Dynamic Updates**: Snap is designed to evolve. Each scheduled workflow automatically uses the most mature plugin for that step, unless the collection is pinned to a specific version (e.g. get `/intel/psutil/load/load1/v1`). Loading a new plugin automatically upgrades running workflows in tasks. Load plugins dynamically, without a restart to the service or server. This dynamically extends the metric catalog when loaded, giving access to new measurements immediately. Swapping a newer version plugin for an old one in a safe transaction. All of these behaviors allow for simple and secure bug fixes, security patching, and improving accuracy in production.
 
@@ -304,7 +305,7 @@ When you're ready to move on, walk through other uses of Snap available in the [
 Documentation for building a task can be found [here](docs/TASKS.md).
 
 ### Plugin Catalog
-All known plugins are tracked in the [plugin catalog](https://github.com/intelsdi-x/snap/blob/master/docs/PLUGIN_CATALOG.md) and are tagged as collectors, processors and publishers.
+All known plugins are tracked in the [plugin catalog](https://github.com/intelsdi-x/snap/blob/master/docs/PLUGIN_CATALOG.md) and are tagged as collectors, processors, publishers and streaming collectors.
 
 If you would like to write your own, read through [Author a Plugin](#author-a-plugin) to get started. Let us know if you begin to write one by [joining our Slack channel](https://intelsdi-x.herokuapp.com/). When you finish, please open a Pull Request to add yours to the catalog!
 

docs/PLUGIN_AUTHORING.md

@@ -39,11 +39,12 @@ Before writing a new Snap plugin, please check out the [Plugin Catalog](./PLUGIN
 
 ### Plugin Type
 
-Snap supports three type of plugins:
+Snap supports four types of plugins:
 
-* collector: gathering metrics
+* collector: gathering metrics based on the specified interval
 * processor: transforming metrics
 * publisher: publishing metrics
+* streaming collector: gathering metrics when they are available 
 
 ### Plugin Name
 

docs/STREAMING.md

@@ -0,0 +1,116 @@
+# Streaming 
+
+
+Streaming plugins use grpc streams to allow the plugin to send data immediately or after a certain period of time instead of on an interval governed by Snap.
+Streaming by snap enables - 
+* Improved performance by enabling event based data flows
+* Runtime configuration controlling event throughput
+* Buffer configurations which dispatch events after a given duration and/or when a given event count has been reached
+
+Currently there are two plugins that support streaming - [snap relay](https://github.com/intelsdi-x/snap-relay) and [snap-plugin-collector-rand-streaming](https://github.com/intelsdi-x/snap-plugin-lib-go/tree/master/examples/snap-plugin-collector-rand-streaming). 
+
+# Configuration options 
+MaxCollectDuration and MaxMetricsBuffer are two configuration options that can be set through streaming task manifest or flags. 
+* MaxCollectDuration sets the maximum duration between collections before metrics are sent. It is always greater than 0 and defaults to 10s which means that after 10 seconds if no new metrics are received, the plugin should send whatever data it has in the buffer.
+* MaxMetricsBuffer is the maximum number of metrics the plugin is buffering before sending metrics. It defaults to 0 which means the metrics are sent immediately.  
+
+# Streaming task schedule
+```
+---
+  version: 1
+  schedule:
+    type: "streaming"
+  workflow:
+    collect:
+      metrics:
+        /random/integer: {}
+      config:
+        /random/integer:
+          MaxCollectDuration: "6s"
+          MaxMetricsBuffer: 600
+```
+
+# Streaming configuration flags
+Below is an example of the how to run the snap-relay using the configurable flags. 
+1. Start the Snap daemon:
+* Run
+```
+$ snapteld -l 1 -t 0
+```
+The option "-l 1" is for setting the debugging log level and "-t 0" is for disabling plugin signing.
+
+2. Start the relay using flags: 
+* Run snap-relay plugin
+```
+$ go run main.go --stand-alone --stand-alone-port 8182 --log-level 5 --max-collect-duration 10s --max-metrics-buffer 50
+
+Output: Preamble URL: [::]:8182
+```
+
+3. Obtain meta information:
+* Curl preamble
+```
+$ curl localhost:8182
+{"Meta":{"Type":3,"Name":"plugin-relay","Version":1,"RPCType":3,"RPCVersion":1,"ConcurrencyCount":5,"Exclusive":false,"Unsecure":true,"CacheTTL":0,"RoutingStrategy":0,"CertPath":"","KeyPath":"","TLSEnabled":false,"RootCertPaths":""},"ListenAddress":"127.0.0.1:54541","PprofAddress":"0","Type":3,"State":0,"ErrorMessage":""}
+```
+
+4. Start client in snap-relay:
+* Run client (from a different terminal) using the listen address obtained from the previous step
+```
+$ go run client/main.go 127.0.0.1:54541
+```
+
+5. Load plugin:
+```
+$ snaptel plugin load http://localhost:8182
+Plugin loaded
+Name: plugin-relay
+Version: 1
+Type: streaming-collector
+Signed: false
+Loaded Time: Tue, 05 Sep 2017 17:41:42 PDT
+```
+
+* List the metric catalog by running:
+```
+$ snaptel metric list
+NAMESPACE 		           VERSIONS
+/intel/relay/collectd 	           1
+/intel/relay/statsd 	           1
+```
+
+6. Create a task manifest for snap-relay (see [exemplary files](https://github.com/intelsdi-x/snap-relay/tree/master/examples/tasks)):
+```
+---
+  version: 1
+  schedule:
+    type: "streaming"
+  workflow:
+    collect:
+      metrics:
+       /intel/relay/collectd: {}
+```
+
+7. Create a task:
+```
+$ snaptel task create -t collectd.yaml
+Using task manifest to create task
+Task created
+ID: c168b992-8aaf-4eec-8e3c-883510962789
+Name: Task-c168b992-8aaf-4eec-8e3c-883510962789
+State: Running
+```
+
+# Metrics exposed by streaming collectors
+Below are some of the metrics collected by the streaming plugins currently: 
+```
+/intel/relay/collectd
+/intel/relay/statsd
+/random/integer
+/random/float
+/random/string
+```
+
+
+
+

docs/TASKS.md

@@ -52,6 +52,7 @@ The schedule describes the schedule type and interval for running the task. At t
  - [simple](#simple-schedule) 
  - [windowed](#windowed-schedule) 
  - [cron](#cron-schedule)
+ - [streaming] (#streaming-schedule)
 
 Snap is designed in a way where custom schedulers can easily be dropped in. If a custom schedule is used, it may require more key/value pairs in the schedule section of the manifest.  
 
@@ -176,8 +177,16 @@ Snap is designed in a way where custom schedulers can easily be dropped in. If a
       "max-failures": 10,
    ```
 
-
-
+##### Streaming Schedule
+```yaml
+   ---
+  version: 1
+  schedule:
+    type: "streaming"
+```
+The streaming schedule doesn't support fields such as `interval` and `count`. If those fields are provided as part of the schedule, they will simply be skipped. 
+For more details on streaming, visit [STREAMING.md](STREAMING.md)
+
 #### Max-Failures
 
 By default, Snap will disable a task if there are 10 consecutive errors from any plugins within the workflow.  The configuration

examples/README.md

@@ -33,3 +33,4 @@ For additional examples of using Snap, checkout the examples in these repositori
  - [snap-plugin-publisher-influxdb](https://github.com/intelsdi-x/snap-plugin-publisher-influxdb)
  - [snap-plugin-publisher-graphite](https://github.com/intelsdi-x/snap-plugin-publisher-graphite)
  - [snap-plugin-publisher-file](https://github.com/intelsdi-x/snap-plugin-publisher-file)
+ - [snap-relay](https://github.com/intelsdi-x/snap-relay) (streaming collector plugin to retrieve data from collectd or statsd and include them into Snap workflow)