feat(outputs.opensearch): opensearch output plugin

plugins/outputs/all/opensearch.go

@@ -0,0 +1,5 @@
+//go:build !custom || outputs || outputs.opensearch
+
+package all
+
+import _ "github.com/influxdata/telegraf/plugins/outputs/opensearch" // register plugin

plugins/outputs/opensearch/README.md

@@ -0,0 +1,353 @@
+# OpenSearch Output Plugin
+
+This plugin writes to [OpenSearch](https://opensearch.org/) via HTTP
+
+It supports OpenSearch releases from 1 and 2. Future comparability with 1.x is
+not guaranteed and instead will focus on 2.x support. Consider using the
+existing Elasticsearch plugin for 1.x.
+
+## Global configuration options <!-- @/docs/includes/plugin_config.md -->
+
+In addition to the plugin-specific configuration settings, plugins support
+additional global and plugin configuration settings. These settings are used to
+modify metrics, tags, and field or create aliases and configure ordering, etc.
+See the [CONFIGURATION.md][CONFIGURATION.md] for more details.
+
+[CONFIGURATION.md]: ../../../docs/CONFIGURATION.md#plugins
+
+## Configuration
+
+```toml @sample.conf
+# Configuration for OpenSearch to send metrics to.
+[[outputs.OpenSearch]]
+  ## URLs
+  ## The full HTTP endpoint URL for your OpenSearch instance. Multiple URLs can
+  ## be specified as part of the same cluster, but only one URLs is used to
+  ## write during each interval.
+  urls = ["http://node1.os.example.com:9200"]
+
+  ## Index Name
+  ## Target index name for metrics (OpenSearch will create if it not exists).
+  ## This is a Golang template (see https://pkg.go.dev/text/template)
+  ## You can also specify
+  ## metric name (`{{.Name}}`), tag value (`{{.Tag "tag_name"}}`), field value (`{{.Field "feild_name"}}`) 
+  ## If the tag does not exist, the default tag value will be empty string "".
+  ## the timestamp (`{{.Time.Format "xxxxxxxxx"}}`).
+  ## For example: "telegraf-{{.Time.Format "2006-01-02"}}-{{.Tag "host"}}" would set it to telegraf-2023-07-27-HostName
+  index_name = ""
+
+  ## Timeout
+  ## OpenSearch client timeout
+  # timeout = "5s"
+
+  ## Sniffer
+  ## Set to true to ask OpenSearch a list of all cluster nodes,
+  ## thus it is not necessary to list all nodes in the urls config option
+  # enable_sniffer = false
+
+  ## GZIP Compression
+  ## Set to true to enable gzip compression
+  # enable_gzip = false
+
+  ## Health Check Interval
+  ## Set the interval to check if the OpenSearch nodes are available
+  ## Setting to "0s" will disable the health check (not recommended in production)
+  # health_check_interval = "10s"
+
+  ## Set the timeout for periodic health checks.
+  # health_check_timeout = "1s"
+  ## HTTP basic authentication details.
+  # username = ""
+  # password = ""
+  ## HTTP bearer token authentication details
+  # auth_bearer_token = ""
+
+  ## Optional TLS Config
+  # tls_ca = "/etc/telegraf/ca.pem"
+  # tls_cert = "/etc/telegraf/cert.pem"
+  # tls_key = "/etc/telegraf/key.pem"
+  ## Use TLS but skip chain & host verification
+  # insecure_skip_verify = false
+
+  ## Template Config
+  ## Manage templates
+  ## Set to true if you want telegraf to manage its index template.
+  ## If enabled it will create a recommended index template for telegraf indexes
+  # manage_template = true
+
+  ## Template Name
+  ## The template name used for telegraf indexes
+  # template_name = "telegraf"
+
+  ## Overwrite Templates
+  ## Set to true if you want telegraf to overwrite an existing template
+  # overwrite_template = false
+
+  ## Document ID
+  ## If set to true a unique ID hash will be sent as
+  ## sha256(concat(timestamp,measurement,series-hash)) string. It will enable
+  ## data resend and update metric points avoiding duplicated metrics with
+  ## different id's
+  # force_document_id = false
+
+  ## Value Handling
+  ## Specifies the handling of NaN and Inf values.
+  ## This option can have the following values:
+  ##    none    -- do not modify field-values (default); will produce an error
+  ##               if NaNs or infs are encountered
+  ##    drop    -- drop fields containing NaNs or infs
+  ##    replace -- replace with the value in "float_replacement_value" (default: 0.0)
+  ##               NaNs and inf will be replaced with the given number, -inf with the negative of that number
+  # float_handling = "none"
+  # float_replacement_value = 0.0
+
+  ## Pipeline Config
+  ## To use a ingest pipeline, set this to the name of the pipeline you want to use.
+  # use_pipeline = "my_pipeline"
+
+  ## Pipeline Name
+  ## Additionally, you can specify a tag name using the notation (`{{.Tag "tag_name"}}`)
+  ## If the tag does not exist, the default tag value will be used.
+  # default_tag_value = ""
+  ## If the tag does not exist, the default pipeline will be used as the
+  ## pipeline. If no default pipeline is set, no pipeline is used for the
+  ## metric.
+  # default_pipeline = ""
+```
+
+### Required parameters
+
+* `urls`: A list containing the full HTTP URL of one or more nodes from your
+  OpenSearch instance.
+* `index_name`: The target index for metrics. You can use the date format
+
+For example: "telegraf-{{.Time.Format "2006-01-02"}}" would set it to telegraf-2023-07-27
+You can also specify
+metric name (`{{Name}}`), tag value (`{{Tag "tag_name"}}`), field value (`{{Field "feild_name"}}`) 
+If the tag does not exist, the default tag value will be used. default_tag_value = ""
+
+## Permissions
+
+If you are using authentication within your OpenSearch cluster, you need to
+create an account and create a role with at least the manage role in the Cluster
+Privileges category. Otherwise, your account will not be able to connect to your
+OpenSearch cluster and send logs to your cluster.  After that, you need to
+add "create_index" and "write" permission to your specific index pattern.
+
+## OpenSearch indexes and templates
+
+### Indexes per time-frame
+
+This plugin can manage indexes per time-frame, as commonly done in other tools
+with OpenSearch. The timestamp of the metric collected will be used to decide
+the index destination. For more information about this usage on OpenSearch,
+check [the docs][1].
+
+[1]: https://opensearch.org/docs/latest/
+
+### Template management
+
+Index templates are used in OpenSearch to define settings and mappings for
+the indexes and how the fields should be analyzed.  For more information on how
+this works, see [the docs][2].
+
+This plugin can create a working template for use with telegraf metrics. It uses
+OpenSearch dynamic templates feature to set proper types for the tags and
+metrics fields.  If the template specified already exists, it will not overwrite
+unless you configure this plugin to do so. Thus you can customize this template
+after its creation if necessary.
+
+Example of an index template created by telegraf on OpenSearch 2.x:
+
+```json
+{
+  "telegraf-2022.10.02" : {
+    "aliases" : { },
+    "mappings" : {
+      "properties" : {
+        "@timestamp" : {
+          "type" : "date"
+        },
+        "disk" : {
+          "properties" : {
+            "free" : {
+              "type" : "long"
+            },
+            "inodes_free" : {
+              "type" : "long"
+            },
+            "inodes_total" : {
+              "type" : "long"
+            },
+            "inodes_used" : {
+              "type" : "long"
+            },
+            "total" : {
+              "type" : "long"
+            },
+            "used" : {
+              "type" : "long"
+            },
+            "used_percent" : {
+              "type" : "float"
+            }
+          }
+        },
+        "measurement_name" : {
+          "type" : "text",
+          "fields" : {
+            "keyword" : {
+              "type" : "keyword",
+              "ignore_above" : 256
+            }
+          }
+        },
+        "tag" : {
+          "properties" : {
+            "cpu" : {
+              "type" : "text",
+              "fields" : {
+                "keyword" : {
+                  "type" : "keyword",
+                  "ignore_above" : 256
+                }
+              }
+            },
+            "device" : {
+              "type" : "text",
+              "fields" : {
+                "keyword" : {
+                  "type" : "keyword",
+                  "ignore_above" : 256
+                }
+              }
+            },
+            "host" : {
+              "type" : "text",
+              "fields" : {
+                "keyword" : {
+                  "type" : "keyword",
+                  "ignore_above" : 256
+                }
+              }
+            },
+            "mode" : {
+              "type" : "text",
+              "fields" : {
+                "keyword" : {
+                  "type" : "keyword",
+                  "ignore_above" : 256
+                }
+              }
+            },
+            "path" : {
+              "type" : "text",
+              "fields" : {
+                "keyword" : {
+                  "type" : "keyword",
+                  "ignore_above" : 256
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "settings" : {
+      "index" : {
+        "creation_date" : "1664693522789",
+        "number_of_shards" : "1",
+        "number_of_replicas" : "1",
+        "uuid" : "TYugdmvsQfmxjzbGRJ8FIw",
+        "version" : {
+          "created" : "136247827"
+        },
+        "provided_name" : "telegraf-2022.10.02"
+      }
+    }
+  }
+}
+
+```
+
+[2]: https://opensearch.org/docs/latest/opensearch/index-templates/
+
+### Example events
+
+This plugin will format the events in the following way:
+
+```json
+{
+  "@timestamp": "2017-01-01T00:00:00+00:00",
+  "measurement_name": "cpu",
+  "cpu": {
+    "usage_guest": 0,
+    "usage_guest_nice": 0,
+    "usage_idle": 71.85413456197966,
+    "usage_iowait": 0.256805341656516,
+    "usage_irq": 0,
+    "usage_nice": 0,
+    "usage_softirq": 0.2054442732579466,
+    "usage_steal": 0,
+    "usage_system": 15.04879301548127,
+    "usage_user": 12.634822807288275
+  },
+  "tag": {
+    "cpu": "cpu-total",
+    "host": "opensearhhost",
+    "dc": "datacenter1"
+  }
+}
+```
+
+```json
+{
+  "@timestamp": "2017-01-01T00:00:00+00:00",
+  "measurement_name": "system",
+  "system": {
+    "load1": 0.78,
+    "load15": 0.8,
+    "load5": 0.8,
+    "n_cpus": 2,
+    "n_users": 2
+  },
+  "tag": {
+    "host": "opensearhhost",
+    "dc": "datacenter1"
+  }
+}
+```
+
+## Known issues
+
+Integer values collected that are bigger than 2^63 and smaller than 1e21 (or in
+this exact same window of their negative counterparts) are encoded by golang
+JSON encoder in decimal format and that is not fully supported by OpenSearch
+dynamic field mapping. This causes the metrics with such values to be dropped in
+case a field mapping has not been created yet on the telegraf index. If that's
+the case you will see an exception on OpenSearch side like this:
+
+```json
+{
+  "error": {
+    "root_cause": [
+      {"type": "mapper_parsing_exception", "reason": "failed to parse"}
+    ],
+    "type": "mapper_parsing_exception",
+    "reason": "failed to parse",
+    "caused_by": {
+      "type": "illegal_state_exception",
+      "reason": "No matching token for number_type [BIG_INTEGER]"
+    }
+  },
+  "status": 400
+}
+```
+
+The correct field mapping will be created on the telegraf index as soon as a
+supported JSON value is received by OpenSearch, and subsequent insertions
+will work because the field mapping will already exist.
+
+This issue is caused by the way OpenSearch tries to detect integer fields,
+and by how golang encodes numbers in JSON. There is no clear workaround for this
+at the moment.