Add additional stats providers

[StephenRadachy]

Influxdb is sweet, but not everyone is using it.

Here's Grafana's list of supported data sources (excluding influxdb):

• Graphite *
• Elasticsearch *
• CloudWatch
• OpenTSDB
• KairosDB
• Prometheus *

I starred the ones that I think should get priority.

[liclac]

I would love to see collectors for these contributed! Prometheus would be really easy, but how would time series work with ES?

[danron]

+1 on Prometheus.

[ghost]

Prometheus. InfluxDB can or soon be plugged into Prometheus.

[liclac]

If anyone wants to take a dig at this, all that's needed is an implementation of the lib.Collector interface (see stats/influxdb for an example), then you add that to cmd/collectors.go to allow -o yourcollector to work.

[ivoreis]

Hi @liclac, I'm considering using this tool but I'd need Datadog integration.

I've started a spike with some rough Datadog / StatsD code, do you have any recommendation / opinion about this?

Cheers

[liclac]

Start a PR and I'll be happy to give you feedback!

[V3ckt0r]

Hey guys,

Please see Prometheus feature at #415. You thoughts and feedback much appreciated

Thanks.

[SaberStrat]

@liclac
Regarding ES, you can insert the documents (one doc per http_request) containing a field of datatype "date". For ES itself, it's just a field of some type. You get a time-series out of it when visualizing it with, e.g.

Kibana:
tell Kibana that your indices of interest contain time-based events and pick that date field as the one to use for the timestamp. And you can do quite a lot time-series analysis with the Kibana-ES combo.

Grafana:
similar. Sadly, Grafana's ES interfacing doesn't quite support as much of ES's capabilities as Kibana does.

k6 is intriguing enough that I'd be interested in contributing with an ES output, although the crux being that I've never done Go before. Though I certainly wouldn't mind getting into it for this.

[na--]

Something else it might be worth to support: https://m3db.io/

[darrenhaken]

Any idea when Prometheus might be supported?

[mstoykov]

@darrenhaken There is a PR #478 but it has been somewhat on hold and it needs more work ... probably some redesign as prometheus does things differently from other stats providers.
We do have statsd support and supposedly there statsd -> prometheus exporter :) . If you try it can you please write us back if it works :)

[na--]

This issue could mostly be implemented by using telegraf as a library as described in #1064. The only output of the initial list that telegraf doesn't support is KairosDB, but there's an open pull request for it ...

[novettaberin]

@SaberStrat, if you do contribute support for ES, can you please support the Elastic Common Schema (ECS) as much as possible?

That would streamline setting up Kibana out of the box to make sense of the data coming in. The current versions of ES have some auto-configuration support if Kibana recognizes the content as ECS.

[TheSecMaven]

Yes! @bericoberin exactly. use ECS, and please get support for Elasticsearch! Also could use generic rest api. just send the results off in a POST to some endpoint specified in command line

[valyala]

There is also VictoriaMetrics, which can be used as stats provider. It supports many commonly used protocols for data ingestion (i.e. Prometheus remote_write, InfluxDB line protocol, OpenTSDB protocol, Graphite protocol, CSV, etc. - see https://victoriametrics.github.io/#how-to-import-time-series-data ) and it provides Prometheus-compatible querying API, so it can be used as drop-in replacement for Prometheus in Grafana - see https://victoriametrics.github.io/#prometheus-querying-api-usage and https://victoriametrics.github.io/#grafana-setup .

[codebien]

k6 v0.42.0 now supports Prometheus via the Prometheus remote write output.
Some of the other storages also support the Prometheus remote write protocol directly or using additional components. For example, it is the case for Elasticsearch using Metricbeat.

[olegbespalov]

The team decided to close this issue.

The rationale is that k6 will unlikely contain all possible outputs. Instead, we see that the xk6 extensions could already sort the issue and are a preferable way of having support for new output.