influxdb_v2 output does not queue metrics in the case of invalid token

[russorat]

Relevant telegraf.conf:

# Global tags can be specified here in key="value" format.
[global_tags]

# Configuration for telegraf agent
[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 100000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "10s"
  precision = ""
  debug = true
  quiet = false
  logfile = ""
  omit_hostname = false


[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = true
  report_active = true

[[outputs.influxdb_v2]]
  ## The URLs of the InfluxDB cluster nodes.
  ##
  ## Multiple URLs can be specified for a single cluster, only ONE of the
  ## urls will be written to each interval.
  urls = ["http://127.0.0.1:9999"]

  ## Token for authentication.
  token = "token"

  ## Organization is the name of the organization you wish to write to.
  organization = "influxdata"

  ## Bucket to the name fo the bucket to write into; must exist.
  bucket = "telegraf"

System info:

mac os
go build -ldflags " -X main.commit=e8248fcc -X main.branch=master" ./cmd/telegraf

Steps to reproduce:

1. configure telegraf to push metrics from the default plugins into an instance of influxdb v2
2. notice the metrics are written
3. disable the token being used to write metrics into the system
4. notice that while the token is disabled, the buffer size does not increase, even though data is not being written to the system.

Expected behavior:

When the output is unable to write metrics, they are stored in the internal buffer until it is able to write.

Actual behavior:

Metrics are dropped even if they are unable to write to influxdb v2

Additional info:

➜  telegraf git:(master) ✗ make
dep ensure -vendor-only
go build -ldflags " -X main.commit=e8248fcc -X main.branch=master" ./cmd/telegraf
➜  telegraf git:(master) ✗ ./telegraf --config-directory ~/telegraf.d/
2019-04-30T21:52:21Z I! Starting Telegraf
2019-04-30T21:52:21Z I! Using config file: /Users/rsavage/.telegraf/telegraf.conf
2019-04-30T21:52:21Z I! Loaded inputs: cpu disk diskio dns_query internal kernel mem net netstat ping processes procstat procstat procstat procstat smart swap system
2019-04-30T21:52:21Z I! Loaded aggregators:
2019-04-30T21:52:21Z I! Loaded processors:
2019-04-30T21:52:21Z I! Loaded outputs: influxdb_v2
2019-04-30T21:52:21Z I! Tags enabled: host=rsavage.prod
2019-04-30T21:52:21Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"rsavage.prod", Flush Interval:10s
2019-04-30T21:52:21Z D! [agent] Connecting outputs
2019-04-30T21:52:21Z D! [agent] Attempting connection to output: influxdb_v2
2019-04-30T21:52:21Z D! [agent] Successfully connected to output: influxdb_v2
2019-04-30T21:52:21Z D! [agent] Starting service inputs
2019-04-30T21:52:47Z D! [outputs.influxdb_v2] wrote batch of 498 metrics in 42.162282ms
2019-04-30T21:52:47Z D! [outputs.influxdb_v2] buffer fullness: 0 / 100000 metrics.
2019-04-30T21:52:59Z D! [outputs.influxdb_v2] wrote batch of 251 metrics in 29.995068ms
2019-04-30T21:52:59Z D! [outputs.influxdb_v2] buffer fullness: 0 / 100000 metrics.
2019-04-30T21:53:08Z E! [outputs.influxdb_v2] Failed to write metric: forbidden: insufficient permissions for write
2019-04-30T21:53:08Z D! [outputs.influxdb_v2] wrote batch of 251 metrics in 11.771334ms
2019-04-30T21:53:08Z D! [outputs.influxdb_v2] buffer fullness: 0 / 100000 metrics.
2019-04-30T21:53:17Z E! [outputs.influxdb_v2] Failed to write metric: forbidden: insufficient permissions for write
2019-04-30T21:53:17Z D! [outputs.influxdb_v2] wrote batch of 252 metrics in 7.336106ms
2019-04-30T21:53:17Z D! [outputs.influxdb_v2] buffer fullness: 0 / 100000 metrics.
2019-04-30T21:53:21Z E! [outputs.influxdb_v2] Failed to write metric: forbidden: insufficient permissions for write
2019-04-30T21:53:21Z D! [outputs.influxdb_v2] wrote batch of 125 metrics in 7.077869ms
2019-04-30T21:53:21Z D! [outputs.influxdb_v2] buffer fullness: 0 / 100000 metrics.
2019-04-30T21:53:32Z D! [outputs.influxdb_v2] wrote batch of 293 metrics in 45.792624ms
2019-04-30T21:53:32Z D! [outputs.influxdb_v2] buffer fullness: 2 / 100000 metrics.
2019-04-30T21:53:40Z D! [outputs.influxdb_v2] wrote batch of 176 metrics in 30.014755ms
2019-04-30T21:53:40Z D! [outputs.influxdb_v2] buffer fullness: 1 / 100000 metrics.