http_headers trouble

[KonicRUS]

Bug report

Dear all.
It is required to send data in InfluxDB on the languages other than English.

There is the parameter in configuration file telegraf.conf :
## Optional HTTP headers
# http_headers = {"X-Special-Header" = "Special-Value"}

If I set http_headers = {"Content-Type" = "text/plain; charset=UTF-8"} then the header of request, which is being sent to the server, becomes «Content-Type: text/plain» ignoring my settings.

I investigated that in code the customer headers are set from configuration and then the customer defined header is changed to "text/plain"

Please review the code and fix the bug or suggest the workaround.

Best regards.

Relevant telegraf.conf:

[[outputs.influxdb]]
...
http_headers = {"Content-Type" = "text/plain; charset=UTF-8"}
...
[[inputs.exec]]
commands = ["powershell -ExecutionPolicy UnRestricted C:/telegraf/scripts/OH_stats.ps1"]
timeout = "30s"
interval = "10s"
data_format = "json"
name_suffix = "_oh"
tag_keys = [
"service_type",
"point_name",
"gw_name",
"service_name",
"_state"
]

System info:

Ubuntu 16.04.3 LTS
Telegraf 1.4.5

Steps to reproduce:

1. powershell generate JSON
   [{
   "service_type": "Переводы",
   "point_name": "Интернет-банк",
   "gw_name": "Transfer Gateway",
   "service_name": "Перевод внутри карты",
   "_state": 4000,
   "op_count": 1
   }]
2. telegraf send request
   POST /write?db=telegraf HTTP/1.1
   Host: 10.0.9.75:8086
   Transfer-Encoding: chunked
   Content-Type: text/plain
   Accept-Encoding: gzip

body
exec_oh,point_name= ୥ - ,gw_name=Transfer\ Gateway,service_name= ॢ \ \ ,host=SP-011,_state=4000,service_type= ॢ op_count=1 1513255431000000000

3. value in DB
   1513255431000000000 4000 Transfer Gateway SP-011 1 ����୥�-���� ��ॢ�� ������ ����� ��ॢ���

[danielnelson]

We always set Content-Type: text/plain which is overwriting your custom setting. We can fix this to prefer custom headers, and I think we should send a charset as well which would make your setting the default.

However I'm not sure this is the cause of your error, I tried to duplicate using a sh script and it is stored correctly for me. I wonder if the data is not property encoded as utf-8, I see you are using powershell but then you say it is on Ubuntu 16.04, can you clarify what your system setup is, are you actually using powershell on Linux?

Is there any chance you are using a proxy between Telegraf and InfluxDB?

For reference here is my values in the database:

time                _state gw_name          host   op_count point_name    service_name         service_type
----                ------ -------          ----   -------- ----------    ------------         ------------
1513305622000000000 4000   Transfer Gateway loaner 1        Интернет-банк Перевод внутри карты Переводы

And here is my shell script used for testing:

#!/bin/bash
echo '[{
        "service_type": "Переводы",
        "point_name": "Интернет-банк",
        "gw_name": "Transfer Gateway",
        "service_name": "Перевод внутри карты",
        "_state": 4000,
        "op_count": 1
}]'

[KonicRUS]

Windows 2012
Telegraf 1.4.5
PowerShell (collects application metrics)

Ubuntu 16.04.3
InfluxDB
Grafana

On the Windows server, the Telegraf is installed as a service. It receives data from the PowerShell in the correct encoding. The encoding fails exactly when sending data from the Telegraf to InfluxDB.

If send data directly from Powershell to InfluxDB with a custom header "text / plain; charset = UTF-8", normal lines will be stored in the database.

Proxy server is not used. For testing, I set the proxy server and changed the headers to my values. In this case, the values in the correct encoding also fall into the database.

Sorry for bad english :)

[danielnelson]

My guess is that the corruption occurs on the input plugin side. You could test this by replacing the influxdb output with a file output and checking if they are correctly converted. In the meantime I'll fix the Content-Type header.

[danielnelson]

Fixed in #3593