http_response - HTTP Request Body seems doesn't works

[anthosz]

Relevant telegraf.conf

[[inputs.http_response]]
  urls = ["https://example.com/test.php"]
  method = "POST"
  body = '''
  {'fake':'data'}
  '''
  response_body_field = 'test'

Logs from Telegraf

2022-03-17T15:46:48Z I! Starting Telegraf 1.21.4

http_response,host=XXXX,method=POST,result=success,server=https://example.com/test.php,status_code=200 content_length=13i,http_response_code=200i,response_time=0.044556621,result_code=0i,result_type="success",test="array(0) {
}
" 1647532009000000000

System info

Telegraf 1.15.2 / Telegraf 1.21.4

Docker

No response

Steps to reproduce

1. Add a webserver with php
2. Add this php file

<?php
var_dump($_POST);

3. Try a curl

curl --request POST 'https://example.com/test.php' --form 'fake=data'
array(1) {
  ["fake"]=>
  string(4) "data"
}

4. Try with telegraf
5. The array output is empty

Expected behavior

PHP is supposed to receive the data during the POST

Actual behavior

PHP didn't receive any body data during the POST

Additional info

No response

[powersj]

Looking at server logs between the two requests the thing that stood out was the lack of content-type with the Telegraf response. With the curl request the content-type seen by the server appears to be:

Content-Type: application/x-www-form-urlencoded\r\n

The Telegraf request however does not set this as our "body" config option can take any random-string. I built Telegraf with a hard-coded value of:

	request.Header.Add("Content-Type", "application/x-www-form-urlencoded")

and got the following in return:

2022-03-17T17:24:27Z I! Starting Telegraf 1.22.0-7e652fdd
2022-03-17T17:24:27Z I! Loaded inputs: http_response
2022-03-17T17:24:27Z I! Loaded aggregators: 
2022-03-17T17:24:27Z I! Loaded processors: 
2022-03-17T17:24:27Z I! Loaded outputs: file
2022-03-17T17:24:27Z I! Tags enabled: 
2022-03-17T17:24:27Z I! [agent] Hang on, flushing any cached metrics before shutdown
http_response,method=POST,result=success,server=http://192.168.100.209/test.php,status_code=200 response_time=0.000789019,http_response_code=200i,response="array(1) {
  [\"fake\"]=>
  string(4) \"data\"
}

",content_length=46i,result_type="success",result_code=0i 1647537868000000000

Next steps: I think the possible options here would be to let the user set the content-type or set headers, like we do on the HTTP plugin:

  ## Optional HTTP headers
  # headers = {"X-Special-Header" = "Special-Value"}

[aratik711]

@powersj

[inputs.http_response.headers]
     Host = "github.com"

This won't work ?

[powersj]

@aratik711

Correct, that is not a valid URL, see https://goplay.tools/snippet/ADk_8jSijTn

Edit: I should add that the code uses url.Parse and then goes on to ensure that HTTP or HTTPS is specified, which gets us something closer to url.ParseRequestURI

[aratik711]

@powersj I meant setting the Content-Type header in

[inputs.http_response.headers]

won't work?

Referring to your comment:

Next steps: I think the possible options here would be to let the user set the content-type or set headers, like we do on the HTTP plugin:

  ## Optional HTTP headers
  # headers = {"X-Special-Header" = "Special-Value"}
  

Do we need to add support for headers config in http_response plugin or the existing [inputs.http_response.headers] should work ?

[anthosz]

I used inputs.http_response.headers like workaround but doesn't help

[[inputs.http_response]]
  urls = ["https://example.com/test.php"]
  method = "POST"
  body = '''
  {'fake':'data'}
  '''
  response_body_field = 'test'
  [inputs.http_response.headers]
    Content-Type = "application/x-www-form-urlencoded"

Result:

test="array(1) {
  [\"{'fake':_'data'}
__\"]=>
  string(0) \"\"

OR (with & without boundary)

[[inputs.http_response]]
  urls = ["https://example.com/test.php"]
  method = "POST"
  body = '''
  {'fake':'data'}
  '''
  response_body_field = 'test'
  [inputs.http_response.headers]
    Content-Type = "multipart/form-data; boundary=----------------------------XXXXXX"

Result:
empty

Can be interesting to add a parameter to be able to choose the type of data that we want to send or something like that to be flexible :)

[powersj]

edit: sorry, still waking up...

This won't work ?

Correct, that is for the response header not for sending something in a header.

[anthosz]

edit: sorry, still waking up...

This won't work ?

Correct, that is for the response header not for sending something in a header.

Heu it works, it's not the response header but the header set for http_response input ^^

You can try to add an Authorization and it will be sent to your backend.

But concerning the Content-Type (to send POST datas), seems not really stable

[powersj]

Heu it works, it's not the response header but the header set for http_response input ^^

ah, we need to update the docs to make this more clear.

Does this at least resolve your original issue?

[anthosz]

Heu it works, it's not the response header but the header set for http_response input ^^

ah, we need to update the docs to make this more clear.

Does this at least resolve your original issue?

Nop :/ If I try to add the header, I have a different result but the format is really strange.
Via curl:

array(1) {
  ["fake"]=>
  string(4) "data"
}

Via telegraf + header:

test="array(1) {
  [\"{'fake':_'data'}
__\"]=>
  string(0) \"\"

Concerning the doc, probably I can do a PR but concerning the code, unfortunately, it's another story :/

[anthosz]

Hello,

Do you have an estimation when this issue can be fixed?

Best regards,

[powersj]

Do you have an estimation when this issue can be fixed?

Trying to catch back up on this, I believe this is a feature request to add the ability to post a form, something the plugin does not do currently.

Right now the function takes one of the http methods to create a request and send it. To replicate what you are doing with curl, the use of the PostForm method, which uses the right header (e.g. golang post vs postform) and urlencodes the body as keys and values.

No estimate, but anyone in the community free to put up a PR with a new setting to allow for this and we can review.