Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

POST request to http_listener fails with EOF #2280

Closed
salarali opened this issue Jan 17, 2017 · 14 comments · Fixed by #6974
Closed

POST request to http_listener fails with EOF #2280

salarali opened this issue Jan 17, 2017 · 14 comments · Fixed by #6974
Assignees
Labels
bug unexpected problem or unintended behavior
Milestone

Comments

@salarali
Copy link

Bug report

Intermittently the post request to the http_listener fails and I get the following error message:
Post http://localhost:8186/write: EOF

Relevant telegraf.conf:

Vanilla conf. with http_listener enabled.

@salarali salarali changed the title telegraf: POST request to http_listener fails with EOF POST request to http_listener fails with EOF Jan 17, 2017
@sparrc
Copy link
Contributor

sparrc commented Jan 17, 2017

what version are you using? do you have any steps to reproduce? can you provide more log context and the full log line?

@salarali
Copy link
Author

salarali commented Jan 17, 2017

Sorry for the lack of information.

Telegraf version: 1.1.2

I am making the http post request from a golang program.

resp, err := http.Post(API_URL, "", bytes.NewBuffer([]byte(metricsString)))

if err != nil || resp.StatusCode != 204 {
	logger.Error("Telegraf push failed: ", err)
} else {
	defer resp.Body.Close()
}

metricsString looks like this:

test num_error_no_category=122 1484680136892545372
test num_error_http_setup=0 1484680136892545372
test num_active_go_routines=2045 1484680136892545372
test num_error=1803 1484680136892545372
test num_duplicate=760 1484680136892545372

And this is all the log that go provides to me. Is there a way I can get more information?

I have turned debug logging on for telegraf but I do not see any errors there.

I am sending metrics to telegraf every 10 seconds. Currently, I have only experienced this error on production. I will try to reproduce the error locally.

@sparrc sparrc added this to the Future Milestone milestone Feb 3, 2017
@danielnelson
Copy link
Contributor

@salarali Any chance you have found a way to reproduce this?

@danielnelson danielnelson added the bug unexpected problem or unintended behavior label Apr 18, 2017
@danielnelson danielnelson removed this from the Future Milestone milestone Jun 14, 2017
@ncode
Copy link

ncode commented Jun 20, 2017

I am receiving the same error, but it only happens with 21+ Goroutines pushing metrics to telegraf. Also It never happens if you use the discard as output.

@danielnelson
Copy link
Contributor

Possibly related to #2919?

@jpereira
Copy link

I am experiencing the same issue in my setup.

2017-09-29T17:53:01Z E! InfluxDB Output Error: Post https://metrics.internal.lan:443/write?db=telegraf&rp=default: EOF

@danielnelson
Copy link
Contributor

@jpereira Just to clarify, is the error when you are sending to the http_listener in Telegraf, or when you are sending to InfluxDB?

@jpereira
Copy link

@danielegozzi Sending to the InfluxDB Cluster.

@danielnelson
Copy link
Contributor

Okay, this is a different issue. It indicates that InfluxDB closed the connection, I would check the InfluxDB logs to see if anything is reported.

@jpereira
Copy link

@danielegozzi I am using the latest version 1.4.1

@ILoGM
Copy link

ILoGM commented Apr 18, 2018

We're experiencing the same issue as @salarali did.
We have erlang app which sends metrics to telegraf via http locally. Sometimes it looks like http_listener just closes the connection. Just like that:

2018-04-17 16:08:17.147 Undefined [error] influx_client: HTTP sending error: {error,closed}
2018-04-17 16:08:17.147 Undefined [warning] influx_client: Reconnecting success: {http,<<"127.0.0.1">>,8186,undefined,undefined}

Telegraf debug logs say nothing useful about what happens. I'd rather see some info about max_body_size/max_line_size exceeding or something like that.

Version used:

telegraf --version
Telegraf v1.5.3 (git: release-1.5 1e519698)

We're still investigating.

@ILoGM
Copy link

ILoGM commented Apr 18, 2018

We figured out that we exceeded max_line_size.
I set it as twice as default value and disconnects disappeared.
But still, I believe that such kind of situations must be mentioned in logs, at least at debug level.

@danielnelson
Copy link
Contributor

Thanks for debugging this @ILoGM, I think we should return 413 Request Entity Too Large.

@russorat
Copy link
Contributor

russorat commented Aug 5, 2019

let's look at a refactor of the entire http_listener code for the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug unexpected problem or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants