Fixing intermittent panic on metric submission #92
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In certain (unknown) cases, submitting metrics via the Datadog API fails with the error `EOF` and a `nil` http response. It appears that while the API config has a `compress` field, setting it to `true` does not actually enable compression. It only prevents it from disabling compression:
```golang
if !c.Cfg.Compress {
// gzip is on by default, so disable it by setting encoding to identity
localVarRequest.Header.Add("Accept-Encoding", "identity")
}
```
Despite the Datadog client indicating the gzip compression is on by default, an inspection of the code and dumping the request indicates that it is not. Manually setting the content encoding on each request is required to actually enable compression.
In some case the lack of compression caused a failure in the underlying http client which resulted in an `EOF` error. It's unclear what payload size or conditions resulted in this issue, but adding the content encoding resolved the issue.
- Added content encoding of `gzip` when `compress` is set to `true`
- Added a `nil` check on the http response to prevent trying to get the status code from it when it is `nil`
| if _, r, err := s.metricsApi.SubmitMetrics(ddCtx, *datadogV2.NewMetricPayload(dp[rangeStart:rangeEnd]), *datadogV2.NewSubmitMetricsOptionalParameters()); err != nil { | ||
| if r.StatusCode == http.StatusRequestEntityTooLarge { | ||
| if r != nil && r.StatusCode == http.StatusRequestEntityTooLarge { |
There was a problem hiding this comment.
Yeah when err is nil, there's usually no guarantee on the response. This is how built-in http works, at least.
daniel-nichter
approved these changes
Feb 1, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In certain (unknown) cases, submitting metrics via the Datadog API fails with the error
EOFand anilhttp response. It appears that while the Datadog API config has aCompressfield, setting it totruedoes not actually enable compression. It only prevents it from disabling compression:Despite the Datadog client indicating the gzip compression is on by default, an inspection of the code and dumping the request indicates that it is not. Manually setting the content encoding on each request is required to actually enable compression.
In some case the lack of compression caused a failure in the underlying http client which resulted in an
EOFerror. It's unclear what payload size or conditions resulted in this issue, but adding the content encoding resolved the issue.gzipwhencompressis set totruenilcheck on the http response to prevent trying to get the status code from it when it isnil