-
Notifications
You must be signed in to change notification settings - Fork 514
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove high cardanility metrics from otelhttp #4277
Remove high cardanility metrics from otelhttp #4277
Conversation
Codecov Report
@@ Coverage Diff @@
## main #4277 +/- ##
======================================
Coverage 79.4% 79.5%
======================================
Files 166 166
Lines 10376 10677 +301
======================================
+ Hits 8246 8491 +245
- Misses 1996 2045 +49
- Partials 134 141 +7 |
instrumentation/github.com/gin-gonic/gin/otelgin/internal/semconvutil/httpconv.go
Show resolved
Hide resolved
instrumentation/github.com/gin-gonic/gin/otelgin/internal/semconvutil/httpconv.go
Outdated
Show resolved
Hide resolved
instrumentation/github.com/emicklei/go-restful/otelrestful/internal/semconvutil/httpconv.go
Show resolved
Hide resolved
Make the otelhttp use the new function Add test for new function.
Weird. Why @dmathieu 's previous suggestion is not applied to this PR:
And what's the community's suggestion towards similar attributes? My own instrumentation library also needs some |
This PR solves from a potential memory leak issue, since the values from those attributes are coming from untrusted sources. |
@dmathieu @pellared @MadVikingGod do you plan to add a similar solution to otelgrpc to avoid high cardinality there as well? (I think reported here: #3071) |
…elhttp to v0.44.0 [SECURITY] (main) (grafana#11002) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp](https://github.com/open-telemetry/opentelemetry-go-contrib) | indirect | minor | `v0.42.0` -> `v0.44.0` | ### GitHub Vulnerability Alerts #### [CVE-2023-45142](https://github.com/open-telemetry/opentelemetry-go-contrib/security/advisories/GHSA-5r5m-65gx-7vrh) ### Summary This handler wrapper https://github.com/open-telemetry/opentelemetry-go-contrib/blob/5f7e6ad5a49b45df45f61a1deb29d7f1158032df/instrumentation/net/http/otelhttp/handler.go#L63-L65 out of the box adds labels - `http.user_agent` - `http.method` that have unbound cardinality. It leads to the server's potential memory exhaustion when many malicious requests are sent to it. ### Details HTTP header User-Agent or HTTP method for requests can be easily set by an attacker to be random and long. The library internally uses [httpconv.ServerRequest](https://github.com/open-telemetry/opentelemetry-go/blob/v1.12.0/semconv/internal/v2/http.go#L159) that records every value for HTTP [method](https://github.com/open-telemetry/opentelemetry-go/blob/38e1b499c3da3107694ad2660b3888eee9c8b896/semconv/internal/v2/http.go#L204) and [User-Agent](https://github.com/open-telemetry/opentelemetry-go/blob/38e1b499c3da3107694ad2660b3888eee9c8b896/semconv/internal/v2/http.go#L223). ### PoC Send many requests with long randomly generated HTTP methods or/and User agents (e.g. a million) and observe how memory consumption increases during it. ### Impact In order to be affected, the program has to configure a metrics pipeline, use [otelhttp.NewHandler](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/5f7e6ad5a49b45df45f61a1deb29d7f1158032df/instrumentation/net/http/otelhttp/handler.go#L63-L65) wrapper, and does not filter any unknown HTTP methods or User agents on the level of CDN, LB, previous middleware, etc. ### Others It is similar to already reported vulnerabilities - GHSA-5r5m-65gx-7vrh ([open-telemetry/opentelemetry-go-contrib](https://github.com/open-telemetry/opentelemetry-go-contrib)) - GHSA-cg3q-j54f-5p7p ([prometheus/client_golang](https://github.com/prometheus/client_golang)) ### Workaround for affected versions As a workaround to stop being affected [otelhttp.WithFilter()](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp/filters) can be used, but it requires manual careful configuration to not log certain requests entirely. For convenience and safe usage of this library, it should by default mark with the label `unknown` non-standard HTTP methods and User agents to show that such requests were made but do not increase cardinality. In case someone wants to stay with the current behavior, library API should allow to enable it. The other possibility is to disable HTTP metrics instrumentation by passing [`otelhttp.WithMeterProvider`](https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp#WithMeterProvider) option with [`noop.NewMeterProvider`](https://pkg.go.dev/go.opentelemetry.io/otel/metric/noop#NewMeterProvider). ### Solution provided by upgrading In PR [open-telemetry/opentelemetry-go-contrib#4277, released with package version 0.44.0, the values collected for attribute `http.request.method` were changed to be restricted to a set of well-known values and other high cardinality attributes were removed. ### References - [open-telemetry/opentelemetry-go-contrib#4277 - https://github.com/open-telemetry/opentelemetry-go-contrib/releases/tag/v1.19.0 --- ### Configuration 📅 **Schedule**: Branch creation - "" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/grafana/loki). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xOS4yIiwidXBkYXRlZEluVmVyIjoiMzcuMzEuNSIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This does not upgrade to the latest semantic convention, that will be a different PR.
This removes the following attributes from the metrics that
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
generates:net.sock.peer.addr
net.sock.peer.port
http.user_agent
enduser.id
http.client_ip
This also only allows
http.request.method
to be known HTTP methods, or_OTHER
.Resolve open-telemetry/opentelemetry-go#3765