Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Jaeger v1.59 doesn't accept OpenTelemetry data on ports 4317/4318 #5737

Closed
mpbnkg opened this issue Jul 12, 2024 · 5 comments · Fixed by #5739
Closed

[Bug]: Jaeger v1.59 doesn't accept OpenTelemetry data on ports 4317/4318 #5737

mpbnkg opened this issue Jul 12, 2024 · 5 comments · Fixed by #5739

Comments

@mpbnkg
Copy link

mpbnkg commented Jul 12, 2024

See this ticket for more info - open-telemetry/opentelemetry-collector-contrib#33995

What happened?

When I try to export metrics to jaeger via OTLP collector it fails with the following error. What is causing this given the fact that it was working a few weeks back?

2024-07-09T18:35:39.864Z info exporterhelper/retry_sender.go:118 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 172.25.0.2:4317: connect: connection refused"", "interval": "3.136185958s"}

@mpbnkg mpbnkg added the bug label Jul 12, 2024
@dosubot dosubot bot added the area/otel label Jul 12, 2024
@PedemonteGiacomo
Copy link

Yes I have the same problem that we have already discussed here.

With older image version like the 1.58 of the jaeger-all-in-one it works fine.

@yurishkuro
Copy link
Member

yurishkuro commented Jul 12, 2024

Most likely due to recent OTEL Collector change to default receivers to use localhost instead of 0.0.0.0. There is an upstream issue to revert this change open-telemetry/opentelemetry-operator#3126

The workaround is to pass host IP explicitly to Jaeger, e.g. via these env variables, as in this fix:

      - COLLECTOR_OTLP_GRPC_HOST_PORT=0.0.0.0:4317
      - COLLECTOR_OTLP_HTTP_HOST_PORT=0.0.0.0:4318

Alternatively, refer to OTEL guidelines https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks

@mpbnkg
Copy link
Author

mpbnkg commented Jul 13, 2024

@yurishkuro Yep, it works when I added those two lines :)

yurishkuro added a commit that referenced this issue Jul 13, 2024
## Which problem is this PR solving?
- Resolves #5737

## Description of the changes
- Revert #5734
- Default OTLP receiver endpoints to `:port` instead of relying on OTEL
Collector defaults
- Clean up tests to use ephemeral ports when possible

## How was this change tested?
- Tested similar to #5734 by using `COLLECTOR_OTLP_HTTP_HOST_PORT=:4318`

---------

Signed-off-by: Yuri Shkuro <github@ysh.us>
haines added a commit to cerbos/cerbos-sdk-javascript that referenced this issue Jul 15, 2024
haines added a commit to cerbos/cerbos-sdk-javascript that referenced this issue Jul 15, 2024
* chore(deps-dev): update development dependencies

Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

* Update transitive dependencies

Signed-off-by: Andrew Haines <haines@cerbos.dev>

* Configure Jaeger's OTLP collector to listen on all addresses

jaegertracing/jaeger#5737

Signed-off-by: Andrew Haines <haines@cerbos.dev>

---------

Signed-off-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Signed-off-by: Andrew Haines <haines@cerbos.dev>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Andrew Haines <haines@cerbos.dev>
@yurishkuro
Copy link
Member

In the next release we are going back to listening on all IPs.

@yurishkuro yurishkuro changed the title [Bug]: Latest version of jaeger doesn't accept open telemetry data on port 4317. [Bug]: Jaeger v1.59 doesn't accept OpenTelemetry data on ports 4317/4318 Jul 22, 2024
johnnyomair pushed a commit to vivid-planet/comet-starter that referenced this issue Jul 29, 2024
Due to a recent change in the OpenTelemetry Collector, Jaeger didn't
accept OpenTelemetry data anymore. Until this change is reverted
upstream, we explicitly pass the the host IP to Jaeger as a workaround.
See jaegertracing/jaeger#5737 for more
information.
johnnyomair added a commit to vivid-planet/comet that referenced this issue Aug 12, 2024
Due to a recent change in the OpenTelemetry Collector, Jaeger didn't accept OpenTelemetry data anymore. Until this change is reverted upstream, we explicitly pass the the host IP to Jaeger as a workaround. See jaegertracing/jaeger#5737 for more information.
johnnyomair added a commit to vivid-planet/comet that referenced this issue Aug 12, 2024
Due to a recent change in the OpenTelemetry Collector, Jaeger didn't
accept OpenTelemetry data anymore. Until this change is reverted
upstream, we explicitly pass the the host IP to Jaeger as a workaround.
See jaegertracing/jaeger#5737 for more
information.
hellais added a commit to ooni/data that referenced this issue Sep 2, 2024
@matthewgrossman
Copy link

FYI, this env-based workaround discussed in this issue wasn't working for me; no services external to my jaeger container could send (gRPC) traces to 4317. I noticed in this https://github.com/jaegertracing/jaeger/pull/6134/files that you can do settings overrides, which did work for me:

docker run --rm --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  jaegertracing/jaeger:2.0.0 \
  --set=receivers.otlp.protocols.grpc.endpoint="0.0.0.0:4317"

Not sure if this is a regression for 2.0, or if this is just expected behavior now that jaeger has more settings parity with the otel-collector

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants