Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OIDC Extension Keycloak Token Auth #34879

Closed
fthrslntgy opened this issue Aug 27, 2024 · 5 comments
Closed

OIDC Extension Keycloak Token Auth #34879

fthrslntgy opened this issue Aug 27, 2024 · 5 comments

Comments

@fthrslntgy
Copy link

fthrslntgy commented Aug 27, 2024

Component(s)

extension/oidcauth

What happened?

Description

Hi. I want to trace my golang application's metrics with Opentelemetry and Clickhouse. To achieve this, I use the opentelemetry clickhouse exporter. When I perform the exporter configuration without using any authenticator, I can view the traces of the transactions I make on ClickHouse in a healthy way. In other words, I have actually successfully established the architecture. However, I need to add an authenticator to the exporter. When I do this process with basic auth, I do not encounter an error. When I add the basic auth options that I set in otel-collector-config.yaml to the opentelemetry header in my application as Basic Authorization, I can still run it.

As for the problem, I cannot do this when I want to use the oidc extension. To achieve this, I give the header as "Authorization: Bearer _TOKEN" on the application side. But I keep getting this error: 2024/08/27 15:58:52 traces export: failed to send to http://localhost:4318/v1/traces: 401 Unauthorized. When I encounter this error, no logs appear on the exporter (even though I started it in debug mode).

Expected Result

Accepted Keycloak Token

Actual Result

401 Unauthorized

Collector version

v0.105.0

Environment information

Environment

OS: Ubuntu 20.04 on WSL 2
Compiler(if manually compiled): go1.22.5 linux/amd64

OpenTelemetry Collector configuration

extensions:
  oidc/server:                                                         
    issuer_url: http://172.29.151.124:8888/realms/otel
    audience:  myclient
    attribute: Authorization

receivers:
  fluentforward:
    endpoint: 0.0.0.0:24224
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
        auth:
          authenticator: oidc/server
exporters:
  clickhouse:
    endpoint: tcp://172.29.151.124:9000
    database: otel
    username: default
    password: "1"
    logs_table_name: otel_logs
    traces_table_name: otel_traces
    ttl: 12h
    timeout: 10s
    sending_queue:
      queue_size: 100
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

processors:
 batch:
   timeout: 5s
   send_batch_size: 100000

service:
 extensions: [oidc/server]
 pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]
    traces:
     receivers: [otlp]
     processors: [batch]
     exporters: [clickhouse]

Log output

There is not any log about Keycloak login but I am putting whole exporter log.

2024-08-27T12:59:25.696Z        info    service@v0.105.1-0.20240717163034-43ed6184f9fe/service.go:116   Setting up own telemetry...
2024-08-27T12:59:25.696Z        info    service@v0.105.1-0.20240717163034-43ed6184f9fe/service.go:119   OpenCensus bridge is disabled for Collector telemetry and will be removed in a future version, use --feature-gates=-service.disableOpenCensusBridge to re-enable
2024-08-27T12:59:25.696Z        info    service@v0.105.1-0.20240717163034-43ed6184f9fe/telemetry.go:96  Serving metrics {"address": ":8888", "metrics level": "Normal"}
2024-08-27T12:59:25.696Z        debug   exporter@v0.105.1-0.20240717163034-43ed6184f9fe/exporter.go:278 Alpha component. May change in the future.      {"kind": "exporter", "data_type": "traces", "name": "clickhouse"}
2024-08-27T12:59:25.696Z        info    exporter@v0.105.1-0.20240717163034-43ed6184f9fe/exporter.go:280 Development component. May change in the future.        {"kind": "exporter", "data_type": "traces", "name": "debug"}
2024-08-27T12:59:25.696Z        debug   exporter@v0.105.1-0.20240717163034-43ed6184f9fe/exporter.go:278 Beta component. May change in the future.       {"kind": "exporter", "data_type": "logs", "name": "clickhouse"}
2024-08-27T12:59:25.696Z        info    exporter@v0.105.1-0.20240717163034-43ed6184f9fe/exporter.go:280 Development component. May change in the future.        {"kind": "exporter", "data_type": "logs", "name": "debug"}
2024-08-27T12:59:25.696Z        debug   processor@v0.105.1-0.20240717163034-43ed6184f9fe/processor.go:306       Beta component. May change in the future.       {"kind": "processor", "name": "batch", "pipeline": "logs"}
2024-08-27T12:59:25.696Z        debug   receiver@v0.105.1-0.20240717163034-43ed6184f9fe/receiver.go:313 Beta component. May change in the future.       {"kind": "receiver", "name": "otlp", "data_type": "logs"}
2024-08-27T12:59:25.696Z        debug   processor@v0.105.1-0.20240717163034-43ed6184f9fe/processor.go:306       Beta component. May change in the future.       {"kind": "processor", "name": "batch", "pipeline": "traces"}
2024-08-27T12:59:25.696Z        debug   receiver@v0.105.1-0.20240717163034-43ed6184f9fe/receiver.go:313 Stable component.       {"kind": "receiver", "name": "otlp", "data_type": "traces"}
2024-08-27T12:59:25.696Z        debug   extension@v0.105.1-0.20240717163034-43ed6184f9fe/extension.go:170       Beta component. May change in the future.       {"kind": "extension", "name": "oidc/server"}
2024-08-27T12:59:25.697Z        info    service@v0.105.1-0.20240717163034-43ed6184f9fe/service.go:198   Starting otelcontribcol...      {"Version": "0.105.0-dev", "NumCPU": 12}
2024-08-27T12:59:25.697Z        info    extensions/extensions.go:34     Starting extensions...
2024-08-27T12:59:25.697Z        info    extensions/extensions.go:37     Extension is starting...        {"kind": "extension", "name": "oidc/server"}
2024-08-27T12:59:25.704Z        info    extensions/extensions.go:52     Extension started.      {"kind": "extension", "name": "oidc/server"}
2024-08-27T12:59:25.722Z        info    zapgrpc/zapgrpc.go:176  [core] [Server #1]Server created        {"grpc_log": true}
2024-08-27T12:59:25.722Z        info    otlpreceiver@v0.105.1-0.20240717163034-43ed6184f9fe/otlp.go:102 Starting GRPC server    {"kind": "receiver", "name": "otlp", "data_type": "logs", "endpoint": "0.0.0.0:4317"}
2024-08-27T12:59:25.722Z        info    otlpreceiver@v0.105.1-0.20240717163034-43ed6184f9fe/otlp.go:152 Starting HTTP server    {"kind": "receiver", "name": "otlp", "data_type": "logs", "endpoint": "0.0.0.0:4318"}
2024-08-27T12:59:25.722Z        info    service@v0.105.1-0.20240717163034-43ed6184f9fe/service.go:224   Everything is ready. Begin running and processing data.
2024-08-27T12:59:25.722Z        info    localhostgate/featuregate.go:63 The default endpoints for all servers in components have changed to use localhost instead of 0.0.0.0. Disable the feature gate to temporarily revert to the previous default.       {"feature gate ID": "component.UseLocalHostAsDefaultHost"}
2024-08-27T12:59:25.722Z        info    zapgrpc/zapgrpc.go:176  [core] [Server #1 ListenSocket #2]ListenSocket created  {"grpc_log": true}

Additional context

If I need to provide detailed information about possible questions that may arise:

  • I am sure that I have set up the architecture correctly because the system works properly when I do not use any authorization or when I use basic auth.
  • I am sure that there is no communication problem between Keycloak and the exporter because when I knowingly enter incorrect information about Keycloak in the exporter config, the exporter does not work and gives errors such as "cannot be accessed" or "wrong realm name". When I enter the correct config, it works without any problems.
  • I don't think there is a problem with the header or token on the application side because when I give the basic auth information to the header, there is no problem. I get the token with the query I send via Postman and check its accuracy (I can log in with the token I get using the Keycloak libraries with a small Golang application).
  • The only source I could find about Opentelemetry OIDC authorization and Keycloak integration is: https://medium.com/opentelemetry/securing-your-opentelemetry-collector-1a4f9fa5bd6f. However, this source does not have the architecture I want to build. I only want to send tokens from my own applications and verify this on the exporter side, and only clients that have received tokens from Keycloak (verified) can send traces. However, since this is the only source I could find, I set the settings in Keycloak (such as realm, client and scope) in the same way as this source.
  • Even though I have started Keycloak and Exporter in "DEBUG" mode as the log level, I can only see the "401 Unauthorized" error on the application. When I get this error, the log is not dropped anywhere else.
@fthrslntgy fthrslntgy added bug Something isn't working needs triage New item requiring triage labels Aug 27, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@open-telemetry open-telemetry deleted a comment Aug 27, 2024
@jpkrohling
Copy link
Member

jpkrohling commented Aug 27, 2024

What's the body of the HTTP response you are receiving?

@jpkrohling jpkrohling added waiting for author and removed needs triage New item requiring triage labels Aug 27, 2024
@fthrslntgy
Copy link
Author

fthrslntgy commented Aug 27, 2024

I understand that what you mean by HTTP body is the body of the "401 unauthorized" response. Since I developed the application with golang and used opentelemetry libraries for tracing, I cannot manipulate these parts manually and cannot view the detailed body. If you can give me instructions, I can share the body with you.

Opentelemetry libraries and versions that I use in the application:

go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.53.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.53.0
go.opentelemetry.io/otel v1.28.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.28.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.28.0
go.opentelemetry.io/otel/sdk v1.28.0

go.opentelemetry.io/contrib v1.17.0 // indirect
go.opentelemetry.io/otel/metric v1.28.0 // indirect
go.opentelemetry.io/otel/trace v1.28.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect

In addition, for example, when I try to get Keycloak token with wrong credentials via Postman or give wrong Exporter config etc. I can view detailed error logs on Keycloak for these operations. But when I get this "401" response, no log is sent to Keycloak (even though I started Keycloak with KC_LOG_LEVEL=DEBUG config).

Additional Note: I am adding the block where I integrated opentelemetry into the application for help.

import (
	"context"

	"github.com/limanmys/netex-server/internal/constants"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)

func InitTracer() *sdktrace.TracerProvider {
	client := otlptracehttp.NewClient(
		otlptracehttp.WithEndpoint(constants.OTEL_ADDRESS), // Exporter address eg. http://localhost:4318
		otlptracehttp.WithInsecure(),
		otlptracehttp.WithCompression(otlptracehttp.NoCompression),
		otlptracehttp.WithHeaders(map[string]string{
			"Authorization": constants.OTEL_AUTHORIZATION, // (Bearer: <TOKEN>) or (Basic: <USER:PASS _ in base64 form>)
		}),
	)
	exporter, err := otlptrace.New(context.Background(), client)
	if err != nil {
		return nil
	}
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithSampler(sdktrace.AlwaysSample()),
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(
			resource.NewWithAttributes(
				semconv.SchemaURL,
				semconv.ServiceNameKey.String(constants.OTEL_SERVICE_NAME), // Service name eg. network-explorer
			)),
	)
	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
	return tp
}

I am also adding otelfiber middleware to fiber.

app.Use(otelfiber.Middleware())

@jpkrohling
Copy link
Member

Can you use a similar setup from the blog post to test the collector setup? Like, use an agent with a static bearer token, and a server with the oidc auth for the OTLP receiver. If you can configure the collector as agent and it works, then we know your server is correctly configured and that there might be a problem in the Go SDK (although it looks like a config issue to me so far).

@fthrslntgy
Copy link
Author

Hi @jpkrohling. As you suggested, when I used a similar setup as in your blogpost, I was able to solve the problem!

Thanks to this setup, I was able to see the error body that was not visible in my application (only 401 unauthorized). The problem is that I send traces using the HTTP protocol. I think the OIDC authentication method does not work in HTTP receivers. When I changed the places where I used HTTP in the application to GRPC, I was able to run it successfully.

So, in Golang applications, the library required to use bearer tokens from the header should be "otlptracegrpc" instead of "otlptracehttp" and the client (in the InitTracer function I posted in the above message) should be defined as follows:

client := otlptracegrpc.NewClient(
	otlptracegrpc.WithEndpoint(constants.OTEL_ADDRESS),
	otlptracegrpc.WithInsecure(),
	otlptracegrpc.WithCompressor("gzip"),
	otlptracegrpc.WithHeaders(map[string]string{
		"authorization": constants.OTEL_AUTHORIZATION,
	}),
)

Thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants