Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

THREESCALE-7735 THREESCALE-8865 opentelemetry instrumentation #1379

Merged
merged 13 commits into from
Feb 23, 2023

Conversation

eguzki
Copy link
Member

@eguzki eguzki commented Oct 24, 2022

what

implements THREESCALE-7735
Specifically the subtask THREESCALE-8865

Adds OpenTelemetry distributed tracing support to Apicast.

Supported propagation types: W3C

Jaeger has deprecated the jaeger native clients. From Jaeger doc

For new applications, we recommend using the OpenTelemetry APIs, SDKs, and instrumentation. Since v1.35, the Jaeger backend can receive trace data from the OpenTelemetry SDKs in their native OpenTelemetry Protocol (OTLP).

Jaeger client (based on opentracing) support is still in place, but gets deprecated in favor of the Opentelemetry use. When opentracing is enabled, APIcast will log warning:

2023/02/07 13:35:19 [warn] 47387#47387: [lua] environment.lua:91: read_opentracing_tracer(): opentracing use is DEPRECATED. Use Opentelemetry instead with OPENTELEMETRY env va

Add local dev environment with docker compose make opentelemetry-gateway IMAGE_NAME=my-local-image

Tasks

  • Expose opentelemetry feature and its configuration via env vars
  • In community, docker-compose test opentracing works
  • Install opentelemetry SDK and test jaeger traces
  • Bring opentelemetry SDK to the devel image
  • Bring opentelemetry SDK to the community image
  • Documentation
  • Testing

Verification steps

Build runtime image

make runtime-image

Run test env with APIcast enabled with Opentelemetry and with jeager instance up and running

IMAGE_NAME=apicast-runtime-image:latest make opentelemetry-gateway

Get the IP of APIcast

# docker ps
CONTAINER ID   IMAGE                             COMMAND                  CREATED         STATUS         PORTS                                                                                                    NAMES
9598cc711f5e   apicast-runtime-image:latest      "container-entrypoin…"   8 seconds ago   Up 7 seconds                                                                                                            apicast_build_0_opentelemetry-instrumented-gateway_run_87b6f818be1b
00e3146b48b8   jaegertracing/all-in-one:latest   "/go/bin/all-in-one-…"   8 seconds ago   Up 7 seconds   5775/udp, 5778/tcp, 14250/tcp, 6831-6832/udp, 14268/tcp, 0.0.0.0:16686->16686/tcp, :::16686->16686/tcp   apicast_build_0-jaeger-1

# APICAST_IP=$(docker inspect apicast_build_0_opentelemetry-instrumented-gateway_run_87b6f818be1b | yq e -P '.[0].NetworkSettings.Networks.apicast_build_0_default.IPAddress' -)

Send request to APIcast

curl -v -H "Host: one"  http://${APICAST_IP}:8080/get?user_key=foo
*   Trying 172.20.0.3:8080...
* Connected to 172.20.0.3 (172.20.0.3) port 8080 (#0)
> GET /get?user_key=foo HTTP/1.1
> Host: one
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: openresty
< Date: Tue, 07 Feb 2023 14:34:41 GMT
< Content-Type: application/json
< Content-Length: 371
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< 
{
  "args": {
    "user_key": "foo"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "Traceparent": "00-4335058ae8ec72f9636d8c0da08c62be-137a4beaae638572-01", 
    "User-Agent": "curl/7.81.0", 
    "X-Amzn-Trace-Id": "Root=1-63e26181-28866f82081058f20cfe47a1"
  }, 
  "origin": "91.126.156.31", 
  "url": "http://httpbin.org/get?user_key=foo"
}
* Connection #0 to host 172.20.0.3 left intact

Note that upstream echo'ed request headers show Traceparent W3C standard tracing header

Open in local browser jaeger dashboard

http://127.0.0.1:16686/search

Hit "Find Traces" with Services set to apicast. There should be at lease one trace

Screenshot 2023-02-07 at 15-38-27 Jaeger UI

And opening the, the "process" should show attribute telemetry.sdk.name = opentelemetry

Screenshot 2023-02-07 at 15-40-22 Jaeger UI

@eguzki
Copy link
Member Author

eguzki commented Oct 24, 2022

Tests and image building failing because opentelemetry rpms have not been built yet. We need to build them and publish in packages.dev.3sca.net

@eguzki eguzki force-pushed the opentelemetry-instrumentation branch from d0c04fe to 50d21e8 Compare January 25, 2023 15:14
@eguzki eguzki changed the title opentelemetry instrumentation THREESCALE-7735 THREESCALE-7735 opentelemetry instrumentation Jan 30, 2023
@eguzki eguzki changed the title THREESCALE-7735 opentelemetry instrumentation THREESCALE-7735 THREESCALE-8865 opentelemetry instrumentation Jan 30, 2023
@eguzki eguzki force-pushed the opentelemetry-instrumentation branch 2 times, most recently from 59a3814 to 13d4a9e Compare February 2, 2023 22:59
@eguzki eguzki marked this pull request as ready for review February 7, 2023 14:26
@eguzki eguzki requested review from a team as code owners February 7, 2023 14:26
@eguzki
Copy link
Member Author

eguzki commented Feb 7, 2023

ready for review @3scale/apicast-core

Copy link
Contributor

@lcavalle lcavalle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Text proofreading

@eguzki eguzki force-pushed the opentelemetry-instrumentation branch from 1f230df to 8af8b1c Compare February 17, 2023 10:05
@eguzki eguzki force-pushed the opentelemetry-instrumentation branch from 8af8b1c to 189c6bd Compare February 21, 2023 14:41
@eguzki eguzki merged commit 50827a0 into master Feb 23, 2023
@eguzki eguzki deleted the opentelemetry-instrumentation branch February 23, 2023 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants