Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No metrics exported using otel-stream-handler, while otel-handler does #1051

Open
sge93 opened this issue Dec 13, 2023 · 1 comment
Open
Labels
bug Something isn't working

Comments

@sge93
Copy link

sge93 commented Dec 13, 2023

Describe the bug
I am using the java wrapper for an AWS Lambda to export metrics to Amazon Managed Prometheus. The Lambda is setup with Spring Cloud Functions in a declarative way with the handler:
org.springframework.cloud.function.adapter.aws.FunctionInvoker::handleRequest
which is implementing the AWS Lambda RequestStreamHandler.

When I invoke the Lambda through an S3 Event, it starts and processes it without any problems and the handler seems to be correct as I have the wrapper set to otel-stream-handler:
[ main] o.s.c.f.a.aws.CustomRuntimeInitializer : AWS Handler: io.opentelemetry.instrumentation.awslambdacore.v1_0.TracingRequestStreamWrapper

Unfortunately no metrics are exported to AMP. I have this Bean where I add a counter for the sake of testing:

@Bean
    Meter meter() {
        var meter = GlobalOpenTelemetry.getMeterProvider().meterBuilder("aws-otel").setInstrumentationVersion("1.0").build(); 
        var test = meter.counterBuilder("test1").setDescription("test").build(); 
        test.add(1234); 
        return meter; 
    }

When I set the AWS_LAMBDA_EXEC_WRAPPER to /opt/otel-handler this counter is exported but the function cannot be processed as the otel-handler implementation does not support an InputStream which is expected. Still the metric defined in the Bean is exported to AMP and accessible via PromQL query.

Steps to reproduce

  • Spring Cloud Function with S3 Event Input deployed as AWS Lambda.

  • Specify layer as:
    arn:aws:lambda:eu-central-1:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-31-0:1

  • Lambda Handler
    org.springframework.cloud.function.adapter.aws.FunctionInvoker::handleRequest

    @Bean
    public Function<S3Event, String> function() {
       return event -> "OK";
   }
  • Collector config:
# collector.yaml
extensions:
  sigv4auth:
    service: "aps" 
    region: <workspace_region>

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  prometheusremotewrite:
    endpoint: <workspace_remote_write_url>
    auth: 
      authenticator: sigv4auth

service:
  extensions: [sigv4auth]
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheusremotewrite]
  • Use opentelemetry-api to export counter metrics.
  • Lambda timeout is 120 seconds and used RAM is 1024MB.

What did you expect to see?
I would expect to at least see the metrics defined in the Meter bean as the dependency is required in further processing components (which runs through successfully) and therefore should be available in the ApplicationContext. Prometheus workspace queries show no results.

What did you see instead?
No logs indicating a failure. At some point minutes later the logs show:
{"level":"info","ts":1702461394.1761792,"logger":"lifecycle.manager","msg":"Received SHUTDOWN event"}

What version of collector/language SDK version did you use?

  • OpenTelemetry Lambda extension v0.35.0
  • arn:aws:lambda:eu-central-1:901920570463:layer:aws-otel-java-wrapper-amd64-ver-1-31-0:1
  • io.opentelemetry:opentelemetry-bom:1.32.0
  • Java 17
  • org.springframework.cloud:spring-cloud-function-dependencies:4.1.0

What language layer did you use?

  • Java
@sge93 sge93 added the bug Something isn't working label Dec 13, 2023
@serkan-ozal
Copy link
Contributor

@sge93

I think, this also should be fixed with this PR: open-telemetry/opentelemetry-java-instrumentation#11868

Before that PR, traces and metrics were flushed in case of error and output stream close, but with that PR, traces and metrics are flushed in any case before the invocation completed.

Screenshot 2024-09-02 at 12 30 54

Can you try with the latest version of the wrapper layer and let us know whether it works or not?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants