-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[connector/routing] Not included in contrib distribution #26495
Comments
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
@yanamg7 can you make sure that connectors ends in an "s" (rename connector -> connectors) in your yaml?
|
Actually scrap that idea @yanamg7. I was able to reproduce your error with similar yaml. I will look into this further and let you know what I find. |
@yanamg7 after taking a closer look, the issue is that the routing connector is still in
If you want to use the routing connector today, you will have to build your own image. You could use the collector builder to do this. The reason the status of this component is development is that there is still some major feature work that needs to take place. Most notably, context based routing. If you are interested using the routing connector in its bleeding edge state, and would like it to be part of the public contrib images, let me know and we can consider changing the status to |
Hi @mwer |
While I'd love the connector to be ready, the routing processor is still there. Is there a specific reason you can't use it? |
@yanamg7, as @jpkrohling points out, the routing processor exists and more feature complete. The goal is for the routing connector to eventually replace it, but there is more work that needs to be done for it to have parity. Let us know if the processor will work for you for now or if you would still prefer to use the connector. |
Hi @mwear and @jpkrohling, I am working with @yanamg7, on the same project. Thank you for your quick response. We are checking several routing options for our OTEL Collector based on resources and attributes. We are currently working with Metrics and Logs resource types. In our current implementation, we allow the dynamic creation of Otlp Exporters using API and adding these exporters to the pipelines based on the API input. The routing connector seems to be better suited to our needs, since it routes to Pipelines and not to Exporters. In case we will use the Routing Processor then, the exporter creation will have to be embedded in the routing logic. In addition, the routing processer is going to be deprecated according to our understanding of this #19738 I have some questions: |
No ETA.
This is the general direction we want to go. I'd say the decision is final, but we are not going to remove the processor until the connector is feature-complete.
Check the contribution guidelines for general information, but your contribution is certainly welcome! I believe the one thing missing is support for context-based routing, so that request metadata (like HTTP headers) can be used for making the decision on where to route. That being implemented, I think there will be feature parity between the processor and the connector. I'm not sure OTTL has support for contexts at this point, though. @mwear, does that match your understanding as well? |
That matches my understanding. I'll second that we welcome any and all help. If you are interesting in getting involved there are weekly Collector SIG meetings. You can find the times on the public calendar. There is an #otel-collector channel on the CNCF Slack. To join the slack look here. When and if you are ready to contribute code, you will need to sign the CLA. For the earlier question, can we mark the routing connector as alpha, so it will be part of the contrib release, I think we probably can, but I will ask around and give an update. The definition of alpha is:
|
I'm in favor of the connector being moved to alpha. Although it has only a limited subset of the processor's capabilities, it is useful and as far as I'm aware has been working without issues. |
Thank you all for your response. We will check the contribution guidelines and our possibilities. For now, I'm trying to create a custom image. I succeeded in adding all except on zipkin extension. It doesn't exist in regular go.opentelemetry.io/collector/extension. I found something in go.opentelemetry.io/otel/exporters/zipkin v1.18.0 - but it is not what I'm looking for. Can somebody help me with it? |
Hello @yanamg7. I opened a PR to change the stability of the routing to connector to alpha so that it will be included in future collector contrib images. I think I initially linked you to the archived colllector builder repo, when I should have linked you here. Sorry if that caused any confusion. In any case, I was able to build a custom collector that includes the routing connector and the zipkin exporter. Here are the steps that I followed: $ GO111MODULE=on go install go.opentelemetry.io/collector/cmd/builder@latest
$ cat > otelcol-builder.yaml <<EOF
dist:
name: otelcol-custom
description: Local OpenTelemetry Collector binary
output_path: /tmp/dist
exporters:
- gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.85.0
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/zipkinexporter v0.85.0
receivers:
- gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.85.0
processors:
- gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.85.0
connectors:
- gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/routingconnector v0.85.0
EOF
$ builder --config=otelcol-builder.yaml You can start with that as an example and customize it for your needs. |
**Description:** This PR changes the stability of the routing connector to alpha so that it will be included in the contrib images for usage by early adopters. The connector is not yet feature complete, but resource attribute based routing is. There are users interested in using it and potentially contributing additional functionality. **Link to tracking Issue:** #26495
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
Closing, since routing connector is now included in the contrib distro |
Component(s)
connector/routing
What happened?
Using artifactory.f5net.com/dockerhub-remote/otel/opentelemetry-collector-contrib:0.84.0 I trying to add connector routing
Steps to Reproduce
I added to config.yaml this part:
connector:
routing:
default_pipelines: [logs/all]
error_mode: ignore
table:
- statement: route() where resource.attributes["Severity"] == "err"
pipelines: [logs/err]
- statement: route() where resource.attributes["Severity"] == "info"
pipelines: [logs/info]
Trying to run
Expected Result
Run successfully, no errors, correct output
Actual Result
error decoding 'connectors': unknown type: "routing" for id: "routing" (valid values: [count datadog servicegraph spanmetrics forward])
Collector version
0.84.0
Environment information
Environment
OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")
OpenTelemetry Collector configuration
Log output
Additional context
No response
The text was updated successfully, but these errors were encountered: