Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[connector/routing] Not included in contrib distribution #26495

Closed
yanamg7 opened this issue Sep 7, 2023 · 15 comments
Closed

[connector/routing] Not included in contrib distribution #26495

yanamg7 opened this issue Sep 7, 2023 · 15 comments
Labels
bug Something isn't working connector/routing question Further information is requested Stale

Comments

@yanamg7
Copy link

yanamg7 commented Sep 7, 2023

Component(s)

connector/routing

What happened?

Using artifactory.f5net.com/dockerhub-remote/otel/opentelemetry-collector-contrib:0.84.0 I trying to add connector routing

Steps to Reproduce

I added to config.yaml this part:
connector:
routing:
default_pipelines: [logs/all]
error_mode: ignore
table:
- statement: route() where resource.attributes["Severity"] == "err"
pipelines: [logs/err]
- statement: route() where resource.attributes["Severity"] == "info"
pipelines: [logs/info]

Trying to run

Expected Result

Run successfully, no errors, correct output

Actual Result

error decoding 'connectors': unknown type: "routing" for id: "routing" (valid values: [count datadog servicegraph spanmetrics forward])

Collector version

0.84.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  filelog:
    include: [ /opt/parser/*.log ]
    start_at: beginning
    operators:
      - type: add
        field: attributes.sev
        value: unspecified
      - type: json_parser
      - type: severity_parser
        parse_from: attributes.sev
        mapping:
          fatal: emergency
          info4: notice
      - type: move
        id: move5
        from: attributes.sev
        to: resource["Severity"]
      

exporters:
  file/file1:
    path: /var/log/collector/logging1/otlp.json
  file/file2:
    path: /var/log/collector/logging2/otlp.json
  file/file3:
    path: /var/log/collector/logging3/otlp.json
  logging:
    loglevel: debug


connector:
  routing:
      default_pipelines: [logs/all]
      error_mode: ignore
      table:
        - statement: route() where resource.attributes["Severity"] == "err"
          pipelines: [logs/err]
        - statement: route() where resource.attributes["Severity"] == "info"
          pipelines: [logs/info]

service:
  pipelines:
    logs/in:
      receivers: [filelog]
      exporters: [routing, logging]
    logs/all:
      receivers: [routing]
      exporters: [file/file3, logging]
    logs/info:
      receivers: [routing]
      processors: []
      exporters: [file/file1, logging]
    logs/err:
      receivers: [routing]
      processors: []
      exporters: [file/file2, logging]

Log output

otel-contrib  | Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
otel-contrib  | 
otel-contrib  | * error decoding 'connectors': unknown type: "routing" for id: "routing" (valid values: [count datadog servicegraph spanmetrics forward])
otel-contrib  | 2023/09/07 07:53:12 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
otel-contrib  | 
otel-contrib  | * error decoding 'connectors': unknown type: "routing" for id: "routing" (valid values: [count datadog servicegraph spanmetrics forward])
otel-contrib exited with code 1

Additional context

No response

@yanamg7 yanamg7 added bug Something isn't working needs triage New item requiring triage labels Sep 7, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Sep 7, 2023

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@mwear
Copy link
Member

mwear commented Sep 7, 2023

@yanamg7 can you make sure that connectors ends in an "s" (rename connector -> connectors) in your yaml?

connectors:
  routing:
      default_pipelines: [logs/all]
      error_mode: ignore
      table:
        - statement: route() where resource.attributes["Severity"] == "err"
          pipelines: [logs/err]
        - statement: route() where resource.attributes["Severity"] == "info"
          pipelines: [logs/info]

@mwear mwear added question Further information is requested and removed needs triage New item requiring triage labels Sep 7, 2023
@mwear
Copy link
Member

mwear commented Sep 7, 2023

Actually scrap that idea @yanamg7. I was able to reproduce your error with similar yaml. I will look into this further and let you know what I find.

@mwear
Copy link
Member

mwear commented Sep 7, 2023

@yanamg7 after taking a closer look, the issue is that the routing connector is still in development and components with development status are not included in the collector contrib public images.

development: [traces_to_traces, metrics_to_metrics, logs_to_logs]

If you want to use the routing connector today, you will have to build your own image. You could use the collector builder to do this.

The reason the status of this component is development is that there is still some major feature work that needs to take place. Most notably, context based routing. If you are interested using the routing connector in its bleeding edge state, and would like it to be part of the public contrib images, let me know and we can consider changing the status to alpha.

@yanamg7
Copy link
Author

yanamg7 commented Sep 11, 2023

Hi @mwer
Thank you for the quick response,
We would much appreciate it if you could include the routing processor in the public image/build,
It would be beneficial for us.
We are currently investigating routing options for our OTEL, and we are developing some PoC for several routing scenarios.
Would you happen to have a recommended method to compare different setups of OTEL Collectors?
We are considering several options for routing our Resources (Logs and Metrics) based on Resources attributes.
We want a method to measure each setup based on resource consumption among other criteria
We would appreciate your advice

@jpkrohling
Copy link
Member

While I'd love the connector to be ready, the routing processor is still there. Is there a specific reason you can't use it?

@mwear
Copy link
Member

mwear commented Sep 11, 2023

@yanamg7, as @jpkrohling points out, the routing processor exists and more feature complete. The goal is for the routing connector to eventually replace it, but there is more work that needs to be done for it to have parity. Let us know if the processor will work for you for now or if you would still prefer to use the connector.

@nadavleva
Copy link

Hi @mwear and @jpkrohling, I am working with @yanamg7, on the same project.

Thank you for your quick response.

We are checking several routing options for our OTEL Collector based on resources and attributes. We are currently working with Metrics and Logs resource types.

In our current implementation, we allow the dynamic creation of Otlp Exporters using API and adding these exporters to the pipelines based on the API input.

The routing connector seems to be better suited to our needs, since it routes to Pipelines and not to Exporters.

In case we will use the Routing Processor then, the exporter creation will have to be embedded in the routing logic.

In addition, the routing processer is going to be deprecated according to our understanding of this #19738

I have some questions:
1. Regarding the Routing Connector, what is the ETA for its release?
2. Is the Routing processor deprecation decision final or both the routing connector and processor will be available?
3. In case we want to contribute to the development of the Otel Collector, especially the routing connector, what is the procedure and policies to become an active contributor?

@jpkrohling
Copy link
Member

  1. Regarding the Routing Connector, what is the ETA for its release?

No ETA.

  1. Is the Routing processor deprecation decision final or both the routing connector and processor will be available?

This is the general direction we want to go. I'd say the decision is final, but we are not going to remove the processor until the connector is feature-complete.

  1. In case we want to contribute to the development of the Otel Collector, especially the routing connector, what is the procedure and policies to become an active contributor?

Check the contribution guidelines for general information, but your contribution is certainly welcome! I believe the one thing missing is support for context-based routing, so that request metadata (like HTTP headers) can be used for making the decision on where to route. That being implemented, I think there will be feature parity between the processor and the connector. I'm not sure OTTL has support for contexts at this point, though.

@mwear, does that match your understanding as well?

@mwear
Copy link
Member

mwear commented Sep 13, 2023

That matches my understanding. I'll second that we welcome any and all help. If you are interesting in getting involved there are weekly Collector SIG meetings. You can find the times on the public calendar. There is an #otel-collector channel on the CNCF Slack. To join the slack look here. When and if you are ready to contribute code, you will need to sign the CLA.

For the earlier question, can we mark the routing connector as alpha, so it will be part of the contrib release, I think we probably can, but I will ask around and give an update. The definition of alpha is:

The component is ready to be used for limited non-critical workloads and the authors of this component would welcome your feedback. Bugs and performance problems should be reported, but component owners might not work on them right away. The configuration options might change often without backwards compatibility guarantees.

@djaglowski
Copy link
Member

I'm in favor of the connector being moved to alpha. Although it has only a limited subset of the processor's capabilities, it is useful and as far as I'm aware has been working without issues.

@djaglowski djaglowski changed the title Coonector routing error decoding [connector/routing] Not included in contrib distribution Sep 13, 2023
@yanamg7
Copy link
Author

yanamg7 commented Sep 14, 2023

Thank you all for your response. We will check the contribution guidelines and our possibilities. For now, I'm trying to create a custom image. I succeeded in adding all except on zipkin extension. It doesn't exist in regular go.opentelemetry.io/collector/extension. I found something in go.opentelemetry.io/otel/exporters/zipkin v1.18.0 - but it is not what I'm looking for. Can somebody help me with it?

@mwear
Copy link
Member

mwear commented Sep 14, 2023

Hello @yanamg7. I opened a PR to change the stability of the routing to connector to alpha so that it will be included in future collector contrib images.

I think I initially linked you to the archived colllector builder repo, when I should have linked you here. Sorry if that caused any confusion. In any case, I was able to build a custom collector that includes the routing connector and the zipkin exporter. Here are the steps that I followed:

$ GO111MODULE=on go install go.opentelemetry.io/collector/cmd/builder@latest
$ cat > otelcol-builder.yaml <<EOF
dist:
  name: otelcol-custom
  description: Local OpenTelemetry Collector binary
  output_path: /tmp/dist
exporters:
  - gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.85.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/zipkinexporter v0.85.0

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.85.0

processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.85.0

connectors:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/routingconnector v0.85.0
EOF
$ builder --config=otelcol-builder.yaml

You can start with that as an example and customize it for your needs.

djaglowski pushed a commit that referenced this issue Sep 14, 2023
**Description:**
This PR changes the stability of the routing connector to alpha so that
it will be included in the contrib images for usage by early adopters.
The connector is not yet feature complete, but resource attribute based
routing is. There are users interested in using it and potentially
contributing additional functionality.

**Link to tracking Issue:** #26495
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Nov 14, 2023
@djaglowski
Copy link
Member

Closing, since routing connector is now included in the contrib distro

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working connector/routing question Further information is requested Stale
Projects
None yet
Development

No branches or pull requests

5 participants