[connector/routing] Not included in contrib distribution

[yanamg7]

Component(s)

connector/routing

What happened?

Using artifactory.f5net.com/dockerhub-remote/otel/opentelemetry-collector-contrib:0.84.0 I trying to add connector routing

Steps to Reproduce

I added to config.yaml this part:
connector:
routing:
default_pipelines: [logs/all]
error_mode: ignore
table:
- statement: route() where resource.attributes["Severity"] == "err"
pipelines: [logs/err]
- statement: route() where resource.attributes["Severity"] == "info"
pipelines: [logs/info]

Trying to run

Expected Result

Run successfully, no errors, correct output

Actual Result

error decoding 'connectors': unknown type: "routing" for id: "routing" (valid values: [count datadog servicegraph spanmetrics forward])

Collector version

0.84.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  filelog:
    include: [ /opt/parser/*.log ]
    start_at: beginning
    operators:
      - type: add
        field: attributes.sev
        value: unspecified
      - type: json_parser
      - type: severity_parser
        parse_from: attributes.sev
        mapping:
          fatal: emergency
          info4: notice
      - type: move
        id: move5
        from: attributes.sev
        to: resource["Severity"]
      

exporters:
  file/file1:
    path: /var/log/collector/logging1/otlp.json
  file/file2:
    path: /var/log/collector/logging2/otlp.json
  file/file3:
    path: /var/log/collector/logging3/otlp.json
  logging:
    loglevel: debug


connector:
  routing:
      default_pipelines: [logs/all]
      error_mode: ignore
      table:
        - statement: route() where resource.attributes["Severity"] == "err"
          pipelines: [logs/err]
        - statement: route() where resource.attributes["Severity"] == "info"
          pipelines: [logs/info]

service:
  pipelines:
    logs/in:
      receivers: [filelog]
      exporters: [routing, logging]
    logs/all:
      receivers: [routing]
      exporters: [file/file3, logging]
    logs/info:
      receivers: [routing]
      processors: []
      exporters: [file/file1, logging]
    logs/err:
      receivers: [routing]
      processors: []
      exporters: [file/file2, logging]

Log output

otel-contrib  | Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
otel-contrib  | 
otel-contrib  | * error decoding 'connectors': unknown type: "routing" for id: "routing" (valid values: [count datadog servicegraph spanmetrics forward])
otel-contrib  | 2023/09/07 07:53:12 collector server run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
otel-contrib  | 
otel-contrib  | * error decoding 'connectors': unknown type: "routing" for id: "routing" (valid values: [count datadog servicegraph spanmetrics forward])
otel-contrib exited with code 1

Additional context

No response

[github-actions]

Pinging code owners:

• connector/routing: @jpkrohling @mwear

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mwear]

@yanamg7 can you make sure that connectors ends in an "s" (rename connector -> connectors) in your yaml?

connectors:
  routing:
      default_pipelines: [logs/all]
      error_mode: ignore
      table:
        - statement: route() where resource.attributes["Severity"] == "err"
          pipelines: [logs/err]
        - statement: route() where resource.attributes["Severity"] == "info"
          pipelines: [logs/info]

[mwear]

Actually scrap that idea @yanamg7. I was able to reproduce your error with similar yaml. I will look into this further and let you know what I find.

[mwear]

@yanamg7 after taking a closer look, the issue is that the routing connector is still in development and components with development status are not included in the collector contrib public images.

opentelemetry-collector-contrib/connector/routingconnector/metadata.yaml

Line 6 in 9424ae3

development: [traces_to_traces, metrics_to_metrics, logs_to_logs]

If you want to use the routing connector today, you will have to build your own image. You could use the collector builder to do this.

The reason the status of this component is development is that there is still some major feature work that needs to take place. Most notably, context based routing. If you are interested using the routing connector in its bleeding edge state, and would like it to be part of the public contrib images, let me know and we can consider changing the status to alpha.

[yanamg7]

Hi @mwer
Thank you for the quick response,
We would much appreciate it if you could include the routing processor in the public image/build,
It would be beneficial for us.
We are currently investigating routing options for our OTEL, and we are developing some PoC for several routing scenarios.
Would you happen to have a recommended method to compare different setups of OTEL Collectors?
We are considering several options for routing our Resources (Logs and Metrics) based on Resources attributes.
We want a method to measure each setup based on resource consumption among other criteria
We would appreciate your advice

[jpkrohling]

While I'd love the connector to be ready, the routing processor is still there. Is there a specific reason you can't use it?

[mwear]

@yanamg7, as @jpkrohling points out, the routing processor exists and more feature complete. The goal is for the routing connector to eventually replace it, but there is more work that needs to be done for it to have parity. Let us know if the processor will work for you for now or if you would still prefer to use the connector.

[nadavleva]

Hi @mwear and @jpkrohling, I am working with @yanamg7, on the same project.

Thank you for your quick response.

We are checking several routing options for our OTEL Collector based on resources and attributes. We are currently working with Metrics and Logs resource types.

In our current implementation, we allow the dynamic creation of Otlp Exporters using API and adding these exporters to the pipelines based on the API input.

The routing connector seems to be better suited to our needs, since it routes to Pipelines and not to Exporters.

In case we will use the Routing Processor then, the exporter creation will have to be embedded in the routing logic.

In addition, the routing processer is going to be deprecated according to our understanding of this #19738

I have some questions:
1. Regarding the Routing Connector, what is the ETA for its release?
2. Is the Routing processor deprecation decision final or both the routing connector and processor will be available?
3. In case we want to contribute to the development of the Otel Collector, especially the routing connector, what is the procedure and policies to become an active contributor?

[jpkrohling]

1. Regarding the Routing Connector, what is the ETA for its release?

No ETA.

2. Is the Routing processor deprecation decision final or both the routing connector and processor will be available?

This is the general direction we want to go. I'd say the decision is final, but we are not going to remove the processor until the connector is feature-complete.

3. In case we want to contribute to the development of the Otel Collector, especially the routing connector, what is the procedure and policies to become an active contributor?

Check the contribution guidelines for general information, but your contribution is certainly welcome! I believe the one thing missing is support for context-based routing, so that request metadata (like HTTP headers) can be used for making the decision on where to route. That being implemented, I think there will be feature parity between the processor and the connector. I'm not sure OTTL has support for contexts at this point, though.

@mwear, does that match your understanding as well?

[mwear]

That matches my understanding. I'll second that we welcome any and all help. If you are interesting in getting involved there are weekly Collector SIG meetings. You can find the times on the public calendar. There is an #otel-collector channel on the CNCF Slack. To join the slack look here. When and if you are ready to contribute code, you will need to sign the CLA.

For the earlier question, can we mark the routing connector as alpha, so it will be part of the contrib release, I think we probably can, but I will ask around and give an update. The definition of alpha is:

The component is ready to be used for limited non-critical workloads and the authors of this component would welcome your feedback. Bugs and performance problems should be reported, but component owners might not work on them right away. The configuration options might change often without backwards compatibility guarantees.

[djaglowski]

I'm in favor of the connector being moved to alpha. Although it has only a limited subset of the processor's capabilities, it is useful and as far as I'm aware has been working without issues.

[yanamg7]

Thank you all for your response. We will check the contribution guidelines and our possibilities. For now, I'm trying to create a custom image. I succeeded in adding all except on zipkin extension. It doesn't exist in regular go.opentelemetry.io/collector/extension. I found something in go.opentelemetry.io/otel/exporters/zipkin v1.18.0 - but it is not what I'm looking for. Can somebody help me with it?

[mwear]

Hello @yanamg7. I opened a PR to change the stability of the routing to connector to alpha so that it will be included in future collector contrib images.

I think I initially linked you to the archived colllector builder repo, when I should have linked you here. Sorry if that caused any confusion. In any case, I was able to build a custom collector that includes the routing connector and the zipkin exporter. Here are the steps that I followed:

$ GO111MODULE=on go install go.opentelemetry.io/collector/cmd/builder@latest
$ cat > otelcol-builder.yaml <<EOF
dist:
  name: otelcol-custom
  description: Local OpenTelemetry Collector binary
  output_path: /tmp/dist
exporters:
  - gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.85.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/zipkinexporter v0.85.0

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.85.0

processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.85.0

connectors:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/routingconnector v0.85.0
EOF
$ builder --config=otelcol-builder.yaml

You can start with that as an example and customize it for your needs.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• connector/routing: @jpkrohling @mwear

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

Closing, since routing connector is now included in the contrib distro