Kafka Exporter w/ AWS_MSK_IAM Mechanism Not Working #19747

flyerjayden · 2023-03-16T06:37:49Z

Component(s)

exporter/kafka

What happened?

Description

I am getting following error message when authentication mechanism is set to AWS_MSK_IAM:

Error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)
2023/03/16 06:28:18 collector server run finished with error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)

Steps to Reproduce

Deploy OpenTelemetry collector using following config:

exporters:
  kafka:
    topic: {{ kafka_topic_name }}
    brokers:
      - {{ broker1_host }}:9098
      - {{ broker2_host}}:9098
    auth:
      sasl:
        username: {{ aws_access_key }}
        password: {{ aws_secret_key }}
        mechanism: AWS_MSK_IAM
        aws_msk:
          region: us-west-2
          broker_addr: {{ broker1_host }}:9098

Expected Result

OpenTelemetry collector sends traces to MSK topic

Actual Result

Error message:

Error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)
2023/03/16 06:28:18 collector server run finished with error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)

Collector version

0.73

Environment information

Environment

OS: EKS 1.24 / Amazon Linux 2
Compiler(if manually compiled): Used Docker image from Docker hub

OpenTelemetry Collector configuration

exporters:
  kafka:
    topic: {{ kafka_topic_name }}
    brokers:
      - {{ broker1_host }}:9098
      - {{ broker2_host}}:9098
    auth:
      sasl:
        username: {{ aws_access_key }}
        password: {{ aws_secret_key }}
        mechanism: AWS_MSK_IAM
        aws_msk:
          region: us-west-2
          broker_addr: {{ broker1_host }}:9098

Log output

Error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)
2023/03/16 06:28:18 collector server run finished with error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)

Additional context

It seems Sarama (https://github.com/Shopify/sarama/blob/7dbf0b5d85538641227a30a281f1a54e18a94cb7/broker.go#L64) does not support (and they are not planning to support) AWS_MSK_IAM mechanism.

If anyone was able to configure Kafka exporter with IAM, your advices would be very appreciated.

The text was updated successfully, but these errors were encountered:

github-actions · 2023-03-16T06:38:06Z

Pinging code owners:

exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

atoulme · 2023-03-17T02:23:25Z

I think we have an open PR for this. Would you like to help test it out or review it?

flyerjayden · 2023-03-17T04:32:44Z

Definitely I would love to help!

atoulme · 2023-03-17T13:31:20Z

I was wrong sorry, it’s for another component: #19650 (review)

I bet we can reuse the logic though.

flyerjayden · 2023-03-17T21:26:30Z

Do you know which component has that implementation? Commit that you shared is about README.md update.

atoulme · 2023-03-18T01:47:55Z

I will look into it for you. Give me a bit.

atoulme · 2023-03-18T21:59:40Z

Well, I'll be... no this is for the Kafka exporter. Check out the latest README and see how this is now documented. It should work for your use case, or there is a bug.

flyerjayden · 2023-03-19T02:04:36Z

Error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)
2023/03/19 02:00:16 collector server run finished with error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)

It does not work.

flyerjayden · 2023-03-19T02:05:48Z

AWS_MSK_ISM is not one of mechanism that sarama library supports:

authentication.go line 125

case "AWS_MSK_IAM":
		saramaConfig.Net.SASL.SCRAMClientGeneratorFunc = func() sarama.SCRAMClient {
			return awsmsk.NewIAMSASLClient(config.AWSMSK.BrokerAddr, config.AWSMSK.Region, saramaConfig.ClientID)
		}
		saramaConfig.Net.SASL.Mechanism = awsmsk.Mechanism

sarama@v1.38.1/broker.go line 63:

// SASLMechanism specifies the SASL mechanism the client uses to authenticate with the broker
type SASLMechanism string

const (
	// SASLTypeOAuth represents the SASL/OAUTHBEARER mechanism (Kafka 2.0.0+)
	SASLTypeOAuth = "OAUTHBEARER"
	// SASLTypePlaintext represents the SASL/PLAIN mechanism
	SASLTypePlaintext = "PLAIN"
	// SASLTypeSCRAMSHA256 represents the SCRAM-SHA-256 mechanism.
	SASLTypeSCRAMSHA256 = "SCRAM-SHA-256"
	// SASLTypeSCRAMSHA512 represents the SCRAM-SHA-512 mechanism.
	SASLTypeSCRAMSHA512 = "SCRAM-SHA-512"
	SASLTypeGSSAPI      = "GSSAPI"
	// SASLHandshakeV0 is v0 of the Kafka SASL handshake protocol. Client and
	// server negotiate SASL auth using opaque packets.
	SASLHandshakeV0 = int16(0)
	// SASLHandshakeV1 is v1 of the Kafka SASL handshake protocol. Client and
	// server negotiate SASL by wrapping tokens with Kafka protocol headers.
	SASLHandshakeV1 = int16(1)
	// SASLExtKeyAuth is the reserved extension key name sent as part of the
	// SASL/OAUTHBEARER initial client response
	SASLExtKeyAuth = "auth"
)

michaelsaah · 2023-03-20T17:22:18Z

here's the PR: #5763

@MovieStoreGuy did you ever get around to testing this? also playing with it and seeing the same error from sarama as above.

michaelsaah · 2023-03-20T18:00:06Z

hmmm so perhaps this comment means the implementation is known to be broken?

IBM/sarama#1985 (comment)

MovieStoreGuy · 2023-03-23T04:42:58Z

Sorry, yes I had tried to run and validate this however, how the client is configured, it wasn't possible to do so.

github-actions · 2023-05-23T03:29:19Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

TaoYang526 · 2023-07-19T10:45:15Z

@MovieStoreGuy Any updates on this? I saw the same issue.
IIUIC, AWS_MSK_ISM should be an internal mechanism and has not been supported by sarama yet, according to updates on SCRAMClientGeneratorFunc by this mechanism, maybe the saramaConfig.Net.SASL.Mechanism should be sarama.SASLTypeSCRAMSHA512 or sarama.SASLTypeSCRAMSHA256 ?

github-actions · 2023-09-18T03:31:36Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions · 2023-11-17T05:19:14Z

This issue has been closed as inactive because it has been stale for 120 days with no activity.

bpalermo · 2023-12-20T23:02:18Z

Should we leverage this? https://github.com/aws/aws-msk-iam-sasl-signer-go

koh-satoh-wpg · 2024-03-25T08:28:33Z

@flyerjayden did you happen to find any workaround?

Otherwise I believe this is the only way to go 🤔

Should we leverage this? https://github.com/aws/aws-msk-iam-sasl-signer-go

jnavarro86 · 2024-06-25T14:08:19Z

I have built and deployed this version and it is working for me using iam roles.

…S_MSK_IAM_OAUTHBEARER" to kafkaexporter (#32500) **Description:** This PR added the SASL mechanism "AWS_MSK_IAM_OAUTHBEARER" to kafkaexporter and kafkareceiver. This mechanism use the AWS MSK IAM SASL Signer for Go https://github.com/aws/aws-msk-iam-sasl-signer-go. This mechanism is added because the "AWS_MSK_IAM" is not working in our cluster and also in this [issue](#19747). We added an new mechanism instead of replace the existing one because we want to keep the backward compatibility just in case someone is using "AWS_MSK_IAM".  **Link to tracking Issue:** [19747](#19747) **Testing:** We built the images and tested the SASL mechanism in our team. We added related unit tests. **Documentation:** We updated the kafkaexporter and kafakreciever README on the SASL mechanism. --------- Co-authored-by: Sean Marciniak <30928402+MovieStoreGuy@users.noreply.github.com>

…S_MSK_IAM_OAUTHBEARER" to kafkaexporter (open-telemetry#32500) **Description:** This PR added the SASL mechanism "AWS_MSK_IAM_OAUTHBEARER" to kafkaexporter and kafkareceiver. This mechanism use the AWS MSK IAM SASL Signer for Go https://github.com/aws/aws-msk-iam-sasl-signer-go. This mechanism is added because the "AWS_MSK_IAM" is not working in our cluster and also in this [issue](open-telemetry#19747). We added an new mechanism instead of replace the existing one because we want to keep the backward compatibility just in case someone is using "AWS_MSK_IAM".  **Link to tracking Issue:** [19747](open-telemetry#19747) **Testing:** We built the images and tested the SASL mechanism in our team. We added related unit tests. **Documentation:** We updated the kafkaexporter and kafakreciever README on the SASL mechanism. --------- Co-authored-by: Sean Marciniak <30928402+MovieStoreGuy@users.noreply.github.com>

flyerjayden added bug Something isn't working needs triage New item requiring triage labels Mar 16, 2023

github-actions bot added the exporter/kafka label Mar 16, 2023

atoulme added enhancement New feature or request and removed bug Something isn't working needs triage New item requiring triage labels Mar 17, 2023

github-actions bot added the Stale label May 23, 2023

MovieStoreGuy removed the Stale label May 24, 2023

github-actions bot added the Stale label Sep 18, 2023

github-actions bot added the closed as inactive label Nov 17, 2023

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 17, 2023

donald-cheung mentioned this issue Apr 18, 2024

[kafka/internal, kafkaexporter, kafkareceiver] Add SASL mechanism "AWS_MSK_IAM_OAUTHBEARER" to kafkaexporter #32500

Merged

mattiasnensen mentioned this issue Jun 5, 2024

Add an otelcol.exporter.kafka component grafana/alloy#717

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kafka Exporter w/ AWS_MSK_IAM Mechanism Not Working #19747

Kafka Exporter w/ AWS_MSK_IAM Mechanism Not Working #19747

flyerjayden commented Mar 16, 2023

github-actions bot commented Mar 16, 2023

atoulme commented Mar 17, 2023

flyerjayden commented Mar 17, 2023

atoulme commented Mar 17, 2023

flyerjayden commented Mar 17, 2023

atoulme commented Mar 18, 2023

atoulme commented Mar 18, 2023

flyerjayden commented Mar 19, 2023

flyerjayden commented Mar 19, 2023 •

edited

Loading

michaelsaah commented Mar 20, 2023

michaelsaah commented Mar 20, 2023

MovieStoreGuy commented Mar 23, 2023

github-actions bot commented May 23, 2023

TaoYang526 commented Jul 19, 2023

github-actions bot commented Sep 18, 2023

github-actions bot commented Nov 17, 2023

bpalermo commented Dec 20, 2023

koh-satoh-wpg commented Mar 25, 2024

jnavarro86 commented Jun 25, 2024

Kafka Exporter w/ AWS_MSK_IAM Mechanism Not Working #19747

Kafka Exporter w/ AWS_MSK_IAM Mechanism Not Working #19747

Comments

flyerjayden commented Mar 16, 2023

Component(s)

What happened?

Description

Steps to Reproduce

Expected Result

Actual Result

Collector version

Environment information

Environment

OpenTelemetry Collector configuration

Log output

Additional context

github-actions bot commented Mar 16, 2023

atoulme commented Mar 17, 2023

flyerjayden commented Mar 17, 2023

atoulme commented Mar 17, 2023

flyerjayden commented Mar 17, 2023

atoulme commented Mar 18, 2023

atoulme commented Mar 18, 2023

flyerjayden commented Mar 19, 2023

flyerjayden commented Mar 19, 2023 • edited Loading

michaelsaah commented Mar 20, 2023

michaelsaah commented Mar 20, 2023

MovieStoreGuy commented Mar 23, 2023

github-actions bot commented May 23, 2023

TaoYang526 commented Jul 19, 2023

github-actions bot commented Sep 18, 2023

github-actions bot commented Nov 17, 2023

bpalermo commented Dec 20, 2023

koh-satoh-wpg commented Mar 25, 2024

jnavarro86 commented Jun 25, 2024

flyerjayden commented Mar 19, 2023 •

edited

Loading