Adding output with kafka has SSL security

[ducmeit1]

In century, K6 has supported sending the raw output format to Kafka Clusters. However, mostly Kafka usually secured by an authentication method. The Kafka Cluster usually protected by SSL. It requires the client must provide a certificate that was signed by Kafka Cluster to produce the data or consume.

This PR would like to improve the Kafka producer client with Sarama and supports SSL authentication.

Sample usage:

k6 run --out kafka="brokers={broker2,broker3:9092},topic=someTopic,format=json,tls_security=true,certificate=cert.pem,private_key=key.pem,certificate_authority=ca.pem,insecure_skip=true" scripts.js

Parameter requires:

• tls_security: true
• certificate: location of the certificate file (provide a relative or absolute path)
• private_key: location of the private key file
• certificate_authority: location of the CA (optional)
• insecure_skip_verify: true (set false, if you want to validate the CA certificate)
• topic: the topic name
• brokers: the brokers list

[ducmeit1]

Hi @mostafa.
Long time no see.
Could you help me test this Pull Request?
I already tested and it's worked in my local and remote server.

Thanks

[na--]

Thanks for making this pull request! I skimmed it and didn't spot any obvious issues, but a detailed review will probably have to wait until next week, sorry. This week we're focused on wrapping up the planned changes for k6 v0.28.0, which is going to be released early next week, if there are no issues. We'll review the PR after the release, so it should become a part of k6 v0.29.0, scheduled for mid-November.

[mostafa]

Hey @ducmeit1,

Good to see you back. Could you please address linting issues related to your code in here before my test/review:

https://app.circleci.com/pipelines/github/loadimpact/k6/719/workflows/cfe15faf-eb5b-4300-9cbf-b97529ac2990/jobs/12880

[ducmeit1]

Hey @ducmeit1,

Good to see you back. Could you please address linting issues related to your code in here before my test/review:

https://app.circleci.com/pipelines/github/loadimpact/k6/719/workflows/cfe15faf-eb5b-4300-9cbf-b97529ac2990/jobs/12880

Hi @mostafa , could you verify again!

[mostafa]

Wow, you are fast!

Please test your code against Go 1.15.1: https://app.circleci.com/pipelines/github/loadimpact/k6/723/workflows/05569c46-67ba-4992-adc2-591a2423962e/jobs/12897

BTW, I'm not sure if this is related to your changes.

[imiric]

Hi guys, the TestExternallyControlledRun failure is a known flaky test, some of which were fixed in #1357, but there are still a few that pop up ocasionally, so I don't think this is related to Go 1.15.

Thanks for your contribution @ducmeit1!

[ducmeit1]

Hi guys, the TestExternallyControlledRun failure is a known flaky test, some of which were fixed in #1357, but there are still a few that pop up ocasionally, so I don't think this is related to Go 1.15.

Thanks for your contribution @ducmeit1!

You're welcome! Hope this repository will be passed and releasing asap.

[mostafa]

Hey @imiric,

Thanks for the clarification, I suspected that doesn't have anything to do with this PR.

@ducmeit1 I'll test it then. 🙂

[mostafa]

@ducmeit1,

I've used https://github.com/lensesio/fast-data-dev to setup a Kafka dev-env and used this guide to transform trust-store and client JKS files into PEM files. Yet, I still get an error while running this command:

$ k6 run --out kafka=brokers=127.0.0.1:9093,topic=k6-output,format=json,tls_security=true,certificate=client.cer.pem,private_key=client.key.pem,certificate_authority=server.cer.pem,insecure_skip=false test.js

The error:

ERRO[0001] kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Using kafkatool, I've confirmed that JKS files work as expected, so the 127.0.0.1:9093 is available using SSL/TLS keys. To double-check, I've also used the following command to confirm that the SSL/TLS is working:

$ openssl s_client -debug -connect localhost:9093 -tls1

Also, the unencrypted channel works well on 127.0.0.1:9092.

Can you elaborate more on your testing environment?

[ducmeit1]

k6 run --out kafka=brokers=127.0.0.1:9093,topic=k6-output,format=json,tls_security=true,certificate=client.cer.pem,private_key=client.key.pem,certificate_authority=server.cer.pem,insecure_skip=false test.js

Could you set insecure_skip=true and retry

[mostafa]

Could you set insecure_skip=true and retry

Tried that, too. No luck!

[monwolf]

@ducmeit1 I would like to suggest not forcing to enable mTLS , in my case we are using Kafka with SSL only to cipher the communication not for authenticating the client, for this we are using ACLs.

[mstoykov]

Thanks for all the work done here, but given that we are:

1. depricateing kafka output
2. have moved it out as an extension

I will close this PR and ask that if you want to add this functionality to do so in the extension repo.