Use higher level kafka Writer api instead of low level Conn api

[zemek]

What type of PR is this?

/kind bug

Any specific area of the project related to this PR?

/area outputs

What this PR does / why we need it:
If no messages to the kafka broker has been sent within connections.max.idle.ms set on the broker (default 10 minutes) then the connection gets closed. The existing way of setting up the kafka connection does not re-open the connection and fails with error messages like:

10.1.2.13:42642->10.2.3.4:9092: write: broken pipe

I also think it's an anti-pattern to explicitly set the partition number to write to, this means we cannot balance the load across multiple brokers.

---

From the kafka-go readme:

To produce messages to Kafka, a program may use the low-level Conn API, but
the package also provides a higher level Writer type which is more appropriate
to use in most cases as it provides additional features:

• Automatic retries and reconnections on errors.
• Configurable distribution of messages across available partitions.
• Synchronous or asynchronous writes of messages to Kafka.
• Asynchronous cancellation using contexts.
• Flushing of pending messages on close to support graceful shutdowns.

compared with: https://github.com/segmentio/kafka-go#connection-

The Conn type is the core of the kafka-go package. It wraps around a raw network connection to expose a low-level API to a Kafka server.

---

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

[Issif]

Thank you for this PR, it's really valuable. As it introduces a breaking change, we need to be very careful and clean in next Changelog.

@cpanato who's written this output will take a look for sure when he can. Thank you to both of you.

[cpanato]

/lgtm the code

/hold to get some confirmation if it can post messages to the kafka, I did not have time to do my own check

[poiana]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cpanato

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [cpanato]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[poiana]

LGTM label has been added.

Git tree hash: 607747c8cc33f1ae48f27bea789b494c38a4971a

[cpanato]

/retest

[leogr]

I'm trying to trigger the CI, so I'm closing and reopening
/close

[poiana]

@leogr: Closed this PR.

In response to this:

I'm trying to trigger the CI, so I'm closing and reopening
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[leogr]

/reopen

[poiana]

@leogr: Reopened this PR.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[zemek]

@cpanato @Issif Is there anything else needed to merge this PR?

[Issif]

@zemek definitely not, I removed the hold label, hope it will unlock the merge