[WIP] readmegen: A utility for generating readme from specifications

[lovromazgon]

Description

Closes #114.

This PR contains an experimental utility readmegen that uses the connector specification to generate the connector readme. The specification contains a lot of useful information about how to configure the connector, but it's not enough to generate the full readme, as there might be intricacies that the user wants to add. Still, this could be a useful tool to generate the configuration info and keep it up to date.

Why WIP? We need to figure out how this could fit in the development flow of implementing a connector. Ideally we should be able to trigger a make target that outputs the updated readme, without the need to manually add more stuff. This would allow us to run it as part of a CI action that checks if generated files are up to date (example), this way we'd ensure that the readme contains the newest information after an update of the SDK or after we change some configuration parameters.

There are also some quirks like displaying the parameter values as code blocks.

How to try it out

Build the tool:

go build -o readmegen /cmd/readmegen/main.go

Navigate to a folder containing the connector:

cd /path/to/conduit-connector-kafka

Note that the connector needs to point to the SDK version that contains this tool for it to work, so you need to update the SDK dependency:

go get github.com/conduitio/conduit-connector-sdk@c5a0925

Now you can execute readmegen, which will detect the connector based on the directory you are in:

readmegen

This will output the following text to stdout:

Expand

Conduit Connector Kafka

A Kafka source and destination plugin for Conduit, written in Go.

Source

Configuration Parameters

Name	Type	Default	Description
`caCert`	1	``	caCert is the Kafka broker's certificate.
`clientCert`	1	``	clientCert is the Kafka client's certificate.
`clientID`	1	`conduit-connector-kafka`	clientID is a unique identifier for client connections established by this connector.
`clientKey`	1	``	clientKey is the Kafka client's private key.
`groupID`	1	``	groupID defines the consumer group id.
`insecureSkipVerify`	4	``	insecureSkipVerify defines whether to validate the broker's certificate chain and host name. If 'true', accepts any certificate presented by the server and any host name in that certificate.
`readFromBeginning`	4	``	readFromBeginning determines from whence the consumer group should begin consuming when it finds a partition without a committed offset. If this options is set to true it will start with the first message in that partition.
`saslMechanism`	1	``	saslMechanism configures the connector to use SASL authentication. If empty, no authentication will be performed.
`saslPassword`	1	``	saslPassword sets up the password used with SASL authentication.
`saslUsername`	1	``	saslUsername sets up the username used with SASL authentication.
`servers`	1	``	servers is a list of Kafka bootstrap servers, which will be used to discover all the servers in a cluster.
`tls.enabled`	4	``	tls.enabled defines whether TLS is needed to communicate with the Kafka cluster.
`topic`	1	``	topic is the Kafka topic.

Destination

Configuration Parameters

Name	Type	Default	Description
`acks`	1	`all`	acks defines the number of acknowledges from partition replicas required before receiving a response to a produce request. None = fire and forget, one = wait for the leader to acknowledge the writes, all = wait for the full ISR to acknowledge the writes.
`batchBytes`	2	`1000012`	batchBytes limits the maximum size of a request in bytes before being sent to a partition. This mirrors Kafka's max.message.bytes.
`caCert`	1	``	caCert is the Kafka broker's certificate.
`clientCert`	1	``	clientCert is the Kafka client's certificate.
`clientID`	1	`conduit-connector-kafka`	clientID is a unique identifier for client connections established by this connector.
`clientKey`	1	``	clientKey is the Kafka client's private key.
`compression`	1	`snappy`	compression set the compression codec to be used to compress messages.
`deliveryTimeout`	6	``	deliveryTimeout for write operation performed by the Writer.
`insecureSkipVerify`	4	``	insecureSkipVerify defines whether to validate the broker's certificate chain and host name. If 'true', accepts any certificate presented by the server and any host name in that certificate.
`saslMechanism`	1	``	saslMechanism configures the connector to use SASL authentication. If empty, no authentication will be performed.
`saslPassword`	1	``	saslPassword sets up the password used with SASL authentication.
`saslUsername`	1	``	saslUsername sets up the username used with SASL authentication.
`sdk.batch.delay`	6	`0s`	Maximum delay before an incomplete batch is written to the destination.
`sdk.batch.size`	2	`0`	Maximum size of batch before it gets written to the destination.
`sdk.rate.burst`	2	`0`	Allow bursts of at most X writes (1 or less means that bursts are not allowed). Only takes effect if a rate limit per second is set.
`sdk.rate.perSecond`	3	`0`	Maximum times records can be written per second (0 means no rate limit).
`sdk.record.format`	0	`opencdc/json`	The format of the output record.
`sdk.record.format.options`	0	``	Options to configure the chosen output record format. Options are key=value pairs separated with comma (e.g. opt1=val2,opt2=val2).
`servers`	1	``	servers is a list of Kafka bootstrap servers, which will be used to discover all the servers in a cluster.
`tls.enabled`	4	``	tls.enabled defines whether TLS is needed to communicate with the Kafka cluster.
`topic`	1	``	topic is the Kafka topic.