Creating X509 certificates from JKS format

Kafka supports encryption and authentication via SSL. The required setup is described at http://kafka.apache.org/documentation.html#security_ssl. However, the generated certificates are in JKS format, which ruby-kafka does not support. Luckily, it is possible to convert the generated files into X509 format. Following is a guide how to do this.

When using SSL to encrypt communication, there are actually two different cases: encrypting communication between brokers (server-to-server) and between the client (ruby-kafka) and brokers (client-to-server).

First, we are going to take care of the client-to-server communication. We need to generate certificates in JKS and X509 format, so that both the Kafka process and ruby-kafka can use them.

## 1. Create certificate authority (CA)
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365 -passin pass:foobar -passout pass:foobar -subj "/CN=<domain>/OU=<unit>/O=<org>/L=<loc>/ST=<state>/C=<country>"
## 2. Create client keystore
keytool -noprompt -keystore kafka.client.keystore.jks -genkey -alias localhost -dname "CN=<domain>, OU=<unit>, O=<org>, L=<loc>, ST=<state>, C=<country>" -storepass foobar -keypass foobar
## 3. Sign client certificate
keytool -noprompt -keystore kafka.client.keystore.jks -alias localhost -certreq -file cert-unsigned -storepass foobar
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-unsigned -out cert-signed -days 365 -CAcreateserial -passin pass:foobar
## 4. Import CA and signed client certificate into client keystore
keytool -noprompt -keystore kafka.client.keystore.jks -alias CARoot -import -file ca-cert -storepass foobar
keytool -noprompt -keystore kafka.client.keystore.jks -alias localhost -import -file cert-signed -storepass foobar
## 5. Import CA into client truststore (only for debugging with Java consumer)
keytool -noprompt -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert -storepass foobar
## 6. Import CA into server truststore
keytool -noprompt -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert -storepass foobar
## 7. Create PEM files for Ruby client
### 7.1 Extract signed client certificate
keytool -noprompt -keystore kafka.client.keystore.jks -exportcert -alias localhost -rfc -storepass foobar -file client_cert.pem
### 7.2 Extract client key
keytool -noprompt -srckeystore kafka.client.keystore.jks -importkeystore -srcalias localhost -destkeystore cert_and_key.p12 -deststoretype PKCS12 -srcstorepass foobar -storepass foobar
openssl pkcs12 -in cert_and_key.p12 -nocerts -nodes -passin pass:foobar -out client_cert_key.pem
### 7.3 Extract CA certificate
keytool -noprompt -keystore kafka.client.keystore.jks -exportcert -alias CARoot -rfc -file ca_cert.pem -storepass foobar

Now, kafka.client.keystore.jks, kafka.client.truststore.jks and kafka.server.truststore.jks have to be stored on each broker, and referenced from server.properties and client.properties respectively. Note that the Java client setup is not really needed if you use ruby-kafka exclusively, but if you wanted to use the console consumer for quick testing it is required.

We also have the certificates in the format that ruby-kafka expects: client_cert.pem, client_cert_key.pem and ca_cert.pem. They can be used like this:

kafka = Kafka.new(
  ssl_ca_cert: File.read('ca_cert.pem'),
  ssl_client_cert: File.read('client_cert.pem'),
  ssl_client_cert_key: File.read('client_cert_key.pem'),
  # ...
)

Finally, to also secure communication between brokers, every broker needs to have its own server keystore, so you need to run the following on each host:

## 1. Create server keystore
keytool -noprompt -keystore kafka.server.keystore.jks -genkey -alias <hostname> -dname "CN=<hostname>, OU=<unit>, O=<org>, L=<loc>, ST=<state>, C=<country>" -storepass foobar -keypass foobar

## 2. Sign server certificate
keytool -noprompt -keystore kafka.server.keystore.jks -alias <hostname> -certreq -file cert-unsigned -storepass foobar
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-unsigned -out cert-signed -days 365 -CAcreateserial -passin pass:foobar

## 3. Import CA and signed server certificate into server keystore
keytool -noprompt -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert -storepass foobar
keytool -noprompt -keystore kafka.server.keystore.jks -alias <hostname> -import -file cert-signed -storepass foobar