Enable, test and document the Kafka output

[roncohen]

It would be great to investigate if the APM Server can support Kafka as output.

• test that enabling the Kafka output works as expected
• make sure we have a test in CI that tests APM Server with the Kafka output
• be aware of sourcemap support when using Kafka
• document Kafka usage with APM Server

[graphaelli]

related but not blocking - elastic/beats#7034

[graphaelli]

initial results are promising. Tested with confluent 4.1.0:

$ kafka-console-consumer --bootstrap-server kafka:9092 --topic apm --from-beginning --max-messages 3
{"@timestamp":"2018-05-09T17:08:11.028Z","@metadata":{"beat":"apm-server","type":"doc","version":"6.3.0","topic":"apm"},"listening":"[::]:8200","beat":{"name":"407b36519d01","hostname":"407b36519d01","version":"6.3.0"}}
{"@timestamp":"2018-05-09T17:16:58.156Z","@metadata":{"beat":"apm-server","type":"doc","version":"6.3.0","topic":"apm"},"beat":{"name":"407b36519d01","hostname":"407b36519d01","version":"6.3.0"},"listening":"[::]:8200"}
{"@timestamp":"2018-05-09T17:17:18.051Z","@metadata":{"beat":"apm-server","type":"doc","version":"6.3.0","topic":"apm"},"context":{"response":{"status_code":200,"headers":{"accept-ranges":"bytes","cache-control":"public, max-age=0","last-modified":"Wed, 02 May 2018 14:08:34 GMT","etag":"W/\"243-163212eeed0\"","content-type":"text/html; charset=UTF-8","content-length":"579","x-powered-by":"Express","date":"Wed, 09 May 2018 17:17:18 GMT","connection":"close"}},"service":{"framework":{"name":"express","version":"4.16.3"},"name":"opbeans-node","agent":{"name":"nodejs","version":"1.5.0"},"language":{"name":"javascript"},"runtime":{"name":"node","version":"v6.14.1"}},"process":{"ppid":1,"title":"node /app/server.js","argv":["/usr/local/bin/node","/usr/local/lib/node_modules/pm2/lib/ProcessContainer.js","processes.config.js"],"pid":35},"system":{"hostname":"e9c91adbc3b5","architecture":"x64","platform":"linux","ip":"172.18.0.9"},"request":{"url":{"pathname":"/","full":"http://opbeans-node:3000/","raw":"/","protocol":"http:","hostname":"opbeans-node","port":"3000"},"socket":{"encrypted":false},"headers":{"host":"opbeans-node:3000","connection":"close","user-agent":"workload/2.4.3"},"http_version":"1.1","method":"GET"}},"beat":{"version":"6.3.0","name":"407b36519d01","hostname":"407b36519d01"},"processor":{"name":"transaction","event":"transaction"},"transaction":{"id":"ad51eb7b-2a89-4b80-8ed1-8ff621236313","name":"GET static file","duration":{"us":16276},"type":"request","result":"HTTP 2xx","sampled":true}}
Processed a total of 3 messages

[graphaelli]

@roncohen what version(s) of kafka need to be tested?

[roncohen]

@graphaelli good question. I'd go for whatever beats support initially.