Skip to content

Playground for Kafka/Confluent Docker experimentations

Notifications You must be signed in to change notification settings

NathanNam/kafka-docker-playground

 
 

Repository files navigation

kafka-docker-playground Build Status

Playground for Kafka/Confluent Docker experimentations...

ℹ️ How to run

Table of Contents

🔗 Kafka Connectors

Quick start examples from Confluent docs but in Docker version for ease of use.

Connector Product Category Latest Version (*) Type Release Date Travis
ActiveMQ Sink Message Queue 1.1.6 Confluent Subscription 2020-08-21
ActiveMQ Source Message Queue 5.5.2 Confluent Subscription 2020-09-30
Amazon CloudWatch Logs Source Analytics 1.0.4 Confluent Subscription 2020-06-03
Amazon CloudWatch Metrics Sink Analytics 1.1.2 Confluent Subscription 2020-06-08
Amazon DynamoDB Sink Database 1.1.1 Confluent Subscription 2020-08-20
Amazon Kinesis Source Message Queue 1.3.0 Confluent Subscription 2020-08-31
Amazon Redshift Sink Data Warehouse 1.0.2 Confluent Subscription 2020-04-09
Amazon Redshift Source (using JDBC) Data Warehouse 5.5.2 Confluent Community License 2020-09-30
Amazon S3 Sink Datastore 5.5.2 Confluent Community License 2020-09-30
Amazon S3 Source Datastore 1.3.2 Confluent Subscription 2020-06-26
Amazon SQS Source Message Queue 1.0.6 Confluent Subscription 2020-09-23
AMPS Source Message Queue 1.0.0-preview Confluent Subscription 2020-07-08
Apache Kudu Source Database 1.0.1 Confluent Subscription 2020-01-07
Apache Kudu Sink Database 1.0.1 Confluent Subscription 2020-01-07
AWS Lambda Sink SaaS Apps 1.0.4 Confluent Subscription 2020-10-01
Azure Blob Storage Sink Datastore 1.4.0 Confluent Subscription 2020-07-24
Azure Blob Storage Source Datastore 1.3.2 Confluent Subscription 2020-06-26
Azure Data Lake Storage Gen1 Sink Datastore 1.4.0 Confluent Subscription 2020-07-24
Azure Data Lake Storage Gen2 Sink Datastore 1.4.0 Confluent Subscription 2020-07-24
Azure Event Hubs Source Message Queue 1.0.3 Confluent Subscription 2020-03-28
Azure Search Sink Analytics 1.0.1 Confluent Subscription 2020-01-07
Azure Service Bus Source Message Queue 1.1.0 Confluent Subscription 2020-03-13
Azure SQL Data Warehouse Sink Data Warehouse 1.0.3 Confluent Subscription 2020-04-24
Cassandra Sink Database 1.2.2 Confluent Subscription 2020-06-19
Couchbase Sink Database 3.4.8 Open Source (Couchbase)
Couchbase Source Database 3.4.8 Open Source (Couchbase)
Debezium CDC Microsoft SQL Server Source CDC 1.2.2 Open Source (Debezium Community)
Debezium CDC MySQL Source CDC 1.2.2 Open Source (Debezium Community)
Debezium CDC PostgreSQL Source CDC 1.2.2 Open Source (Debezium Community)
Debezium CDC MongoDB Source CDC 1.2.2 Open Source (Debezium Community)
Data Diode Sink Logs 1.1.1 Confluent Subscription 2019-10-18
Data Diode Source Logs 1.1.1 Confluent Subscription 2019-10-18
Datadog Metrics Sink Analytics 1.1.2 Confluent Subscription 2020-06-08
ElasticSearch Sink Analytics 10.0.0 Confluent Community License 2020-09-24
FTPS Sink Datastore 1.0.3-preview Confluent Subscription 2020-10-01
FTPS Source Datastore 1.0.3-preview Confluent Subscription 2020-10-01
Gemfire Sink Database 1.0.1 Confluent Subscription 2019-11-19
Github Source SaaS Apps 1.0.1 Confluent Subscription 2020-06-05
Google BigQuery Sink Data Warehouse 1.6.1 Open Source (WePay)
Google Cloud Functions Sink SaaS Apps 1.1.1 Confluent Subscription 2020-09-22
Google Cloud Pub/Sub Source Message Queue 1.0.2 Confluent Subscription 2020-01-07
Google Cloud Spanner Sink Database 1.0.2 Confluent Subscription 2019-10-30
Google Cloud Storage Sink Datastore 5.5.3 Confluent Subscription 2020-10-01
Google Cloud Storage Source Datastore 1.3.2 Confluent Subscription 2020-06-26
Google Firebase Realtime Database Sink Database 1.1.1 Confluent Subscription 2020-01-07
Google Firebase Realtime Database Source Database 1.1.1 Confluent Subscription 2020-01-07
HBase Sink Database 1.0.5 Confluent Subscription 2020-07-20
HDFS 2 Source Datastore 1.3.2 Confluent Subscription 2020-06-26
HDFS 3 Source Datastore 1.3.2 Confluent Subscription 2020-06-26
HDFS 2 Sink Datastore 5.5.2 Confluent Community License 2020-09-30
HDFS 3 Sink Datastore 1.0.5 Confluent Subscription 2020-03-04
HTTP Sink SaaS Apps 1.0.16 Confluent Subscription 2020-08-03
IBM MQ Sink Message Queue 1.3.0 Confluent Subscription 2020-09-15
IBM MQ Source Message Queue 5.5.2 Confluent Subscription 2020-09-30
InfluxDB Sink Database 1.2.1 Confluent Subscription 2020-09-30
InfluxDB Source Database 1.2.1 Confluent Subscription 2020-09-30
JDBC Hive Sink Database 5.5.2 Confluent Community License 2020-09-30
JDBC MySQL Sink Database 5.5.2 Confluent Community License 2020-09-30
JDBC Oracle 11 Sink Database 5.5.2 Confluent Community License 2020-09-30
JDBC Oracle 12 Sink Database 5.5.2 Confluent Community License 2020-09-30
JDBC PostGreSQL Sink Database 5.5.2 Confluent Community License 2020-09-30
JDBC Microsoft SQL Server Sink Database 5.5.2 Confluent Community License 2020-09-30
JDBC Vertica Sink Database 5.5.2 Confluent Community License 2020-09-30
JDBC MySQL Source Database 5.5.2 Confluent Community License 2020-09-30
JDBC Oracle 11 Source Database 5.5.2 Confluent Community License 2020-09-30
JDBC Oracle 12 Source Database 5.5.2 Confluent Community License 2020-09-30
JDBC PostGreSQL Source Database 5.5.2 Confluent Community License 2020-09-30
JDBC Microsoft SQL Server Source Database 5.5.2 Confluent Community License 2020-09-30
JIRA Source SaaS Apps 1.0.0-preview Confluent Subscription 2020-03-30
JMS ActiveMQ Sink Message Queue 1.3.0 Confluent Subscription 2020-09-15
JMS Solace Sink Message Queue 1.3.0 Confluent Subscription 2020-09-15
JMS TIBCO EMS Sink Message Queue 1.3.0 Confluent Subscription 2020-09-15
JMS TIBCO EMS Source Message Queue 5.5.2 Confluent Subscription 2020-09-30
Mapr Sink Datastore 1.1.1 Confluent Subscription 2020-02-10
Minio Sink Datastore 5.5.2 Confluent Community License 2020-09-30
MongoDB Sink Database 1.2.0 Open Source (MongoDB) 2020-07-08
MongoDB Source Database 1.2.0 Open Source (MongoDB) 2020-07-08
MQTT Sink IoT 1.3.0 Confluent Subscription 2020-08-28
MQTT Source IoT 1.3.0 Confluent Subscription 2020-08-28
Neo4j Sink Database 1.0.9 Open Source (Neo4j, Inc.) 2020-09-02
OmniSci Sink Database 1.0.2 Confluent Subscription 2019-08-20
PagerDuty Sink SaaS Apps 1.0.1 Confluent Subscription 2020-07-20
Prometheus Sink Analytics 1.1.2-preview Confluent Subscription 2020-06-08
RabbitMQ Sink Message Queue 1.3.1-preview Confluent Subscription 2020-09-22
RabbitMQ Source Message Queue 1.3.1 Confluent Subscription 2020-09-22
Redis Sink Database 0.0.2.11 Open Source (Jeremy Custenborder) 2020-01-22
SalesForce Bulk API Sink SaaS Apps 1.7.0 Confluent Subscription 2020-09-18
SalesForce Bulk API Source SaaS Apps 1.7.0 Confluent Subscription 2020-09-18
SalesForce CDC Source SaaS Apps 1.7.0 Confluent Subscription 2020-09-18
SalesForce Platform Events Sink SaaS Apps 1.7.0 Confluent Subscription 2020-09-18
SalesForce Platform Events Source SaaS Apps 1.7.0 Confluent Subscription 2020-09-18
SalesForce PushTopics Source SaaS Apps 1.7.0 Confluent Subscription 2020-09-18
SalesForce SObject Sink SaaS Apps 1.7.0 Confluent Subscription 2020-09-18
ServiceNow Sink SaaS Apps 2.0.1 Confluent Subscription 2020-07-28
ServiceNow Source SaaS Apps 2.0.1 Confluent Subscription 2020-07-28
SFTP Sink Datastore 2.1.1 Confluent Subscription 2020-09-02
SFTP Sink Datastore 2.1.1 Confluent Subscription 2020-09-02
SNMP Trap Source IoT 1.1.2 Confluent Subscription 2020-04-09
Snowflake Sink Data Warehouse 1.4.3 Open Source (Snowflake, Inc.) 2020-08-05
Solace Sink Message Queue 1.3.0 Confluent Subscription 2020-09-15
Solace Source Message Queue 1.2.0 Confluent Subscription 2020-08-26
Splunk Sink Analytics 2.0 Open Source (Splunk)
Splunk Source Analytics 1.0.2 Confluent Subscription 2020-01-08
Spool Dir Source Datastore 2.0.46 Open Source (Jeremy Custenborder) 2020-08-26
Syslog Source Logs 1.3.1 Confluent Subscription 2020-07-22
TIBCO EMS Sink Message Queue 1.3.0 Confluent Subscription 2020-09-15
TIBCO EMS Source Message Queue 1.2.0 Confluent Subscription 2020-08-26
Vertica Sink Database 1.2.2 Confluent Subscription 2020-09-07
Zendesk Source SaaS Apps 1.0.1 Confluent Subscription 2020-06-04

* You can change default connector version by setting CONNECTOR_TAG environment variable before starting a test, get more details here

☁️ Confluent Cloud

  • How to connect your components to Confluent Cloud
  • How to monitor your Confluent Cloud cluster
  • How to restrict access
  • etc...

Diagram

🔗 Kafka Connectors connected to Confluent Cloud

Other

🔄 Confluent Replicator and Mirror Maker 2

Using Multi-Data-Center setup with US 🇺🇸 and EUROPE 🇪🇺 clusters.

🔐 Environments

Single cluster:

Multi-Data-Center setup:

  • PLAINTEXT: no security
  • SASL_PLAIN: no SSL encryption, SASL/PLAIN authentication
  • Kerberos: no SSL encryption, Kerberos GSSAPI authentication

Confluent Commercial

Easily play with Confluent Platform Ansible playbooks by using Ubuntu based Docker images generated daily from this cp-ansible-playground repository

There is also a Confluent Cloud version available here

👾 Other Playgrounds

📚 Useful Resources

About

Playground for Kafka/Confluent Docker experimentations

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Shell 81.6%
  • Java 10.9%
  • C# 3.5%
  • PLpgSQL 1.7%
  • Dockerfile 1.4%
  • Go 0.7%
  • Other 0.2%