|
Note
|
Do not try to follow this if you’re going to make use of the remotely-provisioned environment. |
|
Note
|
Successfully tested on:
|
-
Docker
-
Docker Compose
-
8GB+ RAM
-
Clone github repository
-
Pull required docker images
|
Important
|
THIS MUST BE DONE BEFORE THE WORKSHOP - The docker images are large and will take time to download. Please don’t wait until the workshop as the wi-fi will not cope! |
-
Install Docker Compose on your system. There are Mac, Windows, and Linux options available at the link.
-
Mac/Windows only:
In Docker’s advanced settings, increase the memory dedicated to Docker to at least 8GB.
-
Confirm that Docker has at least 8GB of memory available to it:
docker system info | grep MemoryShould return a value greater than 8GB - if not, the Kafka stack will probably not work.
-
Clone the workshop repo to your local machine:
git clone https://github.com/confluentinc/demo-scene.git
-
Change to the ksqlDB workshop folder
cd demo-scene/build-a-streaming-pipeline -
Pull all required docker images—this will take a while!
docker-compose pull
-
Did you check that Docker has at least 8GB of memory available?
-
You really do need to do this
-
-
You may find it useful to have
jqinstalled-
Mac:
brew install jq -
RHEL-based:
sudo yum install -y jq -
Debian-based:
sudo apt install -y jq
-
|
Note
|
Make sure that Docker has at least 8GB of memory available (check with docker system info | grep Memory)
|
|
Note
|
For Windows instructions, please see ksql-workshop-windows.adoc |
Make sure you’re in the demo-scene/build-a-streaming-pipeline folder, and then launch the stack:
docker-compose up -d
docker-compose logs -f kafka|grep "INFO Kafka version"Once you see output, then it means Kafka is running and you can proceed
kafka | [2020-02-12 11:34:12,924] INFO Kafka version: 5.4.0-ccs (org.apache.kafka.common.utils.AppInfoParser)Press btn:[Ctrl-C] twice to exit the docker-compose logs command
Run docker-compose ps to confirm that all components are running:
$ docker-compose ps
Name Command State Ports
----------------------------------------------------------------------------------------------------
elasticsearch /usr/local/bin/docker-entr ... Up 0.0.0.0:9200->9200/tcp, 9300/tcp
kafka bash -c echo '127.0.0.1 ka ... Up 0.0.0.0:9092->9092/tcp
kafka-connect-01 bash -c echo "Installing c ... Up (healthy) 0.0.0.0:8083->8083/tcp, 9092/tcp
kafkacat /bin/sh -c apk add jq; Up
wh ...
kibana /usr/local/bin/dumb-init - ... Up 0.0.0.0:5601->5601/tcp
ksqldb /usr/bin/docker/run Up 0.0.0.0:8088->8088/tcp
mysql docker-entrypoint.sh bash ... Up 0.0.0.0:3306->3306/tcp, 33060/tcp
schema-registry /etc/confluent/docker/run Up 0.0.0.0:8081->8081/tcp
zookeeper /etc/confluent/docker/run Up 2181/tcp, 2888/tcp, 3888/tcp|
Important
|
If any components do not show "Up" under the State column (e.g., they say "Exit") then you must rectify this before continuing.
As a first solution, try re-issuing the docker-compose up -d command.
|
Now follow the instructions below ksqlDB prompt, and then open a second terminal window and start a MySQL prompt following the separate instructions below.
From a new shell session, run the following:
docker exec -it ksqldb ksql http://localhost:8088This will take you to the ksqlDB prompt:
===========================================
= _ _ ____ ____ =
= | | _____ __ _| | _ \| __ ) =
= | |/ / __|/ _` | | | | | _ \ =
= | <\__ \ (_| | | |_| | |_) | =
= |_|\_\___/\__, |_|____/|____/ =
= |_| =
= Event Streaming Database purpose-built =
= for stream processing apps =
===========================================
Copyright 2017-2019 Confluent Inc.
CLI v6.0.0-SNAPSHOT, Server v6.0.0-SNAPSHOT located at http://localhost:8088
Having trouble? Type 'help' (case-insensitive) for a rundown of how things work!
ksql>From a new shell session, run the following:
docker exec -it mysql bash -c 'mysql -u $MYSQL_USER -p$MYSQL_PASSWORD demo'This will take you to the MySQL prompt:
mysql: [Warning] Using a password on the command line interface can be insecure.
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.19 MySQL Community Server - GPL
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>Once you’ve finished with the workshop, terminate the workshop environment by running docker-compose down:
$ docker-compose down
Stopping ksqldb ... done
Stopping kafka-connect-01 ... done
Stopping kafkacat ... done
Stopping schema-registry ... done
Stopping kibana ... done
Stopping kafka ... done
Stopping elasticsearch ... done
Stopping zookeeper ... done
Stopping mysql ... done
Removing ksqldb ... done
Removing kafka-connect-01 ... done
Removing kafkacat ... done
Removing schema-registry ... done
Removing kibana ... done
Removing kafka ... done
Removing elasticsearch ... done
Removing zookeeper ... done
Removing mysql ... done
Removing network build-a-streaming-pipeline_defaultIf you want to preserve the state of all containers, run docker-compose stop instead.