An example express server using Meeshkan express middleware to record HTTP traffic to Kafka.
The full tutorial associated with this repository is available on the Meeshkan blog: Building a real-time HTTP traffic stream with Apache Kafka.
- Node.js >= 8.0 and optionally yarn
- Either Docker or Kafka installation
- Optional: Python 3.6+ and pip if building an OpenAPI specification with our HTTP Mocking Toolkit (HMT)
First, clone the repository and navigate inside it:
git clone https://github.com/meeshkan/meeshkan-express-kafka-demo.git
cd meeshkan-express-kafka-demo
Install Express server dependencies:
# If you're using npm
$ npm install
# If you're using yarn
$ yarn
If you have Docker, start a local Kafka cluster with one Zookeeper and Kafka node by using Docker Compose:
$ docker-compose up -d
Note: You can find the configuration for this in our docker-compose.yml file.
Alternatively, if you have Kafka installed, start Zookeeper and Kafka as instructed in Kafka documentation.
Create the destination topic:
# If you're using Docker
$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic http_recordings --create --partitions 3 --replication-factor 1
# If you have Kafka installed
$ docker exec kafka1 kafka-topics --bootstrap-server localhost:9092 --topic http_recordings --create --partitions 3 --replication-factor 1
Start the Express server:
# If you're using npm
$ npm run start
# If you're using yarn
$ yarn start
Start console consumer to read messages from Kafka:
# If you're using Docker
$ docker exec kafka1 kafka-console-consumer --bootstrap-server localhost:9092 --topic http_recordings --from-beginning
# If you have Kafka installed
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic http_recordings --from-beginning
Alternatively, if you use kafkacat
:
$ kafkacat -b localhost:9092 -t http_recordings -C
Start making calls to the server via curl
and see traffic being recorded in Kafka:
# Create a user with POST /users
$ curl -X POST -d '{"name": "Kimmo", "email": "kimmo@example.com" }' -H "Content-Type: application/json" http://localhost:3000/users
# Get user with GET /users/{id}
$ curl http://localhost:3000/users/5720862f-9534-4d97-8afe-c07f38a728b7
Note: The recorded traffic logs are in the HTTP Types format.
To create an OpenAPI specification from recordings, install hmt
:
$ pip install hmt
Then, run the following command:
$ hmt build --source kafka -o my_spec
Note: This instructs
hmt
to write the OpenAPI specification to themy_spec/
directory.
Finally, stop the Express server and close down the Kafka cluster:
$ docker-compose down
Create the topic:
$ kafka-topics.sh --bootstrap-server localhost:9092 --topic http_recordings --create --partitions 3 --replication-factor 1
List topics:
$ kafka-topics.sh --bootstrap-server localhost:9092 --list
Delete the topic:
$ kafka-topics.sh --bootstrap-server localhost:9092 --topic http_recordings --delete
Print consumer groups:
$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups