Kafka_Producer_Consumer_to_SQL_Project

Creating a Kafka producer.py, consumer.py & PgAdmin SQL DB

Required installations if not existing on the same environment:

pip install kafka-python
conda install -c conda-forge kafka-python
conda install -c anaconda sqlalchemy
conda install -c anaconda psycopg2

What the project about:

Created Upstash account
Created New single replica Cluster
producer.py Python base code

On VS created a producer.py to publish random message to Kafka every 5 seconds

consumer.py Python base code

On VS created a consumer.py to fitch the producer.py random messages from Kafka every 2 seconds

After activating the environment: $ conda activate kafka-env
On GitBash: $ python producer.py

After activating the environment: $ conda activate kafka-env
On GitBash: $ python consumer.py

using engine Pushing all produced data thru consumer.py to PgAdmin SQL Database

Benificial links:

https://upstash.com/?gclid=CjwKCAjwvpCkBhB4EiwAujULMu611BMTjPj1328j0xQ8lKUNu55HbCBBtwRG8ysGnVgpFEML33sTRxoCWcMQAvD_BwE https://pypi.org/project/psycopg2-binary/