Automated ETL Tool

"An automated pipeline tool which works like that --> [E]xtract images locally, [T]ransform them and apply a human pose estimation model on them and [L]oad them (+ more details) into PostgreSQL database system."

Quickstart

Download or clone this repo.

Install Docker on your system, following the instructions.

For formal reasons, the application has been developed in WSL2 Ubuntu 20.04 and conda as package and environment management.

Step 1: Configuration

Run the script config.sh parsing the following flags:

Flag	Values
-u	username
-p	password
-d	database's name
-t	table's name

Explanation

- -u, Define the username for the PostgreSQL DBMS.
- -p, Define the password for the PostgreSQL DBMS.
- -d, Give a name for the database that would you like to store the data.
- -t, Give a name for the table into the database which has been created using the above flag. 

For example, root$ bash config.sh -u myusername -p mypassword -d mydatabase -t mytable, where root is the path to the repo in your local filesystem.

After configuration three new files (db_config.ini, POSTGRES_USER.txt & POSTGRES_PASSWORD.txt) will appear in your filesystem. It's recommended to keep secret those files due to sensitive information.

Step 2: Docker-compose Build/Up Locally

Run the script buildup.sh parsing the following flags:

Flag	Values
-b	yes/YES, no/No
-u	yes/YES, no/No

Explanation

+ -b, yes/YES, if you want to build the images, otherwise no/NO.
+ -u, yes/YES, if you want to start running the docker containers in the background after building, otherwise no/NO.

For instance, the command root$ bash buildup.sh -b yes -u yes will build the docker images and then will start running the containers in the background and leaves them running.

See a synopsis of useful docker-compose commands here.

How it works?

A short preview

• For a better resolution, click on the image!

Post-building phase

1. Start running the docker using docker-compose up || docker-compose up -d.
2. app service will download and initiliaze the model. This step needs 2-3 minutes.
3. Then, it will process all images, if exist. Otherwise, it will exit.
4. When you upload new images, you have to run the service again using the command on step 1.
5. Running in the background is more convenient.

Streamlit Dashboard

1. Be sure that docker is running!
2. Open a web browser (i.e. Google Chrome)
   • search for http://localhost:8501/
3. See FAQ section in order to find out how the dashboard works.

PostgreSQL DB

1. Be sure that docker is running!
2. If you need to interact with your database write the below command:
   • docker-compose exec db bash
   • psql -h db -U myusername -d mydatabase
3. The table contains four columns (name CHAR(50), original_img BYTEA, rendered_img BYTEA, keypoints JSONB).

Human Pose Estimation model

1. More info about the model you can read from pytorch.org or a nice article from learnopencv.com.
2. The keypointrcnn_resnet50_fpn is particularly trained to identify key-points in a person.
3. The model infers better if there are no occlusions.