data_dynamos_exercise

There are two ways to setup and run the code in this repository:

1. Setup Directly on the Machine: Install all the Software directly on your machine and run the code.
2. Setup Using Docker: Use a pre-built Docker image containing all the Software already and run the code inside Docker container.

Setup Directly on the Machine

Prerequisites

• Pyenv
• Python
• Poetry

Install Pyenv v2.3.22

curl https://pyenv.run | PYENV_GIT_TAG=v2.3.22 bash

Load pyenv automatically

For bash

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc
exec "$SHELL"

For Zsh

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc
exec "$SHELL"

If you are using shells other than bash or Zsh refer: https://github.com/pyenv/pyenv#set-up-your-shell-environment-for-pyenv

Install Python 3.8.16

pyenv --version
pyenv install 3.8.16
pyenv versions # should list Python 3.8.16 version

Install Poetry 1.5.1

curl -sSL https://install.python-poetry.org | POETRY_VERSION=1.5.1 python3 -

Add Poetry bin directory to PATH

For bash:

echo 'export PATH="$HOME/.local/bin/:$PATH"' >> ~/.bashrc
exec "$SHELL"

For Zsh:

echo 'export PATH="$HOME/.local/bin/:$PATH"' >> ~/.zshrc
exec "$SHELL"

Usage

Create virtualenv and Install Dependencies

# setup virtualenv and install the dependencies
poetry install

# run a command inside the virtualenv
poetry run dbt --version

# start a virtualenv shell and run commands inside it
poetry shell
dbt --version

# install dbt packages
cd data_transformation
dbt deps

Data Transformations

poetry shell

export SNOWFLAKE_ACCOUNT='<your-snowflake-account-name>'
export SNOWFLAKE_DATABASE='<your-database-name>'
export SNOWFLAKE_USER='<your-user-name>'
export SNOWFLAKE_PASSWORD='<your-password>'

cd data_transformation
dbt debug # should connect to snowflake successfully

Data Visualizations

poetry shell

export SNOWFLAKE_ACCOUNT='<your-snowflake-account-name>'
export SNOWFLAKE_DATABASE='<your-database-name>'
export SNOWFLAKE_USER='<your-user-name>'
export SNOWFLAKE_PASSWORD='<your-password>'

cd data_visualization
streamlit run DataDynamos.py # should be able to access the UI

Setup Using Docker

Prerequisites

• Docker

Usage

Note: The code will be mounted inside the container so whatever changes you make in your local machine will be synced to the container.

Data Transformations

• Start the container

docker run --rm -it -v $PWD:/opt/data_dynamos_exercise quay.io/data-dynamos/data_dynamos_exercise bash

• Run the code inside the container

export SNOWFLAKE_ACCOUNT='<your-snowflake-account-name>'
export SNOWFLAKE_DATABASE='<your-database-name>'
export SNOWFLAKE_USER='<your-user-name>'
export SNOWFLAKE_PASSWORD='<your-password>'

cd data_transformation
dbt debug # should connect to snowflake successfully

Data Visualizations

• Start the container

docker run --rm -it -v $PWD:/opt/data_dynamos_exercise quay.io/data-dynamos/data_dynamos_exercise -p 8501:8501 bash

• Run the code inside the container

export SNOWFLAKE_ACCOUNT='<your-snowflake-account-name>'
export SNOWFLAKE_DATABASE='<your-database-name>'
export SNOWFLAKE_USER='<your-user-name>'
export SNOWFLAKE_PASSWORD='<your-password>'

cd data_visualization
streamlit run DataDynamos.py # should be able to access the UI