My personal project for data engineering zoomcamp

This utilizes a static IP address in GCP to pull data from the Clash of Clans API, store it and serve it via data studio. There is also a API client with a sleep of functions you can use to pull, store, and view CoC facts and dimensions.

Easy-to-use Python library for managing files and data in different environments

Scripts used for analyzing GitHub commit locations to create a map visualization

Explore efficient big data matching techniques

A platform for users to upload and share the responses from law enforcement agencies to their traffic violation reports in Taiwan. This system aims to increase transparency and public oversight of traffic law enforcement.

A repo containing auto-triggered Airflow ETL activities for datasets located on GCP storage that flattens and creates analytical views on Big Query

Market Data Ingestion and BigQuery Integration project

Data ingestion from Google BigQuery to PostgreSQL

Webscraping dos dados do site Mercado Livre. Esses dados são exportados para uma base de dados BigQuery e algumas informações relevantes são obtidas através de consultas SQL.

Scrape Premier League Data and load it into Google BigQuery

A word cloud based on browser mentions in GitHub commit messages.

As part of my Data Engineering professional development, I am developing an array of data pipelines which extract, transform and load the data from various sources such as this Quandal API, CSV, JSON and database such as Google BigQuery, Microsft SQL Server and PostgreSQL, just to name a few.

ETL pipeline with pyspark in Google Cloud Platform (GCP) using infrastructure-as-code principles with Terraform

Tool for copy tables, delete tables and write queries for time range

Welcome to the Spotify Insights Data Pipeline Project where I analyze data from my Spotify listening history ~

This is the second version of the Google Serch Trends API. Having completed the initial testing of the first version. The next step was to create a series of functions in line with the principles of object-oriented programming. Initially was created in a linear format as a proof of concept. Later split the code into five main functions.