Home

Getting started guide for data engineering tools

Welcome to the data engineering wiki! We started developing this guide for Insight Data Engineering Fellows and want to share it with the community at large as we feel it can benefit a lot of people trying to break into the field of data engineering.

This is a getting started guide for well adopted data engineering tools. It is intended for anyone who is looking to get hands-on experience with setting up these tools in a distributed manner and getting started with simple examples. It is a work in progress and we are working on adding more tools and advanced examples for the existing ones. If you have more ideas about tools/examples or if you find any mistakes, please feel free to create an issue.

Data Engineering Ecosystem

AWS

• AWS @ Insight
• Adding Volume to AWS EC2 Instances

Ingestion

• Kafka
• Kafka - Advanced
• Kinesis
• Camus
• Secor

File Systems

• HDFS
• AWS S3
• Alluxio

Batch Processing

• Hadoop Intro
• Hadoop MapReduce
• Hive
• Pig
• Spark Intro
• Presto

Stream Processing

• Flink Dev
• Spark Streaming Dev
• Storm Dev

Databases

• Cassandra Dev
• DynamoDB
• Elasticsearch Dev
• HBase Dev
• Neo4j
• Redis Dev

Web Framework

• Flask

Others

• CDH5
• Github Intro
• Maven
• Supervisor
• Zookeeper