SpringBoot+Spark Streaming+Flume+Kafka+Redis+MySql Big Data--Gas Station Management System
👏👏🎉🎉🎉🎉👏👏
Base project made with much ❤️. Contains CRUD, patterns, generated library, and much more!
- Developed data analysis system with Spark, Kafka, ZooKeeper, MySQL, InnoDB, Flume, SpringBoot, and Echarts
- Integrated data distributed on 30 servers to Kafka in JSON using Flume and implemented data ingestion and handling clusters in real-time processing using Kafka.
- -Used ZooKeeper to co-ordinate cluster services.
- Consume live data from Kafka topics with Spark Streaming to create continuous sequences of RDDs.
- -Generated distributed computing Map and Reduce for RDD stream to improve writing speed to MySQL, and implement transaction features with InnoDB.
- -Transfer offsets generated by Spark Streaming to MySQL and Implemented visualization with SpringBoot and Echarts.
This repo is using MySQL. We use MySQL to handle CRUD operations over the Kafka offset, and to store station data.
Have a bug or a feature request? Please first read the issue guidelines and search for existing and closed issues. If your problem or idea is not addressed yet, please open a new issue.
Azanda Lyu
Thanks to all contributors and their support.
If you have an idea or you want to do something, tell me or just do it! I'm always happy to hear your feedback!
Code and documentation copyright 2020 the authors. Code released under the MIT License.