Skip to content

Latest commit

 

History

History
275 lines (227 loc) · 18.8 KB

README.md

File metadata and controls

275 lines (227 loc) · 18.8 KB
Raw
Awesome

Edge Computing & Internet Of Things 🛜

A qualitative compilation of production-ready edge computing projects with a focus on Edge-2-Cloud Data Engineering

Table of Contents

Projects in the curated list are filtered based on the following conditions:

  • High Quality
  • Production Ready
  • Well Documented
  • Well Maintained
  • Cloud Ready
  • Scalable

IoT on Hyperscalers

  • AWS IoT - Unlock your IoT data and accelerate business growth
  • Azure IoT Operations - Build interoperable IoT solutions that transform physical operations at scale.

Major Edge OSS Projects

  • Eclipse Ditto - the open-source project of Eclipse IoT that provides a ready-to-use functionality to manage the state of Digital Twins
  • Eclipse Hono - provides uniform (remote) service interfaces for connecting large numbers of IoT devices to a (cloud) back end.
  • Akri - Akri is a Kubernetes Resource Interface that lets you easily expose heterogeneous leaf devices
  • Akraino - LF Edge is an umbrella organization that establishes an open, interoperable framework for edge computing
  • Shifu - Shifu provides users with a transparent framework for full-scene device hosting and integrated software development

Edge Messaging

MQTT

  • HiveMQ - Enterprise MQTT Platform
  • Nats.io - Connective Technology for Adaptive Edge & Distributed Systems
  • Mosquitto - An Open Source MQTT v3.1/v3.1.1 Broker.
  • VerneMQ - A distributed MQTT message broker.
  • Waterstream - MQTT broker leveraging Apache Kafka as its own storage and distribution engine.
  • NanoMQ - A light-weight and Blazing-fast MQTT Broker for IoT Edge platform.

Industrial

  • Node CoAP - A client and server library for CoAP modeled after the http module.
  • Rust CoAP - A Constrained Application Protocol (CoAP) library implemented in Rust.
  • OPC-UA - Interoperability standard for the secure and reliable exchange of data
  • ModBus - A data communications protocol for use with Modicon PLC's

Brokerless

  • ZeroMQ Overview - An open-source universal messaging library
  • NanoMSG - a simple high-performance implementation of several "scalability protocols"
  • NNG - nanomsg-next-generation -- light-weight brokerless messaging

Websockets

  • uws - Tiny WebSockets (access to the C++ library, µWebSockets, via Node.js)
  • uWebsockets - is a lightweight and efficient MQTT broker designed to raise the bar for pub/sub performance.
  • Tungstenite WS - Lightweight stream-based WebSocket implementation for Rust.
  • Python Websockets - A library for building WebSocket servers and clients in Python
  • Autobahn.ws - Open-source real-time framework for Web, Mobile & Internet of Things.
  • socket.io - Bidirectional and low-latency communication for every platform

Message Serialization

  • Apache Avro - Apache Avro™ is a data serialization system
  • Apache Thrift - scalable cross-language services development
  • CapNProto - Apache Avro™ is a data serialization system.
  • JSON - is a lightweight data-interchange format.
  • MessagePack - MessagePack is an efficient binary serialization format.
  • CBOR - RFC 8949 Concise Binary Object Representation
  • FlatBuffers - an efficient cross platform serialization library
  • Protocol Buffer - a language-neutral, platform-neutral extensible mechanism for serializing structured data

Messaging and Data Streaming

Cloud Managed

  • Azure Event Hub - a fully managed, real-time data ingestion service that’s simple, trusted, and scalable
  • Azure Managed Kafka - Azure HDInsight, a customizable, enterprise-grade service for open-source analytics.
  • Azure Queue Storage - a service for storing large numbers of messages
  • Azure Event Grid - a highly scalable, serverless event broker that you can use to integrate applications using events.
  • Confluent Cloud - Apache Kafka as a Service
  • AWS MSK - Securely stream data with a fully managed, highly available Apache Kafka service
  • AWS SQS - Fully managed message queuing for microservices, distributed systems, and serverless applications
  • AWS Kinesis - Easily stream data at any scale

On-Prem

  • Strimzi - Cloud Native Apache Kafka
  • Apache Pulsar - a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
  • Apache Kafka - distributed publish-subscribe messaging system.
  • Apache NiFi - Apache NiFi is an integrated data logistics platform for automating the movement of data between disparate systems.
  • redpanda - A Kafka® replacement for mission critical systems; 10x faster. Written in C++.
  • Fluentd - tool to collect events and logs.
  • Kestrel - distributed message queue system.
  • Apache ActiveMQ Artemis - The Next Generation Message Broker by ActiveMQ, AMQP 1.0 is a natively supported protocol.
  • Apache Qpid Java Broker - A pure-Java AMQP message broker.
  • Solace - Messaging Platform that supports AMQP 1.0
  • RabbitMQ with AMQP 1.0 plugin.

Stream Processing

Cloud Managed

On-Prem

  • Faust - stream processing and event processing in Python
  • Siddhi - stream Processing and Complex Event Processing Engine
  • KSQL - The database purpose-built for stream processing applications.
  • Benthos - Declarative stream processing for mundane tasks and data engineering
  • Go Streams - A lightweight stream processing library for Go
  • Flogo - Elegant stream processing pipeline written entirely in Golang
  • Apache Heron - A realtime, distributed, fault-tolerant stream processing engine
  • Apache Beam - Implement batch and streaming data processing jobs that run on any execution engine.
  • Apache Flink - Stateful Computations over Data Streams
  • Apache Samza - A distributed stream processing framework

Batch Processing / Analytics

  • Databricks - All your data, analytics and AI on one Lakehouse platform
  • Apache Mahout - For Creating Scalable Performant Machine Learning Applications
  • H2O - The #1 open-source machine learning platform for the enterprise
  • Apache Drill - Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage.
  • Apache Spark - Apache Spark™ is a unified analytics engine for large-scale data processing.

Databases

Cloud Managed

  • Azure SQL - Build apps that scale with managed and intelligent SQL in the cloud
  • Azure CosmosDB - Fast NoSQL / Graph / Relational database with open APIs for planet scale
  • Azure PostgreSQL - Fully managed, intelligent, and scalable PostgreSQL
  • Azure Data Explorer - Fast and highly scalable data exploration service.
  • AWS Aurora - AWS Relational DB
  • AWS DynamoDB - fast, flexible, and serverless NoSQL database for any scale
  • AWS DocumentDB - Scale JSON workloads with ease using a MongoDB-compatible document database
  • AWS Keyspaces - Run your Apache Cassandra workloads on a scalable, highly available, and managed wide column database service.
  • AWS Neptune - Build applications that work with highly connected datasets using a fast, reliable graph database service.
  • AWS Ledger - Provide transparent, immutable, cryptographically verifiable transaction logs with a fully managed ledger database service.
  • AWS Timestream - Provide transparent, immutable, cryptographically verifiable transaction logs with a fully managed ledger database service.

On-Prem

Time Series Database

  • InfluxDB - Scalable datastore for metrics, events, and real-time analytics
  • TimescaleDB - An open-source time-series database optimized for fast ingest and complex queries
  • VictoriaMetrics - fast, cost-effective monitoring solution and time series database
  • Cortex - A horizontally scalable, highly available, multi-tenant, long term Prometheus.

Document Database (NoSQL)

  • Crate Data - is an open source massively scalable data store. It requires zero administration.
  • MongoDB - Document-oriented database system.
  • RavenDB - A transactional, open-source Document Database.
  • RethinkDB - document database that supports queries like table joins and group by.
  • Couchbase - The Modern Database for Enterprise Applications
  • Apache Cassandra - Manage massive amounts of data, fast, without losing sleep

Persistent KV DB

  • TiKV - a highly scalable, low latency, and easy to use key-value database.
  • BadgerDB - an embeddable, persistent and fast key-value (KV) database written in pure Go
  • FoundationDB - scalable distributed database
  • RocksDB - embeddable persistent key-value store for fast storage based on LevelDB.

Relational Database

  • Citus - Distributed PostgreSQL
  • Cockroach - Scalable, Geo-Replicated, Transactional Datastore.
  • TiDB - TiDB is a distributed SQL database
  • yugabyteDB - open source, high-performance, distributed SQL database compatible with PostgreSQL
  • MariaDB - enhanced, drop-in replacement for MySQL
  • Apache Calcite - the foundation for your next high-performance database
  • Vitess - A database clustering system for horizontal scaling of MySQL

In Memory Database

  • VoltDB - claims to be fastest in-memory database.
  • BerkeleyDB - a software library that provides a high-performance embedded database for key/value data.
  • LevelDB - a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
  • Memcached - high-performance, distributed memory object caching system

Industrial IoT with Kubernetes

Kubernetes Hybrid

  • Azure Arc - Azure Arc simplifies governance and management by delivering a consistent multicloud and on-premises management platform.
  • EKS Anywhere - Create and operate Kubernetes clusters on your own infrastructure

Kubernetes Distributions for Edge

  • AKS Edge - AKS Edge Essentials is an on-premises Kubernetes implementation running containers at scale
  • K3s - The certified Kubernetes distribution built for IoT & Edge computing
  • microk8s - The best Kubernetes experience for developers, DevOps, cloud and edge

Kubernetes Networking

  • Cilium - eBPF-based Networking, Observability, and Security
  • Calico - most widely adopted solution for Kubernetes networking and security
  • Flannel - Flannel is a simple and easy way to configure a layer 3 network fabric
  • Antrea - Kubernetes networking based on Open vSwitch
  • KubeEdge - An open platform to enable Edge computing
  • Kube Router - networking solution built for operational simplicity and performance.

Multi-Cluster Networking

  • Liqo - P2P, Seamless, Dynamic and Secure Resource Sharing
  • Admiralty - The simplest way to deploy applications to multiple Kubernetes clusters.
  • Submariner - direct networking between Pods and Services in different Kubernetes clusters
  • Skupper - Skupper Multicloud communication for Kubernetes

Service Mesh

  • Istio - the leading multi-cluster service mesh
  • Linkerd - multi-cluster services with security, observability, and reliability
  • Consul - Service Mesh for any runtime or cloud
  • Traefik Mesh - Traefik Mesh is a straight-forward, easy to configure, and non-invasive service mesh
  • Kuma - The universal Envoy service mesh for distributed service connectivity
  • Istio - the leading service mesh

Distributed Orchestration

  • Serf - decentralized solution for service discovery and orchestration
  • etcd - a distributed, reliable key-value store for the most critical data of a distributed system

Microservice Frameworks

  • Lightbend Akka - runtime for distributed, and fault tolerant event-driven applications on the JVM.
  • Lightbend Lagom - opinionated microservices framework
  • GoKit - A toolkit for microservices
  • Goa - Goa provides a holistic approach for developing remote APIs and microservices in Go.
  • Micro - Micro is a cloud platform for API development
  • Dapr - Dapr is a portable, event-driven, runtime for building distributed applications across cloud and edge.

Distributed Filesystem

  • Rook - Open-Source, Cloud-Native Storage for Kubernetes
  • Minio - Object Storage for the Era of the Hybrid Cloud
  • Longhorn - Cloud native distributed block storage for Kubernetes
  • OpenEBS - OpenEBS builds on Kubernetes to enable Stateful applications to easily access Persistent Volumes

FaaS

  • Serverless - zero-friction serverless development
  • Apache Camel - easily integrate various systems consuming or producing data.
  • Apache Openwhisk - Open Source Serverless Cloud Platform
  • Fission - Open source, Kubernetes-native Serverless Framework
  • KNative - Serverless Kubernetes functions
  • Kubeless - The Kubernetes Native Serverless Framework
  • Kyma - Extend your applications with Kubernetes
  • Nuclio - High-Performance Serverless event and data processing platform