Sangre streams your backend queries in realtime to your clients minimizing the load via diffs.
Sangre lives with your current backend framework.
Example • About The Project • How it works • Limitations • Installation • Contact • Acknowledgments
This example is based on typescript/javascript but feel free to imagine it in your own language (see Limitations).
It showcases a user that has a list of followed users. Each of those users have bookmarked places (restaurants, bars, etc.).
async function main() {
const { app, postgresClient } = await setupExpressApp() // create your express app
// Create users node
const users = await DB.table('users').get({ id: 1 }).joinMany('followed',
await DB.table('users').joinMany('place')
)
// Plug the node to an endpoint
app.sangre('/followeds', users);
app.listen(8080)
}This exposes a websocket endpoint streaming the user (id==1) with its followed users populated with their places.
The websocket streams the updates as this data changes in the database.
Below is a screencast of this example with a demonstration of its websocket output :
Another working example is showcased with a flutter client here.
Sangre provides a generic solution for streaming complex backend queries to clients in realtime.
A complex query is an arbitrary nested query of structured database data and processing of this data in your native backend language (js, python, etc.), only typescript/js is supported ATM.
The result of such query is streamed to client using incremental updates, minimizing network load, and enabling offline sync.
Typical use case is a client-server topology, a mobile or web app consuming an API (expressjs, django, rails, etc.) in front of a database (postgres, mongo, mysql, firebase, etc.). You want realtime data in your app without all the complexity of developing your own data sync.
Sangre implements all the following features and let the developers focus on business logic.
| Features | |
|---|---|
| Realtime processing of relational database query + arbitrary transformation | ✔️ |
| Client streaming over websocket | ✔️ |
| Offline sync for clients | ✔️ |
| Minimal network load (incremental updates) | ✔️ |
| Can be embedded into an existing project | ✔️ |
| You need to adopt a new database | ❌ |
Sangre is an acyclic graph of operator nodes acting on data. This data flows in those nodes as streams of data for reactivness.
Nodes may be filters, joins, populators. Root nodes provide data from the underling database (typically postgres) and listen to changes (via supabase realtime) in order to spread them down the data flow. Leaf nodes are the endpoints consumed by client apps (via websocket).
At this point, Sangre is just a PoC. A lot of shortcuts have been taken to produce a working example. Here are known limitations, if you think about any other, please reach me out via my contact info.
| Limits to overcome | Feasibility |
|---|---|
| Horizontal scalability | ✔️ |
| Observability | ✔️ |
| Upqueries | ✔️ |
| Language agnostic (needs implementation in each) | ❌ |
| Strict consistency | ❌ (enventual only) |
| Parametrized queries | ✔️ |
| Share nodes between similar queries | ✔️ |
Diff algorithm is currently JSON patch. This can be easily changed for a more readable or effecient one (myers, histogram, yours ?)
Note : only postgres supported ATM (more to come)
Note : You can use .docker/docker-compose.yml to get a working example running
Run once on your postgres database :
ALTER SYSTEM SET wal_level = logical;
CREATE PUBLICATION supabase_realtime FOR ALL TABLES;Reload your postgres configuration :
pg_ctl reload
Sangre uses supabase/realtime to listen to database changes (insert/modify/delete of rows).
You can see details of its capabilities and installation process on their repo.
We give you an example of how to run it in docker compose :
realtime:
image: supabase/realtime
environment:
DB_HOST: <your_db_ip>
DB_PASSWORD: <your_db_password>
SECURE_CHANNELS: false
ports:
- 4000:4000
Project Link: https://github.com/pomarec/sangre
-
Inspirations :
- NoriaDB : Huge thanks to Jon Gjengset for clearing up my mind about this topic (Whitepaper). Sangre is not an implementation of this paper though.
- All the work done around materialized views and dataflows (Raw list of sources)
-
Supabase for their realtime tool that transforms postgres replica "stream" to websocket events (easier to consume)
-
Typescript / JS librairies in package.json
-
Dart librairies in example's client's pubspec