Skip to content

Latest commit

 

History

History
253 lines (161 loc) · 12.4 KB

014-realtime-api.md

File metadata and controls

253 lines (161 loc) · 12.4 KB
Raw

Cross System Realtime API

Summary

We are adding a real-time API for the Appwrite server, which allows streaming and listening of all the Appwrite system events. The new real-time API will allow new and interesting use cases of building an app with Appwrite from simple chat apps, multiplayer games, live collaboration to real-time dashboards and enhanced UIs.

Unlike other BaaS products, Appwrite real-time should be completely cross-platform and work well from both client or backend. The new API should also be completely decoupled from the Appwrite database and transmit any supported system events that Appwrite produces, including storage, users, account events, and more.

Problem Statement (Step 1)

What problem are you trying to solve?

The Appwrite current REST API, and upcoming GraphQL are both HTTP based which is not suitable for realtime applications. HTTP classis request-response communication is not very well suited for listening and reporting of events.

Currently the only way to build realtime, events based application with Appwrite is by using a 3rd party server which is hard and complicated to implement espcially when data segmentatation and ACL is a requirement. An current alternative to using a 3rd party service, is to use HTTP-Polling which is costly and not optimized for performance.

What is the context or background in which this problem exists?

Allowing new use-cases for Appwrite, and giving more flexibily for end-developers relaying on Appwrite for user authentication and data proccessing.

Once the proposal is implemented, how will the system change?

We will intorduce a new server entrypoint, allowing any end-platform to connect and subscribe for Appwrite system events in realtime. Likewise, a new service will be provided in the client SDKs, which will allow developers to access these endpoints in a user-friendly way.

Design proposal (Step 2)

Architecture

We will implement the realtime API, using a new entrypoint for the Appwrite main container as demostrated at our POC branch. The new entrypoint will be called realtime and will server as the starting script for the new appwrite-realtime container.

The source for the new entrypoint should be located at the same equilvent to our http entrypoint (./app/http.php) at: /app/realtime.php. We will use the Swoole implementation of Websocket for the implementation of our first realtime protocol. Like as with other Appwrite related process, we should create a bash alias for a pretty entrypoint that will start the server (./bin/realtime).

Loadbalancer

We should update appwrite Traefik loadbalancer to redirect all ws & wss traffic for /v1/realtime to the new appwrite-realtime container.

An example configuration can also be found on the POC branch.

Protocols

As part of the Appwrite agenda of being cross-platform and tech agnostic, we should be able to design the new real-time API with multiple messaging protocols in mind. This does not mean we should support all of them, and for the scope of this RFC, we will only discuss the implementation of the Websockets protocol as the first protocol to support, while keeping in mind to avoid any coupling between the architecture and the resulting protocol.

Possible protocols to take under future considiration:

Channels

Channel Description
account All account related events (session create, name update...)
collections Any update/delete/create events for collections where user has read access
collections.[ID] Any update/delete/create events to a given collection where user has read access
collections.[ID].documents Any update/delete/create events to any document in a given collection where user has read access
documents Any update/delete/create events to documents where user has read access
documents.[ID] Any update/delete/create events to a given document where user has read access
files Any update/delete/create events to file where user has read access
files.[ID] Any update/delete/create events to a given file where user has read access
executions Any update to executions where user has read access
executions.[ID] Any update to a given function execution where user has read access
functions.[ID] Any execution event to a given function where user has read access

Message

The message from the server to the clients should be a JSON string and reflect the following object:

Property Description
event The name of the event equivalent to the system event.
channels An array of channels that can receive this message.
timestamp To ensure consistency across all client platforms and real-time technologies, the event timestamp is included.
payload Payload contains the data equal to the response model.

Error Message

This will be sent to the user if there was a invalid connection established, for eample providing a wrong project id:

Property Description
code A code that imitate the Websocket close codes.
message Related error message.

Subscriptions

Websocket

Subscriptions can be bound to channels and are passed as query parameters and part of the websocket URL. An example looks like this:

wss://appwrite.test/v1/realtime?project=XXXXXX&channels[]=account&channels=documents.XXXXXX

Or as SDK call in JavaScript:

let sdk = new Appwrite();

sdk
    .setEndpoint('https://appwrite.test') // Will also guess the WebSocket Endpoint by replacing protocols
    .setEndpointRealtime('wss://appwrite.test') // Optional: Realtime Endpoint
    .setProject('5df5acd0d48c2') // Your project ID
;

sdk.subscribe('account', response => {
  console.log(response); // Callback will be executed on account event.
});

const documentsUnsubscribe = sdk.realtime.subscribe('documents.XXXXXX', response => {
  console.log(response); // Callback will be executed on documents.XXXXXX event
});

documentsUnsubscribe(); // the subscribe() method will return a method to unsubscribe, invalidate the callback and remove the channel

Setting the Endpoint will also guess the Realtime Endpoint by replacing the http/https Protocol with the WebSocket equivalent. Alternatively you can set the Realtime Endpoint explicitly.

The client SDK will maintain a single connection and will maintain all the channels, if a channel is added or removed - a new connection will be established with given channels.

Scalability

The realtime container can easily be duplicated with Traefik load-balancing it.

Performance

References:

Security

Abuse Control

We should implement an abuse control check to prevent people from abusing the new connection action, which also acts as a validator for JWT tokens. This will completely prevent any large-scale attack from guessing tokens (that has a slim chance, to begin with) or abuse the server.

CORS Validation

On each new connection, we should validate that the client origin is valid similarly to what we do in the REST API. This is a good security practice to avoid project data being presented on un-authorized clients. This will force devs to list their platforms before using the real-time API.

Limit payload size

Since we will not support 2-way communication now - we shouldn't allow/handle receiving messages at all for now. Which makes the payload size not a concern for now.

Cookie Authentication

On HTTP Handshakes, cookies are usually sent along, which we can use for authentication. If a technology does not provide a native handshake - we can imitate this functionality to handle passing authentication tokens to the endpoint.

JWT Authentication (in path / or in message)

Using JWT authentication can be easier to implement and pass to the server.

Logs

We should add the following logs for easy debugging and monitoring of our realtime server. Below is a list of some of the logs we could initialy add:

  • Server start (stdout - using Console::success)
  • Worker start (stdout - using Console::success)
  • Connection Open (total connections per worker)
  • Connection Close (total connections per worker)
  • Errors (stderr - using Console::error)

Debug mode logs:

  • Event received (stdout - using Console::log)
  • Message sent (stdout - using Console::log)

Prior art

Unresolved questions

Future possibilities

2-way Communication

Messaging Channel