-
-
Notifications
You must be signed in to change notification settings - Fork 41
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
2 changed files
with
2,885 additions
and
6 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,101 @@ | ||
|
||
# Protocol Spec | ||
|
||
y-protocol implements different binary communication protocols for efficient exchange of information. | ||
|
||
This is the recommended approach to exchange Awareness updates, and syncing Yjs documents & incremental updates. | ||
|
||
## Base encoding approaches | ||
|
||
We use efficient variable-length encoding where possible. | ||
|
||
The protocol operates on byte arrays. We use the `•` operator to define concatenations on array buffers (i.e. `[1, 2] = [1] • [2]`). A "buffer" shall refer to any byte array. | ||
|
||
* `varUint(number)` - encodes a 53bit unsigned integer to 1-8 bytes. | ||
- unsigned integers are serialized 7 bits at a time, starting with the | ||
least significant bits. | ||
- the most significant bit (msb) in each output byte indicates if there | ||
is a continuation byte (msb = 1). | ||
- A reference implementation can be found in lib0: [encoding.writeVarUint](https://github.com/dmonad/lib0/blob/1ca4b11f355c5ccec25f20ac8d3e2382e6c4303c/encoding.js#L243) [decoding.readVarUint](https://github.com/dmonad/lib0/blob/1ca4b11f355c5ccec25f20ac8d3e2382e6c4303c/decoding.js#L235) | ||
* `varByteArray(buffer) := varUint(length(buffer)) • buffer` - allows us to read any buffer by prepending the size of the buffer | ||
* `utf8(string)` - transforms a string to a utf8-encoded byte array. | ||
* `varString(string) := varByteArray(utf8(string))` | ||
* `json(object) := varString(JSON.stringify(object))` - Write a JavaScript object as a JSON string. | ||
|
||
### Sync protocol (v1 encoding) | ||
|
||
The Sync protocol defines a few message types that allow two peers to efficiently sync Yjs documents with each other. For more information about Yjs updates and sync messages, please look at [Yjs Docs / Document Updates](https://docs.yjs.dev/api/document-updates). | ||
|
||
We initially sync using the state-vector approach. First, each client sends a `SyncStep1` message to the other peer that contains a `state-vector` (see Yjs docs). When receiving `SyncStep1`, one should reply with `SyncStep2` which contains the missing document updates (`Y.encodeStateAsUpdate(remoteYdoc, sv)`). Once a client receives `SyncStep2`, it knows that it is now synced with the other peer. From now on all changes on the Yjs document should be send to the remote client using an `Update` message containing the update messages generated by Yjs. | ||
|
||
#### Message types | ||
|
||
* `SyncStep1MessageType := 0` | ||
* `SyncStep2MessageType := 1` | ||
* `UpdateMessageType:= 2` | ||
|
||
#### Encodings | ||
|
||
* `syncStep1(sv) := varUint(SyncStep1MessageType) • varByteArray(sv)` - Initial sync request. The state vector can be received by calling `Y.encodeStateVector(ydoc)`. | ||
* `syncStep2(documentState) := varUint(SyncStep2MessageType) • varByteArray(documentState)` - As a reply to `SyncStep1`. The document state can be received by calling `Y.encodeStateAsUpdate(ydoc, sv)`. | ||
* `documentUpdate(update) := varUint(UpdateMessageType) • varByteArray(update)` - Incremental updates the Yjs event handler `Y.on('update', update => sendUpdate(update))`. The receiving part should apply incremental updates to the Yjs document `Y.applyUpdate(ydoc, update)`. | ||
|
||
### Awareness protocol | ||
|
||
The Awareness protocol synchronizes the pure state-based Awareness CRDT between peers. This can be useful to exchange ephemeral data like presence, cursor positions, etc.. | ||
|
||
Each peer is allocated a unique entry in the Awareness CRDT that only they can modify. Eventually, this state is going to be removed by either a timeout or by the owner setting the state to `null`. | ||
|
||
Since the Awareness CRDT is purely state-based, we always exchange the whole state of all locally known clients. Eventually, all Awareness CRDT instances will synchronize. The Awareness CRDT must remove clients from clients that haven't been updated for longer than `30` seconds. With each generated update, the `clock: uint` counter must increase. The Awareness CRDT must only apply updates if the received `clock` is newer / larger than the currently known `clock` for that client. | ||
|
||
`awarenessUpdate(clients) := varUint(clients.length) • clients.map(client => varUint(client.clientid) • varUint(client.clock) • json(client.state))` | ||
|
||
|
||
## Combining protocols | ||
|
||
The base protocols can be mixed with your own protocols. The y-protocol package only defines the "base" protocol layers that can be reused across communication providers. | ||
|
||
* `SyncProtocolMessageType := 0` | ||
* `AwarenessProtocolMessageType := 1` | ||
|
||
A message should start with the message-type (e.g. `SyncProtocolMessageType`) and be appended with a specify protocol message (e.g. `SyncStep1MessageType`) | ||
|
||
* E.g. encoding a SyncStep1 message over the communication protocol: `varUint(SyncProtocolMessage) • syncStep1(sv)` | ||
* E.g. encoding an awareness update over the communication protocol: `varUint(AwarenessProtocolMessageType) • awarenessUpdate(clients)` | ||
|
||
A communication provider could parse protocols as follows: | ||
|
||
```js | ||
|
||
import * as decoding from 'lib0/decoding' | ||
import * as encoding from 'lib0/encoding' | ||
import * as sync from 'y-protocols/sync' | ||
import * as awareness from 'y-protocols/awareness' | ||
|
||
const messageTypes = { | ||
[SyncProtocolMessageType]: sync.readSyncMessage, | ||
[SyncProtocolMessageType]: awareness.readAwarenessMessage, | ||
[YourCustomMessageType]: readCustomMessage | ||
} | ||
|
||
function readMessage (buffer) { | ||
const decoder = decoding.createDecoder(buffer) | ||
const messageType = decoding.readVarUint(decoder) | ||
const replyMessage = encoding.createEncoder() | ||
|
||
const messageHandler = messageTypes[messageType] | ||
if (messageHandler) { | ||
messageHandler(decoder, encoder, ydoc) | ||
if (encoding.length(encoder) > 0) { | ||
// the message handler wants to send a reply (e.g. after receiving SyncStep1 the client should respond with SyncStep2) | ||
provider.sendMessage(encoding.toUint8Array(encoder)) | ||
} | ||
} else { | ||
throw new Error('Unknown message type') | ||
} | ||
} | ||
|
||
``` | ||
|
||
|
||
|
Oops, something went wrong.