Skip to content

Commit

Permalink
Feat: Ceramic API using OpenAPI (#141)
Browse files Browse the repository at this point in the history 
* Feat: Ceramic API using REST

* Address comments

* Indenting?

* Byte offset and limit

* Add impl. details
  • Loading branch information
@oed
oed authored Aug 4, 2023
1 parent f4f04fd commit 1f266f3
Showing 1 changed file with 134 additions and 85 deletions.
219 changes: 134 additions & 85 deletions CIPs/cip-137.md
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
cip: 137
title: Ceramic API
author: Danny Browning <dbrowning@3box.io>, Nathaniel Cook <nathaniel@3box.io>, Aaron Goldman <aaron@3box.io>, Joel Thorstensson <joel@3box.io>
author: Danny Browning <dbrowning@3box.io>, Nathaniel Cook <nathaniel@3box.io>, Aaron Goldman <aaron@3box.io>, Joel Thorstensson <joel@3box.io>, Spencer T Brody <spencer@3box.io>
discussions-to: https://forum.ceramic.network/t/cip-137-ceramic-api/
status: Draft
category: Interface
Expand All @@ -18,190 +18,239 @@ An API for reading and writing events on Ceramic event streams.

## Abstract
<!--A short (~200 word) description of the technical issue being addressed.-->
This CIP introduces a JSON-RPC interface that can be used to subscribe to a number of streams inside of an interest range, and write new events to these streams. The API exposes two main ways of retrieving events. A simple method that uses the nodes local ordering of events to retrieve events by simply keeping track of a single vector client side. More advanced users can use a simplified version of the [Recon](https://cips.ceramic.network/CIPs/cip-124) protocol to retrieve events in their logical order.
This CIP introduces a OpenAPI specification that can be used to subscribe to a number of streams inside of an interest range, and write new events to these streams. The API exposes two main ways of retrieving events. A simple method that uses the nodes local ordering of events to retrieve events by simply keeping track of a single vector client side. More advanced users can use a simplified version of the [Recon](https://cips.ceramic.network/CIPs/cip-124) protocol to retrieve events in their logical order.


## Motivation
<!--Motivation is critical for CIPs that want to change the Ceramic protocol. It should clearly explain why the existing protocol specification is inadequate to address the problem that the CIP solves. CIP submissions without sufficient motivation may be rejected outright.-->
In order to enable databases other than ComposeDB to be built on top of Ceramic, an API to read and write directly on event streams is needed. This also enables developers to get direct access to the event streams underlaying ComposeDB.
In order to enable services and databases other than ComposeDB to be built on top of Ceramic, an API to read and write directly on event streams is needed. This also enables developers to get direct access to the event streams underlaying ComposeDB.


## Specification
<!--The technical specification should describe the syntax and semantics of any new feature.-->

This is a work in progress specification that introduces a JSON-RPC interface for interacting directly with event streams on a node.
This specification introduces a OpenAPI spec for interacting directly with event streams on a node.



### `ceramic_subscribe`
### GET `/ceramic/subscribe/{streamid}`

This method tells the Ceramic node to subscribe to a given interest range of streams. A interest range is defined as `ceramic://*?<sort-key>=<sort-value>`, where *sort-key* describes a field in the header of the InitEvent of a stream, and *sort-value* is the value of this field. For example to subscribe to all MIDs of a particular Model you can use `ceramic://*?model=kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr`.
This endpoint is called to subscribe to new events in a given interest range. This endpoint uses the node’s local ordering of events within the given range, constructed in the order in which the node received the events. If you want the global ordering of events you can use the *recon* endpoint.

Once subscribed the Ceramic node will use the Recon protocol to keep in sync with other nodes subscribed to this interest range.
The range is normally defined as `ceramic://*?<sort-key>=<sort-value>`, where *sort-key* describes a field in the header of the InitEvent of a stream, and *sort-value* is the value of this field. For example to subscribe to all MIDs of a particular Model you can use `ceramic://*?model=kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr`.

Active subscriptions need to be persisted. Or can be stateless somehow? (Nathaniel to share more thoughts here)
This endpoint will respond as soon as the node knows about new events, given the *offset*, or when *duration* amount of seconds has passed. If the node already knows about a lot of new events, only *limit* amount of events will be returned.

**Params:**
The first time a client makes a request for a given interest range, the node creates a log file where eventId bytes are written as they are observed from the network (or locally). This log file is then used to allow the node to easily find events given an *offset* from the client.

- *ceramicUrl* <string> - the Ceramic URL to subscribe to
- *options* <object> - options for the subscription
- *ctrlRangeStart* <string> - hex encoded string describing the start of the range of controllers to sync (defaults to `0x0000000000000000`)
- *ctrlRangeEnd* <string> - hex encoded string describing the end of the range of controllers to sync (defaults to `0xffffffffffffffff`)
#### Example:

**Returns:**
```
/ceramic/subscribe/*?model=kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr&ctrlRangeStart=0x0000000000000000&ctrlRangeEnd=0xffffffffffffffff&offset=123&limit=5&duration=1000
```

- *range* <Array<string>> - a Recon message, representing the latest state of the range described by the subscription request. In the form of `[eventid, ahash, eventid]`
#### Parameters:

---
* *streamid* <string> - the stream to subscribe to (Normally `*` is used to subscribe to sets of streams)

* *{sort-key}* <string> - the *sort-key* and *sort-value* to subscribe to (e.g. `model` and `kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr` respectively)
* *ctrlRangeStart* <string> - hex encoded string describing the start of the range of controllers to sync (defaults to `0x0000000000000000`)
* *ctrlRangeEnd* <string> - hex encoded string describing the end of the range of controllers to sync (defaults to `0xffffffffffffffff`)
* *offset* <integer> - the number of eventId bytes already consumed
* *limit* <integer> - the max number of events to return
* *duration* <integer> - the amount of time, in seconds, to wait for a response


### `ceramic_unsubscribe`
#### Returns:

This method tells the Ceramic node to unsubscribe from a given interest range.
There are two different content types supported as return value, as specified by the [`Accept`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept) http header.

**Params:**
**JSON:**

- *ceramicUrl* <string> - the Ceramic URL to unsubscribe from
- *options* <object> - options for the subscription
- *ctrlRangeStart* <string> - hex encoded string describing the start of the range of controllers to sync (defaults to `0x0000000000000000`)
- *ctrlRangeEnd* <string> - hex encoded string describing the end of the range of controllers to sync (defaults to `0xffffffffffffffff`)
Content-Type: `application/json`

**Returns:**
This content type returns a json representation of the events received. Note that this is just a representation of the event content, full verifiability is not retained.

- *success* <boolean> - true if the request was successful
- *events* <array<object>> - the list of events
- *eventId* <string> - the id of the event
- *id* <string> - CID of the InitEvent for this stream
- *prev* <array<string>> - CIDs of previous hash linked events
- *header* <object> - header for the event
- *data* <object> - the data of the event
- *timestamp* <integer> - the unixtime this event was timestamped (if it has been)
- *offset* <integer> - the number of eventId bytes already consumed

---


**CAR file:**

Content-Type: `application/vnd.ipld.car`

This content types returns the underlaying IPLD data structure of the event streams encoded as a CAR file. This data can be used to trustlessly verify the integrity of the event stream.

The root object of the CAR file is a *SubscriptionResult* which contains an array of *EvenIds* the new *offset*. The CAR file also includes all *Event* objects, corresponding to the eventids in the subscription result.

### `ceramic_listSubscriptions`
It's worth noting that an *EventId* always includes the CID of its *Event*.

List interest ranges that this node is subscribed to.
```verilog
// https://developers.ceramic.network/protocol/streams/event-log/#data-event
type Event InitEvent | DataEvent | TimeEvent
// https://cips.ceramic.network/CIPs/cip-124#eventids
type EventId Bytes // sort-data + Event CID
type SubscriptionResult {
events [EventId]
offset Integer
}
```

**Returns:**

* *subscriptions* <array<object>> - a list of all subscriptions
* *ceramicUrl* <string> - the Ceramic URL of the subscription
* *ctrlRangeStart* <string> - hex encoded string describing the start of the range of controllers to sync (defaults to `0x0000000000000000`)
* *ctrlRangeEnd* <string> - hex encoded string describing the end of the range of controllers to sync (defaults to `0xffffffffffffffff`)

---

### GET `/ceramic/recon/{key*}`

Interact with the node using the [Recon protocol](https://cips.ceramic.network/CIPs/cip-124) directly. This allows you to have greater control over the data you consume, but you have to be able to run the Recon algorithm client side.

### `ceramic_poll`
#### Example:

Used by the client to poll for new events. This method uses the Ceramic node's local ordering of events within the given *streamid*The Ceramic node constructs this ordering based on the order in which it received the events. If you want the global ordering of events you can use the `ceramic_recon` method.
```
/ceramic/recon/kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr,kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr,kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr&duration=1000
```

**Params:**
#### Parameters:

- *ceramicUrl* <string> - the Ceramic URL to poll for
- *eventOffset* <integer> - the number of events already consumed
- *duration* <integer> - the amount of time, in seconds, to wait for new data
* _key*_ <Array<string>> - a comma separated list of eventids and ahashes (multicodec encoded), first and last key must be eventid, every even key must be an ahash
* *duration* <integer> - the amount of time, in seconds, to wait for a response

#### Returns:

Content-Type: `application/json`

- _key*_ <array<string>> - a list of eventids and ahashes (multicodec encoded), first and last key must be eventid, every even key must be an ahash

**Returns:**

- *events* <array<object>> - the list of events
- *eventId* <string> - the id of the event
- *id* <string> - CID of the InitEvent for this stream
- *prev* <array<string>> - CIDs of previous hash linked events
- *header* <object> - header for the event
- *data* <object> - the data of the event
- *timestamp* <integer> - the unixtime this event was timestamped (if it has been)
- *eventOffset* <integer> - the number of events consumed

---

### GET `/ceramic/events/{eventid*}`

Retrieve events given an array of eventids.

### `ceramic_recon`
#### Example:

Interact with the Ceramic node using the Recon protocol directly. This allows you to have greater control over the data you consume, but you have to be able to run the Recon algorithm client side.
```
/ceramic/events/kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr,kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr,kjzl6hvfrbw6c82mkud4qs38zl4hd03ifoyg2ksvfjkhuxebfzh3ef89vwvtvrr
```

**Params:**
#### Returns:

- *reconRange* <Array<string>> - a Recon message, *`[eventid or ahash]`*
There are two different content types supported as return value, as specified by the [`Accept`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept) http header.

**Returns:**
**JSON:**

- *reconRange* <Array<string>> - a Recon message, *`[eventid or ahash]`*
Content-Type: `application/json`

---
This content type returns a json representation of the events requested. Note that this is just a representation of the event content, full verifiability is not retained.

- *events* <array<object>> - the list of events
- *eventId* <string> - the id of the event
- *id* <string> - CID of the InitEvent for this stream
- *prev* <array<string>> - CIDs of previous hash linked events
- *header* <object> - header for the event
- *data* <object> - the data of the event
- *timestamp* <integer> - the unixtime this event was timestamped (if it has been)


**CAR file:**

### `ceramic_reconPoll`
Content-Type: `application/vnd.ipld.car`

Same as `ceramic_recon`, but waits for *duration* amount of time in case any new events arrives at the node during this time.
This content types returns the underlaying IPLD data structure of the event streams encoded as a CAR file. This data can be used to trustlessly verify the integrity of the event stream.

**Params:**
The root object of the CAR file is *EventsResult* which is a convenience type containing an array of CIDs corresponding to the eventsids requested. The CAR file also includes all *Event* objects, including all IPLD blocks for these event, but no data from the previous events, e.g. DataEvents include signature envelope, event, and potentially detached payload, while TimeEvents include their entire merkle tree witness.

- *reconRange* <Array<string>> - a Recon message, *`[eventid or ahash]`*
- *duration* <integer> - the amount of time, in seconds, to wait for new data
```verilog
// https://developers.ceramic.network/protocol/streams/event-log/#data-event
type Event InitEvent | DataEvent | TimeEvent
type Events [&Event]
```

**Returns:**

- *reconRange* <Array<string>> - a Recon message, *`[eventid or ahash]`*

---

### POST `/ceramic/events`

Adds a new event to the node.

### `ceramic_exportRawEvents`
#### Request body:

Get the raw IPLD data of a set of events given an array of eventids. This includes all IPLD blocks for this particular event, but no data from the previous events, e.g. DataEvents include signature envelope, event, and potentially detached payload, while TimeEvents include their entire merkle tree witness.
The request body must contain the complete event data.

**Params:**
**CAR:**

- *[eventid]* <array<string>> - the events to fetch
Content-Type: `application/vnd.ipld.car`

**Returns:**
A CAR file where the root is an *Event* as described below.

- *events* <string> - a base64 encoded CAR file containing the event blocks
```verilog
// https://developers.ceramic.network/protocol/streams/event-log/#data-event
type Event InitEvent | DataEvent | TimeEvent
```

---
**JSON:**

Content-Type: `application/json`

A possible convenience method for adding an event by only submitting a JWT.

### `ceramic_putRawEvent`
Only possible once we’ve migrated to use *Varsig* and can create events as plain signed JWTs.

Add an event to a stream.
```javascript
{
event: "<jwt>"
}
```

**Params:**
#### Returns:

- *event* <string> - a base64 encoded CAR file containing the event

**Returns:**

- *success* <boolean> - true if the event was added correctly
**JSON:**

---
- *eventid* <string> - the eventId of this event



### `ceramic_putEvent`
---

Convenience method for adding an event by only submitting a JWT.
### GET `/stats/active-ranges`

Only possible once we’ve migrated to use *Varsig* and can create events as plain signed JWTs.
A status endpoint that returns all ranges that this node is actively monitoring.

**Params:**
#### Returns:

- *event* <string> - a jwt containing the event
* *ranges* <array<object>> - a list of all subscriptions
* *reconRange* <Array<string>> - a Recon message, *`[eventid or ahash]`*
* *offset* <integer> - the total number of eventId bytes observed in this range

**Returns:**

- *success* <boolean> - true if the event was added correctly


## Rationale
<!--The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale may also provide evidence of consensus within the community, and should discuss important objections or concerns raised during discussion.-->

The choice of using json-rpc enables the API to work over multiple different transports. For example, HTTP, web sockets, or event the iframe postMessage api. The latter could be useful in the future if one want's to run a node inside of an iframe.
The choice of using OpenAPI enables ease of integration and content types to more easily send and respond using binary formats such as CAR files. It's also a common pattern to have service workers serve http requests opening up for future browser native implementations of the Ceramic protocol.

There are two main ways of polling for new events. Using recon and a more simple poll. This enables advanced developers to get full control with recon, while a simple poll API based on a single event counter is much easier to use for most developers. The disadvantage of the latter is that events will be received in the order that the node received them, not in the global order of the network.

### Offset using EventId bytes

One question that might arise is why the poll api (e.g. `/ceramic/subscribe/`) couldn't also return events in the global order of the network. The reason for this is that the node might not receive the events in this order from the network, and we want to be able to tell clients about events as they come in. Therefore we store a log of eventId bytes in the order they were observed (and validated) given a particular interest range. This allows clients to have a simple incrementing byte offset counter, while the node doesn't need to keep client specific state.


## Backwards Compatibility
<!--All CIPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The CIP must explain how the author proposes to deal with these incompatibilities. CIP submissions without a sufficient backwards compatibility section may be rejected outright.-->
Expand All @@ -210,7 +259,7 @@ The Ceramic API provides a new way of interacting with event streams. The main b

## Implementation
<!--The implementations must be completed before any CIP is given status "Final", but it need not be completed before the CIP is accepted.-->
No implementation yet. Planned in [rust-ceramic](https://github.com/3box/rust-ceramic/)
A partial implementation exists in [rust-ceramic](https://github.com/3box/rust-ceramic/). It currently implements the following [OpenAPI schema](https://github.com/3box/rust-ceramic/blob/main/api/ceramic.yaml).


## Security Considerations
Expand Down

0 comments on commit 1f266f3

Please sign in to comment.