RLPS: Blockchain Data API

[ryandotsmith]

Announcing v0.0.1 of RLPS

              Read Block Data                           Get Block N
          ┌───Use header's ──┐                ┌───────────────────────────────────┐
          │   bloom filter   │                │         Filter: 0xa...            │
          ▼                  │                ▼                                   │
┌──────────────────┬──────────────────┬──────────────┐                  ┌──────────────────┐
│                  │                  │   edge cdn   │                  │                  │
│                  │                  │              │                  │                  │
│                  │                  ├──────────────┤                  │                  │
│     Eth Node     │       RLPS       │   edge cdn   │                  │      Client      │
│                  │                  │              │                  │    (eg. E2PG)    │
│                  │                  ├──────────────┤                  │                  │
│                  │                  │   edge cdn   │                  │                  │
│                  │                  │              │                  │                  │
└──────────────────┴──────────────────┴──────────────┘                  └──────────────────┘
                                              │                                   ▲
                                              │                                   │
                                              └─RLP([header, bodies, receipts]...)┘

An RLP Server (RLPS) is an HTTPS server that accepts requests which include a block(s) number and returns RLP encoded: header, bodies, and receipts. It also accepts a ‘filter’ query which can contain hashed event signatures so that you can efficiently download the data you are wanting to index.

RLPS is more efficient than a JSON RPC API because:

• JSON has extra data and can be expensive to encode/decode
• A block header’s bloom filter can reduce the amount of data to be processed and transferred
• Large batches in combination with HTTP caching creates highly scalable, low latency access

You can use RLPS to preform a backfill index operation or simple follow the chain. Index Supply’s RLPS API can be found here: https://rlps.indexsupply.net/

I developed RLPS because I needed a simple mechanism for running E2PG without a full node. RLPS is actively being developed and so you can expect performance, features, and stability to dramatically improve over the coming months.

[roninjin10]

This seems like a lot of work for a minor optimization from my naive viewpoint. Would be curious to know more about why this is important.

[ryandotsmith]

Great question! To be perfectly honest, I am still not 100% convinced this is a good idea. I'm still exploring. However, here are the points that motivated me to take this direction:

1. Single, batched API that returns: headers, bodies, and receipts. This doesn't exist in the JSON RPC specs nor do clients offer this.
2. It turns out that JSON decoding really takes a toll on performance. When backfilling 17,500,000 blocks (~400GB) every representation translation adds latency and so the fewer the better. Going from RLP -> ABI -> uint64,address,... is faster than JSON -> hex string -> bytes -> ABI -> uint64,address,...

But more generally, I wanted to build this API because I think it is more elegant for backfilling than what currently exists. I want to use nice things and I don't think the current APIs for backfilling indexes are very nice.

OTOH I would love to be convinced that this is a bad idea. What's an argument against it?

[ryandotsmith]

Keeping an eye on Cloudflare's ETH APIs:

https://developers.cloudflare.com/web3/ethereum-gateway/reference/supported-api-methods/

[ryandotsmith]

Uploading a few charts showing RLPS usage over the last 6 months:

Optimism

Optimism Goerli

Base

Base Goerli