Skip to content

Commit

Permalink
feat: provide first-class ssz support on api (#6749)
Browse files Browse the repository at this point in the history
* Add config route definitions

* Add debug route definitions

* Add events route description

* Add lightclient route definitions

* Flatten function params

* Type safety for optional params in write / parse req

* Method args are optional if only optional props

* Fix genesisValidatorsRoot type issue

* Revert requiring all params in write / parse req

* Update http client errors

* Add lodestar route definitions

* Add node route definitions

* Add proof route definitions

* Add builder route definitions

* Add validator route definitions

* Application method response can be void

* Generic options can be passed to application methods

* Default endpoint request type has body property

* Improve types of transform methods

* Export server types from index (to be removed)

* Update config api impl

* Update lightclient api impl

* Update events api impl

* Update lodestar api impl

* Update proof api impl

* Update node api impl

* Update debug api impl

* Update state api impl

* Update pool api impl

* Update blocks api impl

* Partially update validator api impl

* Update beacon routes export

* Align submitPoolBlsToExecutionChange method args

* Filters are always a object

* Update errors messages

* Add beacon client methods

* Add missing routeId label to stream time metric

* Fix json casing in codecs

* Apply remaining changes from #6227

* Produce block apis only have version meta

* Add block values meta to all produce block apis

* Apply changes from #6337

* Handle unsafe version in WithMeta and WithVersion

* Restore server api error

* Update fastify route types

* Update server routes / handlers

* Remove unnecessary type cast

* Restore per route clients

* Fix beacon route types

* Remove option to patch fetch from http client

* Update eventstream client, remove fetch override

Fallback does not work like this, see #6180 for proper solution

* Use StringType for validator status until #6059

* Remove empty fetch.ts file

* Add a few todos

* Update builder client and server routes

* Update beacon exports

* Update api index exports

* Update builder index imports

* Improve type safety of schema definitions

* Add headers to fastify schema

* Fix schema definition type

* Add missing schemas to route definitions

* Fix response codec type

* Remove response codec type casts

* Fix casing in json only codec

* Reuse EmptyResponseCodec

* Update base rest api server

* Update keymanager routes, client and server

* Reuse data types in keymanager impl

* Do not await setting headers, not a promise

* Improve type safety of empty codecs

* Only require to implement supported req methods

* Handle requests that only support one format

* Handle responses that only support one format

* Add json / ssz only req codecs

* Update only support errors

* Fix assertion

* Set correct accept header if only supports one format

* Fix eslint / prettier issues

* More formatting fixes

* Fix fallback request retries in case of http errors

* Formatting of res.error

* Add add retry functionality to http client (from #6387)

* Update rewards routes and server (#6178 and #6260)

* Allow to omit body in ssz req if not defined

* Always set metadata headers in response

* Cache wire format in api response

* Only call raw body for json meta

* Update api package tests (wip)

* Test json and ssz format in generic server tests

* Add a bunch of todos

* Fix a few broken route definitions

* Fix partial config test

* Another todo note

* Stringify body of json requests

* Override default response json method

* Validate external consensus version headers in request

* Add error handling todo

* Skip body schema validation for ssz request bodies

* Clean up generic server tests

* Pass node, proof, events generic tests

* Use enum for media types

* Fix a bunch of route definitions

* Add justified to blockid type

* Properly handle booleans, remove block values codec

* Create Uint8Array test data without allocating Buffer

* Let fastify handle Buffer conversion

* Convert Buffer to Uint8Array in content type parser

* Fix build issues

* Fix fork type in builder routes

* Add some notes

* Properly parse request headers

* Fix incorrect type assumptions in transform

* Generic server tests are passing (except lightclient)

* Correctly handle APIs with empty responses

* Update getHeader return type to reflect no bid responses

* Do not append '?' to URL if query string is empty

* Let server handler set status code for parsing errors

* Remove unused import

* Rename function, request specific

* Completely drop ssz support from getSpec

* Spec tests are passing against latest releases

* Drop unused fastify route config

* Drop ssz request from builder routes, not yet supported

* Remove import

* Apply change from #6695

* Update execution optimistic meta

* Apply changes from #6645

* Add workaround to fix epoch committees type issue

* Add todo to fix inefficient state conversion

* Convert committee to normal array

* Apply changes from #6655

* Align args of validators endpoints

* Convert indices to str in rewards apis

* Update api spec version of README badges

* Revert table formatting changes

* Make this accessible for class-basd API implementations

* Throw err if metadata is accessed for failed response

* Add assertOk to api response

* Tweak api error message

* Update operationIds match spec value

* Add missing version to blob sidecars metadata

* Test headers and ssz bodies against spec

* Minor reordering of code in spec parsing

* submitBlindedBlock throws err if fork is not execution

* responseOk might be undefined

* Remove statusOk from route definition

* Remove stale comment

* Less build errors in beacon-node

* getBlobSidecars return version from server impl

* Update validator produce block impl

* More expressive pool method args

* Application methods might be undefined in mock implementations

* Adress open TODOs in server handler

* Api response methods are synchronous now

* Fix all remaining build issues

* Use more performant from/toHex in server api impls

* Clean up some TODOs

* Fix ApiError type

* Errors related to parsing return a 400 status code

* Simplify method binding

* Forward api context to application methods

* There is no easy way to make generic opts typesafe

* Better separation of server / client code

* Fix comment about missing builder bid

* Remove todo, not worth the change / extra indentation

* Rename route definitions functions

* Return 400 if data passed to keymanager is invalid

* Properly handle response metadata headers

* Fix lint issues

* Add header jsdoc

* Move metadata related code into separate file

* Remove ssz from POST requests without body

* Only set content-type header if body exists

* Fix headers extra

* POST requests without body are handled similar to GET requests

* Fix http client options tests

* Improve validation and type safety of JSON metadata

* Add type guard for request without body

* Differentiate based on body instead of GET vs POST

* More renaming

* Simplify RequestCode type

* Review routes, improve validation

* Remaining local diff

* Fix accept header handling if only support one wire format

* Update 406 error to more closely match spec example

* Enforce version header via custom check instead of schema

* Use ssz as default request wire format

* Log failure to receive head event to verbose

* Do not set default value for context

* Update getClient return type to better align with method name

* Consistent pattern to get route definitions

* Dedupe api client type for builder and keymanager

* Fix fallback logic if server returns http error

* Update head event error logging

* Retry 415 errors with JSON and cache SSZ not supported

* Use fetch spy to assert call times

* Update comment

* Update getLightClientUpdatesByRange endpoint meta

* Do not forward ssz bytes of blinded block to publishBlock

* Fix lightclient e2e tests

* Version header in publishBlock api is optional

* Reduce type duplication

* Add option to override request init in route definition

* Add JsonOnlyResp codec

* Validate boolean str value from headers

* Document default wire formats

* Simplify merging of inits in http client

* Remove type hacks from fetchBeaconHealth

* Reduce call stack in http client

* Add .ssz() equivalent method for json to api response

* More http client tests

* Ensure topics query is provided to eventstream api

* Validate request content type in handler

Fastify does not cover all edge cases

* Review routes, fix param docs, no empty comments

* Fix typo

* Add note about builder spec not supporting ssz

* Consistently move keymanager jsdoc to routes

* Sanitize user provided init values before merging

* Remove unused ssz only codec

* Allow passing wire formats as string literals

* chore: review proof routes (#6843)

Review proof routes

* chore: review lightclient routes (#6842)

Review lightclient routes

* chore: review node routes (#6844)

Review node routes

* feat: add cli flags to configure http wire format (#6840)

* Review PR, mostly cosmetic changes

* Fix event stream error handling

---------

Co-authored-by: Cayman <caymannava@gmail.com>
  • Loading branch information
nflaig and wemeetagain authored Jun 10, 2024
1 parent 14855ea commit f8593a9
Show file tree
Hide file tree
Showing 226 changed files with 8,009 additions and 6,727 deletions.
2 changes: 1 addition & 1 deletion packages/api/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Lodestar Eth Consensus API

[![Discord](https://img.shields.io/discord/593655374469660673.svg?label=Discord&logo=discord)](https://discord.gg/aMxzVcr)
[![ETH Beacon APIs Spec v2.1.0](https://img.shields.io/badge/ETH%20beacon--APIs-2.1.0-blue)](https://github.com/ethereum/beacon-APIs/releases/tag/v2.1.0)
[![ETH Beacon APIs Spec v2.5.0](https://img.shields.io/badge/ETH%20beacon--APIs-2.5.0-blue)](https://github.com/ethereum/beacon-APIs/releases/tag/v2.5.0)
![ES Version](https://img.shields.io/badge/ES-2021-yellow)
![Node Version](https://img.shields.io/badge/node-22.x-green)

Expand Down
3 changes: 3 additions & 0 deletions packages/api/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@
".": {
"import": "./lib/index.js"
},
"./server": {
"import": "./lib/server/index.js"
},
"./beacon": {
"import": "./lib/beacon/index.js"
},
Expand Down
46 changes: 6 additions & 40 deletions packages/api/src/beacon/client/beacon.ts
Original file line number Diff line number Diff line change
@@ -1,46 +1,12 @@
import {ChainForkConfig} from "@lodestar/config";
import {Api, ReqTypes, routesData, getReqSerializers, getReturnTypes, BlockId} from "../routes/beacon/index.js";
import {IHttpClient, generateGenericJsonClient, getFetchOptsSerializers} from "../../utils/client/index.js";
import {ResponseFormat} from "../../interfaces.js";
import {BlockResponse, BlockV2Response} from "../routes/beacon/block.js";
import {ApiClientMethods, IHttpClient, createApiClientMethods} from "../../utils/client/index.js";
import {Endpoints, getDefinitions} from "../routes/beacon/index.js";

export type ApiClient = ApiClientMethods<Endpoints>;

/**
* REST HTTP client for beacon routes
*/
export function getClient(config: ChainForkConfig, httpClient: IHttpClient): Api {
const reqSerializers = getReqSerializers(config);
const returnTypes = getReturnTypes();
// Some routes return JSON, use a client auto-generator
const client = generateGenericJsonClient<Api, ReqTypes>(routesData, reqSerializers, returnTypes, httpClient) as Api;
const fetchOptsSerializer = getFetchOptsSerializers<Api, ReqTypes>(routesData, reqSerializers);

return {
...client,
async getBlock<T extends ResponseFormat = "json">(blockId: BlockId, format?: T) {
if (format === "ssz") {
const res = await httpClient.arrayBuffer({
...fetchOptsSerializer.getBlock(blockId, format),
});
return {
ok: true,
response: new Uint8Array(res.body),
status: res.status,
} as BlockResponse<T>;
}
return client.getBlock(blockId, format);
},
async getBlockV2<T extends ResponseFormat = "json">(blockId: BlockId, format?: T) {
if (format === "ssz") {
const res = await httpClient.arrayBuffer({
...fetchOptsSerializer.getBlockV2(blockId, format),
});
return {
ok: true,
response: new Uint8Array(res.body),
status: res.status,
} as BlockV2Response<T>;
}
return client.getBlockV2(blockId, format);
},
};
export function getClient(config: ChainForkConfig, httpClient: IHttpClient): ApiClient {
return createApiClientMethods(getDefinitions(config), httpClient);
}
13 changes: 6 additions & 7 deletions packages/api/src/beacon/client/config.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
import {ChainForkConfig} from "@lodestar/config";
import {generateGenericJsonClient, IHttpClient} from "../../utils/client/index.js";
import {Api, getReqSerializers, getReturnTypes, ReqTypes, routesData} from "../routes/config.js";
import {ApiClientMethods, IHttpClient, createApiClientMethods} from "../../utils/client/index.js";
import {Endpoints, getDefinitions} from "../routes/config.js";

export type ApiClient = ApiClientMethods<Endpoints>;

/**
* REST HTTP client for config routes
*/
export function getClient(config: ChainForkConfig, httpClient: IHttpClient): Api {
const reqSerializers = getReqSerializers();
const returnTypes = getReturnTypes();
// All routes return JSON, use a client auto-generator
return generateGenericJsonClient<Api, ReqTypes>(routesData, reqSerializers, returnTypes, httpClient);
export function getClient(config: ChainForkConfig, httpClient: IHttpClient): ApiClient {
return createApiClientMethods(getDefinitions(config), httpClient);
}
58 changes: 5 additions & 53 deletions packages/api/src/beacon/client/debug.ts
Original file line number Diff line number Diff line change
@@ -1,60 +1,12 @@
import {ChainForkConfig} from "@lodestar/config";
import {ApiClientResponse, ResponseFormat} from "../../interfaces.js";
import {HttpStatusCode} from "../../utils/client/httpStatusCode.js";
import {generateGenericJsonClient, getFetchOptsSerializers, IHttpClient} from "../../utils/client/index.js";
import {StateId} from "../routes/beacon/state.js";
import {Api, getReqSerializers, getReturnTypes, ReqTypes, routesData} from "../routes/debug.js";
import {ApiClientMethods, createApiClientMethods, IHttpClient} from "../../utils/client/index.js";
import {Endpoints, getDefinitions} from "../routes/debug.js";

// As Jul 2022, it takes up to 3 mins to download states so make this 5 mins for reservation
const GET_STATE_TIMEOUT_MS = 5 * 60 * 1000;
export type ApiClient = ApiClientMethods<Endpoints>;

/**
* REST HTTP client for debug routes
*/
export function getClient(_config: ChainForkConfig, httpClient: IHttpClient): Api {
const reqSerializers = getReqSerializers();
const returnTypes = getReturnTypes();
// Some routes return JSON, use a client auto-generator
const client = generateGenericJsonClient<Api, ReqTypes>(routesData, reqSerializers, returnTypes, httpClient);
// For `getState()` generate request serializer
const fetchOptsSerializers = getFetchOptsSerializers<Api, ReqTypes>(routesData, reqSerializers);

return {
...client,

// TODO: Debug the type issue
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-expect-error
async getState(stateId: string, format?: ResponseFormat) {
if (format === "ssz") {
const res = await httpClient.arrayBuffer({
...fetchOptsSerializers.getState(stateId, format),
timeoutMs: GET_STATE_TIMEOUT_MS,
});
return {
ok: true,
response: new Uint8Array(res.body),
status: res.status,
} as ApiClientResponse<{[HttpStatusCode.OK]: Uint8Array}>;
}
return client.getState(stateId, format);
},

// TODO: Debug the type issue
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-expect-error
async getStateV2(stateId: StateId, format?: ResponseFormat) {
if (format === "ssz") {
const res = await httpClient.arrayBuffer({
...fetchOptsSerializers.getStateV2(stateId, format),
timeoutMs: GET_STATE_TIMEOUT_MS,
});
return {ok: true, response: new Uint8Array(res.body), status: res.status} as ApiClientResponse<{
[HttpStatusCode.OK]: Uint8Array;
}>;
}

return client.getStateV2(stateId, format);
},
};
export function getClient(config: ChainForkConfig, httpClient: IHttpClient): ApiClient {
return createApiClientMethods(getDefinitions(config), httpClient);
}
80 changes: 42 additions & 38 deletions packages/api/src/beacon/client/events.ts
Original file line number Diff line number Diff line change
@@ -1,59 +1,63 @@
import {Api, BeaconEvent, routesData, getEventSerdes} from "../routes/events.js";
import {stringifyQuery, urlJoin} from "../../utils/client/format.js";
import {ChainForkConfig} from "@lodestar/config";
import {getEventSource} from "../../utils/client/eventSource.js";
import {HttpStatusCode} from "../../utils/client/httpStatusCode.js";
import {stringifyQuery, urlJoin} from "../../utils/client/format.js";
import {ApiClientMethods} from "../../utils/client/method.js";
import {RouteDefinitionExtra} from "../../utils/client/request.js";
import {ApiResponse} from "../../utils/client/response.js";
import {BeaconEvent, Endpoints, getDefinitions, getEventSerdes} from "../routes/events.js";

export type ApiClient = ApiClientMethods<Endpoints>;

/**
* REST HTTP client for events routes
*/
export function getClient(baseUrl: string): Api {
export function getClient(config: ChainForkConfig, baseUrl: string): ApiClient {
const definitions = getDefinitions(config);
const eventSerdes = getEventSerdes();

return {
eventstream: async (topics, signal, onEvent) => {
eventstream: async ({topics, signal, onEvent, onError, onClose}) => {
const query = stringifyQuery({topics});
const url = `${urlJoin(baseUrl, routesData.eventstream.url)}?${query}`;
const url = `${urlJoin(baseUrl, definitions.eventstream.url)}?${query}`;
// eslint-disable-next-line @typescript-eslint/naming-convention
const EventSource = await getEventSource();
const eventSource = new EventSource(url);

try {
await new Promise<void>((resolve, reject) => {
for (const topic of topics) {
eventSource.addEventListener(topic, ((event: MessageEvent) => {
const message = eventSerdes.fromJson(topic, JSON.parse(event.data));
onEvent({type: topic, message} as BeaconEvent);
}) as EventListener);
}

// EventSource will try to reconnect always on all errors
// `eventSource.onerror` events are informative but don't indicate the EventSource closed
// The only way to abort the connection from the client is via eventSource.close()
eventSource.onerror = function onerror(err) {
const errEs = err as unknown as EventSourceError;
// Consider 400 and 500 status errors unrecoverable, close the eventsource
if (errEs.status === 400) {
reject(Error(`400 Invalid topics: ${errEs.message}`));
}
if (errEs.status === 500) {
reject(Error(`500 Internal Server Error: ${errEs.message}`));
}

// TODO: else log the error somewhere
// console.log("eventstream client error", errEs);
};

// And abort resolve the promise so finally {} eventSource.close()
signal.addEventListener("abort", () => resolve(), {once: true});
});
} finally {
const close = (): void => {
eventSource.close();
onClose?.();
signal.removeEventListener("abort", close);
};
signal.addEventListener("abort", close, {once: true});

for (const topic of topics) {
eventSource.addEventListener(topic, (event: MessageEvent) => {
const message = eventSerdes.fromJson(topic, JSON.parse(event.data));
onEvent({type: topic, message} as BeaconEvent);
});
}

return {ok: true, response: undefined, status: HttpStatusCode.OK};
// EventSource will try to reconnect always on all errors
// `eventSource.onerror` events are informative but don't indicate the EventSource closed
// The only way to abort the connection from the client is via eventSource.close()
eventSource.onerror = function onerror(err) {
const errEs = err as unknown as EventSourceError;

// Ignore noisy errors due to beacon node being offline
if (!errEs.message?.includes("ECONNREFUSED")) {
onError?.(new Error(errEs.message));
}

// Consider 400 and 500 status errors unrecoverable, close the eventsource
if (errEs.status === 400 || errEs.status === 500) {
close();
}
};

return new ApiResponse(definitions.eventstream as RouteDefinitionExtra<Endpoints["eventstream"]>);
},
};
}

// https://github.com/EventSource/eventsource/blob/82e034389bd2c08d532c63172b8e858c5b185338/lib/eventsource.js#L143
type EventSourceError = {status?: number; message: string};
type EventSourceError = {status?: number; message?: string};
16 changes: 12 additions & 4 deletions packages/api/src/beacon/client/index.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
import {ChainForkConfig} from "@lodestar/config";
import {Api} from "../routes/index.js";
import {IHttpClient, HttpClient, HttpClientOptions, HttpClientModules} from "../../utils/client/index.js";
import {
ApiClientMethods,
HttpClient,
HttpClientModules,
HttpClientOptions,
IHttpClient,
} from "../../utils/client/index.js";
import {Endpoints} from "../routes/index.js";

import * as beacon from "./beacon.js";
import * as configApi from "./config.js";
Expand All @@ -17,18 +23,20 @@ type ClientModules = HttpClientModules & {
httpClient?: IHttpClient;
};

export type ApiClient = {[K in keyof Endpoints]: ApiClientMethods<Endpoints[K]>};

/**
* REST HTTP client for all routes
*/
export function getClient(opts: HttpClientOptions, modules: ClientModules): Api {
export function getClient(opts: HttpClientOptions, modules: ClientModules): ApiClient {
const {config} = modules;
const httpClient = modules.httpClient ?? new HttpClient(opts, modules);

return {
beacon: beacon.getClient(config, httpClient),
config: configApi.getClient(config, httpClient),
debug: debug.getClient(config, httpClient),
events: events.getClient(httpClient.baseUrl),
events: events.getClient(config, httpClient.baseUrl),
lightclient: lightclient.getClient(config, httpClient),
lodestar: lodestar.getClient(config, httpClient),
node: node.getClient(config, httpClient),
Expand Down
13 changes: 6 additions & 7 deletions packages/api/src/beacon/client/lightclient.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
import {ChainForkConfig} from "@lodestar/config";
import {Api, ReqTypes, routesData, getReqSerializers, getReturnTypes} from "../routes/lightclient.js";
import {IHttpClient, generateGenericJsonClient} from "../../utils/client/index.js";
import {ApiClientMethods, IHttpClient, createApiClientMethods} from "../../utils/client/index.js";
import {Endpoints, getDefinitions} from "../routes/lightclient.js";

export type ApiClient = ApiClientMethods<Endpoints>;

/**
* REST HTTP client for lightclient routes
*/
export function getClient(_config: ChainForkConfig, httpClient: IHttpClient): Api {
const reqSerializers = getReqSerializers();
const returnTypes = getReturnTypes();
// All routes return JSON, use a client auto-generator
return generateGenericJsonClient<Api, ReqTypes>(routesData, reqSerializers, returnTypes, httpClient);
export function getClient(config: ChainForkConfig, httpClient: IHttpClient): ApiClient {
return createApiClientMethods(getDefinitions(config), httpClient);
}
13 changes: 6 additions & 7 deletions packages/api/src/beacon/client/lodestar.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
import {ChainForkConfig} from "@lodestar/config";
import {Api, ReqTypes, routesData, getReqSerializers, getReturnTypes} from "../routes/lodestar.js";
import {IHttpClient, generateGenericJsonClient} from "../../utils/client/index.js";
import {ApiClientMethods, IHttpClient, createApiClientMethods} from "../../utils/client/index.js";
import {Endpoints, getDefinitions} from "../routes/lodestar.js";

export type ApiClient = ApiClientMethods<Endpoints>;

/**
* REST HTTP client for lodestar routes
*/
export function getClient(_config: ChainForkConfig, httpClient: IHttpClient): Api {
const reqSerializers = getReqSerializers();
const returnTypes = getReturnTypes();
// All routes return JSON, use a client auto-generator
return generateGenericJsonClient<Api, ReqTypes>(routesData, reqSerializers, returnTypes, httpClient);
export function getClient(config: ChainForkConfig, httpClient: IHttpClient): ApiClient {
return createApiClientMethods(getDefinitions(config), httpClient);
}
13 changes: 6 additions & 7 deletions packages/api/src/beacon/client/node.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
import {ChainForkConfig} from "@lodestar/config";
import {generateGenericJsonClient, IHttpClient} from "../../utils/client/index.js";
import {Api, getReqSerializers, getReturnTypes, ReqTypes, routesData} from "../routes/node.js";
import {ApiClientMethods, IHttpClient, createApiClientMethods} from "../../utils/client/index.js";
import {Endpoints, getDefinitions} from "../routes/node.js";

export type ApiClient = ApiClientMethods<Endpoints>;

/**
* REST HTTP client for beacon routes
*/
export function getClient(_config: ChainForkConfig, httpClient: IHttpClient): Api {
const reqSerializers = getReqSerializers();
const returnTypes = getReturnTypes();
// All routes return JSON, use a client auto-generator
return generateGenericJsonClient<Api, ReqTypes>(routesData, reqSerializers, returnTypes, httpClient);
export function getClient(config: ChainForkConfig, httpClient: IHttpClient): ApiClient {
return createApiClientMethods(getDefinitions(config), httpClient);
}
Loading

0 comments on commit f8593a9

Please sign in to comment.