Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenRPC Support #4711

Closed
wants to merge 55 commits into from
Closed

Conversation

meowsbits
Copy link
Contributor

@meowsbits meowsbits commented Nov 3, 2020

Rel filecoin-project/devgrants#165

This is an implementation adding OpenRPC Document support for the Filecoin API(s).

Note that the docsgen program (easily accessed via make docsgen) also uses reflection to generate Markdown documentation, and that program has been refactored to share some logic with the OpenRPC documentation to avoid code duplication (eg. examples and comments).

Would love to get some feedback on the general architecture (ie Is plugging directly in to the rpc package a best approach?), as well as a few other things, like Is re-using the given examples from the docsgen acceptable?, Is it desirable to add more detailed schemas for common data types (which has been exemplified by a tentative cid.Cid)?

We'd be happy and eager to schedule a meeting to talk through some of these ideas and questions if interest and time allows.


Here's a gist of what the rendered rpc.discover response can look like at the OpenRPC Playground: https://playground.open-rpc.org/?url=https://gist.githubusercontent.com/meowsbits/2f22b61ce46f6897d343f850ac86b540/raw/openrpc_doc.json

image

This implement the basic functionality for the method
over HTTP RPC.

Signed-off-by: meows <b5c6@protonmail.com>
Signed-off-by: meows <b5c6@protonmail.com>

 Conflicts:
	go.mod
	go.sum
Signed-off-by: meows <b5c6@protonmail.com>
Signed-off-by: meows <b5c6@protonmail.com>

 Conflicts:
	go.mod
	go.sum
This is for development only.
Versions need to be bumped when they're ready for use
as canonical remotes.

Signed-off-by: meows <b5c6@protonmail.com>
This eliminates code duplication.

Signed-off-by: meows <b5c6@protonmail.com>
Signed-off-by: meows <b5c6@protonmail.com>
Also fixes casing stuff for the rest of Filecoin.
methods.

Signed-off-by: meows <b5c6@protonmail.com>
This fixes an issue caused with the latest reverting
commit.

Signed-off-by: meows <b5c6@protonmail.com>
…orted stuff

Signed-off-by: meows <b5c6@protonmail.com>

Makefile: fix docgen refactoring for makefile use of command

Signed-off-by: meows <b5c6@protonmail.com>
There are quite of few of these already registered
for the docgen command, so it makes sense to use
those!

Signed-off-by: meows <b5c6@protonmail.com>
Signed-off-by: meows <b5c6@protonmail.com>
…ect versions

Signed-off-by: meows <b5c6@protonmail.com>
This function will handle the manual configurations
for app-specific data types w/r/t their json schema
representation.

This is useful for cases where the reflect library
is unable to provide a sufficient representation
automatically.

Provided in this commit is an initial implementation
for the integerD type (assuming number are represented
in the API as hexs), and a commonly used cid.Cid type.

Signed-off-by: meows <b5c6@protonmail.com>
@whyrusleeping
Copy link
Member

Oooo, this is interesting. At a first glance the integration seems pretty clean, will review further tomorrow

@meowsbits
Copy link
Contributor Author

Hey @whyrusleeping, just bumping on this. Feedback would be very welcome.

Copy link
Contributor

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this idea, but there are a couple of issues

I'd strongly prefer not running any generation at runtime, to not include new / unaudited (by us) dependencies in the lotus/lotus-miner binaries

Instead of hooking into the RPC handler directly, which I don't think is needed, I'd:

  • Generate the doc json blob with docgen (ok if it uses external deps, but please limit changes to deps we already include); store that blob in build/openrpc/[full/miner].json
  • Create go-rice loaders like the one for proof parameters json file (https://github.com/filecoin-project/lotus/blob/master/build/parameters.go); as long as that call is in the build/ package, go-rice should figure out embedding automagically
  • Add a Discover(ctx) (json.RawMessage, error) method to CommonAPI interfaces/structs in api/, api/struct/, node/impl/common
    • Given that OpenRPC spec requires the method to be called rpc.discover, we can handle this in go-jsonrpc by adding a RPCServer.Alias method, and aliasing that method
  • In the implementation just return that pre-generated blob

go.sum Outdated
Comment on lines 274 to 275
github.com/ethereum/go-ethereum v1.9.12 h1:EPtimwsp/KGDSiXcNunzsI4kefdsMHZGJntKx3fvbaI=
github.com/ethereum/go-ethereum v1.9.12/go.mod h1:PvsVkQmhZFx92Y+h2ylythYlheEDt/uBgFbl61Js/jo=
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll need to tame the dep tree before merging this

go-ethereum is LGPL which isn't exactly compatible with MIT+Apache

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(go-ethereum being LGPL is the reason why we've implemented go-jsonrpc in the first place)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmd/lotus/rpc.go Outdated

doc.RegisterReceiverName("Filecoin", fullAPI)

rpcServer.Register("rpc", &RPCDiscovery{doc})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like it will make all RPC methods be rpc.*, which is a breaking change

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this approach with a63eeef

…flect

This removes a problematic dependency
on github.com/ethereum/go-ethereum, which was
imported as a dependency for a couple github.com/etclabscore/go-openrpc-reflect
tests.

etclabscore/go-openrpc-reflect v0.0.36 has removed this
dependency, so this commit is the result of bumping
that version and then running 'go mod tidy'

This is in response to a review at
filecoin-project#4711 (review)

Date: 2020-11-21 06:52:48-06:00
Signed-off-by: meows <b5c6@protonmail.com>
This allows the command to EITHER
generate the doc for Full or Miner APIs.

See comment for usage.

Date: 2020-11-21 07:48:05-06:00
Signed-off-by: meows <b5c6@protonmail.com>
Generating the Miner API OpenRPC doc
(via 'go run ./api/openrpc/cmd miner') caused
the example logic to panic because some types
were missing.

This commit adds those missing types, although
I'm not an expert in the API so I can't
suggest that the example values provided are
ideal or well representative.

Date: 2020-11-21 07:50:21-06:00
Signed-off-by: meows <b5c6@protonmail.com>
…full/miner].json docs

These will be used as static documents
provided by the rpc.discover method.

Date: 2020-11-21 07:51:39-06:00
Signed-off-by: meows <b5c6@protonmail.com>
Date: 2020-11-21 08:23:06-06:00
Signed-off-by: meows <b5c6@protonmail.com>
Instead of generating the doc on the fly,
we're going to serve a static asset.
Rel filecoin-project#4711 (review)
This removes the runtime implementation from the
RPC server construction.

Date: 2020-11-21 08:41:20-06:00
Signed-off-by: meows <b5c6@protonmail.com>
… and structs

Date: 2020-11-21 08:41:56-06:00
Signed-off-by: meows <b5c6@protonmail.com>
This depends on a currently-forked change at
filecoin-project/go-jsonrpc 8350f9463ee451b187d35c492e32f1b999e80210
which establishes this new method RPCServer.AliasMethod.

This solves the problem that the OpenRPC
spec says that the document should be served
at the system extension-prefixed endpoing
rpc.discover (not Filecoin.Discover).

In fact, the document will be available at BOTH
endpoints, but that duplicity is harmless.

Date: 2020-11-21 09:18:26-06:00
Signed-off-by: meows <b5c6@protonmail.com>
…of string

Instead of casting the JSON asset from bytes to string,
unmarshal it to a map[string]interface{} so the
server will provide it as a JSON object.

Date: 2020-11-21 09:27:11-06:00
Signed-off-by: meows <b5c6@protonmail.com>
Conflicts:
      Makefile
      api/api_common.go
      api/apistruct/struct.go
      api/docgen/docgen.go
      go.mod
      go.sum
Date: 2020-11-22 07:19:36-06:00
Signed-off-by: meows <b5c6@protonmail.com>
This keeps the errcheck linter happy.

Date: 2020-11-24 06:33:14-06:00
Signed-off-by: meows <b5c6@protonmail.com>
Date: 2020-11-24 06:36:07-06:00
Signed-off-by: meows <b5c6@protonmail.com>
Tidying up after resolving the merge conflicts
with master at go.mod

Date: 2020-11-24 06:40:45-06:00
Signed-off-by: meows <b5c6@protonmail.com>
This is a repeat of 76e6fd2, since the latest merge
to master seems to have reverted this.

Date: 2020-11-24 06:42:30-06:00
Signed-off-by: meows <b5c6@protonmail.com>
…schema examples

Removing method example pairings since they were
redundant to schema examples and were not
implemented well.

Improved schema examples by using the ExampleValue
method instead of the map lookup.
Made a note in the comment here that this is
not ideal, since we have to make a shortcut assumption
/workaround by using 'unknown' as the method name
and the typea as its own parent.

Luckily these values aren't heavily used by the
method logic.

Date: 2020-11-27 12:57:36-06:00
Signed-off-by: meows <b5c6@protonmail.com>
Previously used an integer schema assuming
hex encoding. It appears, based on review some
of the examples, that this may not be the case.

Obvioussly this schema could be more descriptive,
but just shooting for mostly likely to be
not wrong at this point.

Date: 2020-12-15 14:44:37-06:00
Signed-off-by: meows <b5c6@protonmail.com>
@meowsbits meowsbits marked this pull request as ready for review January 19, 2021 17:51
@meowsbits meowsbits changed the title OpenRPC Support (WIP) OpenRPC Support Jan 19, 2021
Conflicts:
	api/apistruct/struct.go
	api/docgen/docgen.go
	cmd/lotus/rpc.go
	go.mod
	go.sum
	node/impl/storminer.go
Date: 2021-01-19 12:30:42-06:00
Signed-off-by: meows <b5c6@protonmail.com>
…worker.json.gz: run 'make docsgen'

Date: 2021-01-19 12:33:55-06:00
Signed-off-by: meows <b5c6@protonmail.com>
Date: 2021-01-19 12:39:48-06:00
Signed-off-by: meows <b5c6@protonmail.com>
Date: 2021-01-19 12:52:04-06:00
Signed-off-by: meows <b5c6@protonmail.com>
…docsgen'

Date: 2021-01-19 12:55:52-06:00
Signed-off-by: meows <b5c6@protonmail.com>
This will returns empty comments/docs maps.
This should fix issues like:
https://app.circleci.com/pipelines/github/filecoin-project/lotus/12445/workflows/4ebadce9-a298-4ad1-939b-f19ef4c0a5bf/jobs/107218

where the environment makes file lookups hard or
impossible.

Date: 2021-01-19 13:04:58-06:00
Signed-off-by: meows <b5c6@protonmail.com>
@meowsbits
Copy link
Contributor Author

It seems the failing test is not related to this PR... 🤞

@magik6k magik6k self-requested a review January 30, 2021 11:30
@magik6k magik6k self-assigned this Mar 19, 2021
@magik6k magik6k mentioned this pull request Mar 19, 2021
magik6k added a commit that referenced this pull request Mar 19, 2021
* main: init implement rpc.Discover RPC method

This implement the basic functionality for the method
over HTTP RPC.

Signed-off-by: meows <b5c6@protonmail.com>

* main,go.mod,go.sum: init example with go-openrpc-reflect lib

Signed-off-by: meows <b5c6@protonmail.com>

 Conflicts:
	go.mod
	go.sum

* main: make variable name human-friendly

Signed-off-by: meows <b5c6@protonmail.com>

* main,go.mod,go.sum: init impl of go-openrp-reflect printing document

Signed-off-by: meows <b5c6@protonmail.com>

 Conflicts:
	go.mod
	go.sum

* go.mod,go.sum: use go-openrpc-reflect and open-rpc/meta-schema hackforks

This is for development only.
Versions need to be bumped when they're ready for use
as canonical remotes.

Signed-off-by: meows <b5c6@protonmail.com>

* main,openrpc,main: refactor openrpc supporting code to own package

This eliminates code duplication.

Signed-off-by: meows <b5c6@protonmail.com>

* main: add rpc.Discover to openrpc document

Signed-off-by: meows <b5c6@protonmail.com>

* openrpc: fix rpc.discover method name casing

Also fixes casing stuff for the rest of Filecoin.
methods.

Signed-off-by: meows <b5c6@protonmail.com>

* Revert "main: add rpc.Discover to openrpc document"

This reverts commit 116898e.

* main: fix document creation method name

This fixes an issue caused with the latest reverting
commit.

Signed-off-by: meows <b5c6@protonmail.com>

* main,docgen,openrpc: refactor to share api parsing, etc as docgen exported stuff

Signed-off-by: meows <b5c6@protonmail.com>

Makefile: fix docgen refactoring for makefile use of command

Signed-off-by: meows <b5c6@protonmail.com>

* openrpc: add schema.examples to app reflector

There are quite of few of these already registered
for the docgen command, so it makes sense to use
those!

Signed-off-by: meows <b5c6@protonmail.com>

* openrpc: init method pairing examples

Signed-off-by: meows <b5c6@protonmail.com>

* go.mod,go.sum: bump go.mod to use latest meta-schema and openrpc-reflect versions

Signed-off-by: meows <b5c6@protonmail.com>

* openrpc: init SchemaType mapper function

This function will handle the manual configurations
for app-specific data types w/r/t their json schema
representation.

This is useful for cases where the reflect library
is unable to provide a sufficient representation
automatically.

Provided in this commit is an initial implementation
for the integerD type (assuming number are represented
in the API as hexs), and a commonly used cid.Cid type.

Signed-off-by: meows <b5c6@protonmail.com>

* go.mod,go.sum: tame dependencies by bumping etclabscore/go-openrpc-reflect

This removes a problematic dependency
on github.com/ethereum/go-ethereum, which was
imported as a dependency for a couple github.com/etclabscore/go-openrpc-reflect
tests.

etclabscore/go-openrpc-reflect v0.0.36 has removed this
dependency, so this commit is the result of bumping
that version and then running 'go mod tidy'

This is in response to a review at
#4711 (review)

Date: 2020-11-21 06:52:48-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* main: add 'miner' arg to openrpc gen cmd

This allows the command to EITHER
generate the doc for Full or Miner APIs.

See comment for usage.

Date: 2020-11-21 07:48:05-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* docgen: add missing examples for Miner API

Generating the Miner API OpenRPC doc
(via 'go run ./api/openrpc/cmd miner') caused
the example logic to panic because some types
were missing.

This commit adds those missing types, although
I'm not an expert in the API so I can't
suggest that the example values provided are
ideal or well representative.

Date: 2020-11-21 07:50:21-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* build/openrpc/full.json,build/openrpc/miner.json: add build/openrpc/[full/miner].json docs

These will be used as static documents
provided by the rpc.discover method.

Date: 2020-11-21 07:51:39-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* build: init go-rice openrpc static assets

Date: 2020-11-21 08:23:06-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* main: remove rpc.discover implementation from runtime plugin

Instead of generating the doc on the fly,
we're going to serve a static asset.
Rel #4711 (review)
This removes the runtime implementation from the
RPC server construction.

Date: 2020-11-21 08:41:20-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* api,apistruct,common: add Discover(ctx) method to CommonAPI interface and structs

Date: 2020-11-21 08:41:56-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* main: use rpc server method aliasing for rpc.discover

This depends on a currently-forked change at
filecoin-project/go-jsonrpc 8350f9463ee451b187d35c492e32f1b999e80210
which establishes this new method RPCServer.AliasMethod.

This solves the problem that the OpenRPC
spec says that the document should be served
at the system extension-prefixed endpoing
rpc.discover (not Filecoin.Discover).

In fact, the document will be available at BOTH
endpoints, but that duplicity is harmless.

Date: 2020-11-21 09:18:26-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* api,apistruct,build,common: rpc.discover: return json object instead of string

Instead of casting the JSON asset from bytes to string,
unmarshal it to a map[string]interface{} so the
server will provide it as a JSON object.

Date: 2020-11-21 09:27:11-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* Makefile: merge resolve: docsgen command path

Date: 2020-11-22 07:19:36-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* apistruct,main,docgen,openrpc: merge resolve: fix func exporteds, signatures

Date: 2020-11-22 07:31:03-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* go.mod,go.sum: 'get get' auto-bumps version

Date: 2020-11-22 07:31:44-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* Makefile,docgen,main,build/openrpc: refactor openrpc documentation generation

This creates Makefile command docsgen-openrpc-json,
and refactors the docsgen command to generate both
the markdown and openrpc json documents, redirecting
the output of the openrpc json documentation to
the build/openrpc/ directory, where those json
files will be compiled as static assets via go-rice
boxes.

The api/openrpc/cmd now uses usage argumentation
congruent to that of the docgen command (switching
on API context).

Date: 2020-11-22 08:01:18-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* main,docgen_openrpc: rename api/openrpc -> api/docgen-openrpc

Renames the package as well.

This is intended to parallel the
existing docgen package and command
namespacing.

Date: 2020-11-22 10:34:46-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* api,apistruct,docgen,build,build/openrpc: use typed Discover response

Instead of using a map[string]interface{}, use
a typed response for the Discover method implementation.

This avoids having to set a docgen Example for
the generic map[string]interface{} (as an openrpc document)
which both pollutes the generic type and lacks
useful information for the Discover method example.

Date: 2020-11-22 08:31:16-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* apistruct,build,main,impl: implement Discover method for Worker and StorageMiner APIs

Methods return static compiled assets respective
to the APIs.

Date: 2020-11-22 08:57:18-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* docgen_openrpc,build/openrpc: remove timestamping from openrpc doc info

This should allow openrpc docs generated at different
times to be equal. This is important because the CI
(Circle) runs the docgen command and tests that
the output and the source are unchanged (via git diff).

Date: 2020-11-22 10:47:07-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* main,docgen_openrpc,main,build: fix lint issues

Fixes goimports, staticcheck, golint issues.

Date: 2020-11-22 11:06:46-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* docgenopenrpc: fix: don't use an underscore in package name (golint)

Date: 2020-11-22 11:07:53-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* go.sum: fix: mod-tidy-check (run 'go mod tidy')

Date: 2020-11-22 11:09:48-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* go.mod,go.sum: bump filecoin-project/go-jsonrpc dep to latest

This version includes the necessary RPCServer.AliasMethod
method.

Date: 2020-11-23 12:16:15-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* Makefile,main,build,build/openrpc: init gzipped openrpc static docs

Date: 2020-11-24 06:15:06-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* build: refactor gzip reading

Date: 2020-11-24 06:18:34-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* build: add basic test for openrpc doc from static assets

Date: 2020-11-24 06:30:23-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* build: handle reader Close error

This keeps the errcheck linter happy.

Date: 2020-11-24 06:33:14-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* go.sum: run 'go mod tidy'

Date: 2020-11-24 06:36:07-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* go.mod,go.sum: go mod tidy

Tidying up after resolving the merge conflicts
with master at go.mod

Date: 2020-11-24 06:40:45-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* go.mod,go.sum: bump filecoin-project/go-jsonrpc to latest

This is a repeat of 76e6fd2, since the latest merge
to master seems to have reverted this.

Date: 2020-11-24 06:42:30-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* docgenopenrpc,build/openrpc: remove method example pairings, improve schema examples

Removing method example pairings since they were
redundant to schema examples and were not
implemented well.

Improved schema examples by using the ExampleValue
method instead of the map lookup.
Made a note in the comment here that this is
not ideal, since we have to make a shortcut assumption
/workaround by using 'unknown' as the method name
and the typea as its own parent.

Luckily these values aren't heavily used by the
method logic.

Date: 2020-11-27 12:57:36-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* docgenopenrpc: use generic number jsonschema for number types

Previously used an integer schema assuming
hex encoding. It appears, based on review some
of the examples, that this may not be the case.

Obvioussly this schema could be more descriptive,
but just shooting for mostly likely to be
not wrong at this point.

Date: 2020-12-15 14:44:37-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* cmd/lotus,go.mod,go.sum: maybe fix straggling merge resolution conflicts

Date: 2021-01-19 12:30:42-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* build/openrpc/full.json.gz,build/openrpc/miner.json.gz,build/openrpc/worker.json.gz: run 'make docsgen'

Date: 2021-01-19 12:33:55-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* api/apistruct,node/impl: (lint) gofmt

Date: 2021-01-19 12:39:48-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* api/docgen: maybe fix parse error:  open ./api: no such file or directory

Date: 2021-01-19 12:52:04-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* api/docgen,build/openrpc: maybe fix no such file error and run 'make docsgen'

Date: 2021-01-19 12:55:52-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* api/docgen: return if AST comment/groupdoc parsing encounters any error

This will returns empty comments/docs maps.
This should fix issues like:
https://app.circleci.com/pipelines/github/filecoin-project/lotus/12445/workflows/4ebadce9-a298-4ad1-939b-f19ef4c0a5bf/jobs/107218

where the environment makes file lookups hard or
impossible.

Date: 2021-01-19 13:04:58-06:00
Signed-off-by: meows <b5c6@protonmail.com>

* api: Don't depend on build/

* make: support parallel docsgen

* openrpc gen: Use simple build version

* methodgen

* goimports

Co-authored-by: meows <b5c6@protonmail.com>
@magik6k
Copy link
Contributor

magik6k commented Mar 22, 2021

Merged in #5843

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants