fix: add support for a ws client & batch processing over ws

[zivkovicmilos]

Description

Let me start this PR description by explaining what I wanted to accomplish, so you are not discouraged when reading the file changes.

I wanted to:

• create a WS client from outside the repository (just like we can create http clients)
• have the WS client support batch requests
• have the TM2 JSON-RPC server support batch requests / responses over WS
• not have to rewrite core client / server logic and APIs

It might seem odd, reading the 3rd point and thinking this is not supported. The truth is actually much more troubling.

Our JSON-RPC server (and client!) implementations are not great. HTTP requests are handled and parsed in a completely different flow than WS requests, even though the output of both should be identical (just over different mediums). Lots of logic is coupled, making it hard to extract and simplify. I've broken the WS / HTTP implementation multiple times over the course of this PR, and all of the tests were passing, even though I've critically broken the module at the time.
The client code for the JSON-RPC server requires a response type (of course, for Amino result parsing) to be given at the moment of calling, which is not amazing considering our WS implementation is async, and doesn't have these response types in the result parsing context (these logic flows are handled by different threads).

What I ended up doing:

• added support for a WS client
  • this was a bigger effort than expected; I extracted and simplified the batching logic, but was still blocked by the lack of batch support in WS request handling
• added batch support for the TM2 JSON-RPC server
  • I basically mirrored the HTTP batch request handling (hint that these should be identical flows)
• BREAKING: completely overhauled our JSON-RPC client libraries and actual instances (http / ws), for a couple of reasons:
  • I tried to add support for all previously mentioned items, but it was impossible with the architecture that was in place (baseRPCClient). The slightly tweaked API (for creating HTTP / WS clients, and using batches) is much simpler to use, and actually has error handling.
  • We didn't have nearly enough coverage and good tests for the functionality -- now we have a suite of E2E and unit tests that give confidence.

I will start an effort in the near future for refactoring the JSON-RPC server code from the ground up in a subsequent PR, this time with specs and tests at the forefront.

How to test out the websockets

To test out the WS responses, you can use a tool like websocat.

1. start a local chain
2. run websocat ws://127.0.0.1:26657/websocket (this is the default URL)
3. send a batch request:

[ { "id": 1, "jsonrpc": "2.0", "method": "status", "params": [] }, { "id": 2, "jsonrpc": "2.0", "method": "status", "params": [] } ]

How to test out the updated client code

I created the following snippet for easily testing the functionality updated in this PR:

• single HTTP / WS requests
• batch HTTP / WS requests

package main

import (
	"context"
	"fmt"

	"github.com/gnolang/gno/tm2/pkg/bft/rpc/client"
	ctypes "github.com/gnolang/gno/tm2/pkg/bft/rpc/core/types"
)

func main() {
	// HTTP //
	fmt.Printf("\n--- HTTP CLIENT ---\n")

	// Open HTTP connection
	httpClient, err := client.NewHTTPClient("http://127.0.0.1:26657")
	if err != nil {
		panic("unable to start HTTP client")
	}

	// Get a single status
	status, err := httpClient.Status()
	if err != nil {
		fmt.Println("Unable to send single status (HTTP)!")

		panic(err)
	}

	fmt.Printf("\n\nHTTP - Single status: %v\n\n", status)

	// Get a batch of statuses
	httpBatch := httpClient.NewBatch()

	// Add 10 status requests
	for i := 0; i < 10; i++ {
		if err := httpBatch.Status(); err != nil {
			fmt.Println("Unable to add status request to HTTP batch!")

			panic(err)
		}
	}

	// Send the batch
	results, err := httpBatch.Send(context.Background())
	if err != nil {
		fmt.Println("Unable to send HTTP batch!")

		panic(err)
	}

	for index, resultRaw := range results {
		result, ok := resultRaw.(*ctypes.ResultStatus)
		if !ok {
			panic("Invalid status type in batch response!")
		}

		fmt.Printf("\nStatus %d from batch: %v\n", index, result)
	}

	// WS //
	fmt.Printf("\n--- WS CLIENT ---\n")

	// Open WS connection
	wsClient, err := client.NewWSClient("ws://127.0.0.1:26657/websocket")
	if err != nil {
		panic("unable to start WS client")
	}

	// Get a single status
	status, err = wsClient.Status()
	if err != nil {
		fmt.Println("Unable to send single status (WS)!")

		panic(err)
	}

	fmt.Printf("\n\nWS - Single status: %v\n\n", status)

	// Get a batch of statuses
	wsBatch := wsClient.NewBatch()

	// Add 10 status requests
	for i := 0; i < 10; i++ {
		if err := wsBatch.Status(); err != nil {
			fmt.Println("Unable to add status request to WS batch!")

			panic(err)
		}
	}

	// Send the batch
	results, err = wsBatch.Send(context.Background())
	if err != nil {
		fmt.Println("Unable to send WS batch!")

		panic(err)
	}

	for index, resultRaw := range results {
		result, ok := resultRaw.(*ctypes.ResultStatus)
		if !ok {
			panic("Invalid status type in batch response!")
		}

		fmt.Printf("\nStatus %d from batch: %v\n", index, result)
	}

	if err := wsClient.Close(); err != nil {
		fmt.Println("Unable to gracefully close WS client!")

		panic(err)
	}

	fmt.Printf("\n\nGreat success!\n\n")
}

cc @dongwon8247

Contributors' checklist...

• Added new tests, or not needed, or not feasible
• Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory
• Updated the official documentation or not needed
• No breaking changes were made, or a BREAKING CHANGE: xxx message was included in the description
• Added references to related issues and PRs
• Provided any useful hints for running manual tests
• Added new benchmarks to generated graphs, if any. More info here.

[codecov]

Codecov Report

Attention: Patch coverage is 33.33333% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 48.36%. Comparing base (fdde3d0) to head (133376e).

Files	Patch %	Lines
gno.land/pkg/gnoweb/gnoweb.go	33.33%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1498      +/-   ##
==========================================
+ Coverage   46.41%   48.36%   +1.95%     
==========================================
  Files         487      409      -78     
  Lines       68973    62035    -6938     
==========================================
- Hits        32011    30006    -2005     
+ Misses      34313    29531    -4782     
+ Partials     2649     2498     -151     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[thehowl]

This is amazing. Thank you Milos. 🎉

Would you like to have a preliminary review, or just have the team try to fool around and break ws batch processing?

[zivkovicmilos]

@gfanton @ajnavarro
What should we do about gnofaucet being removed from the Dockerfile?

[thehowl]

From the review meeting:

• Having gnofaucet in contribs is a good idea, and a faucet implementation should stay in the monorepo (as it is one way to get started)
• @zivkovicmilos to split out this PR into another one which does the split specifically.

[zivkovicmilos]

From the review meeting:

• Having gnofaucet in contribs is a good idea, and a faucet implementation should stay in the monorepo (as it is one way to get started)
• @zivkovicmilos to split out this PR into another one which does the split specifically.

I've merged in master changes from #1955 that fix gnofaucet

[gfanton]

Looking good! 👍

Just a few suggestions, nothing critical

[ajnavarro]

After reviewing @gfanton comments, LGTM

[leohhhn]

Hey @zivkovicmilos, it seems that your example for testing out the WS batch request is not working correctly:

❯ websocat ws://127.0.0.1:26657/websocket
[ { "id": 1, "jsonrpc": "2.0", "method": "status", "params": [] }, { "id": 2, "jsonrpc": "2.0", "method": "status", "params": [] } ]
{   "jsonrpc": "2.0",   "id": "",   "error": {     "code": -32700,     "message": "Parse error. Invalid JSON",     "data": "json: cannot unmarshal array into Go value of type struct { JSONRPC string \"json:\\\"jsonrpc\\\"\"; ID interface {} \"json:\\\"id\\\"\"; Method string \"json:\\\"method\\\"\"; Params json.RawMessage \"json:\\\"params\\\"\" }"   } }

Not sure if this is something introduced by a recent change.

[zivkovicmilos]

Hey @zivkovicmilos, it seems that your example for testing out the WS batch request is not working correctly:

❯ websocat ws://127.0.0.1:26657/websocket
[ { "id": 1, "jsonrpc": "2.0", "method": "status", "params": [] }, { "id": 2, "jsonrpc": "2.0", "method": "status", "params": [] } ]
{   "jsonrpc": "2.0",   "id": "",   "error": {     "code": -32700,     "message": "Parse error. Invalid JSON",     "data": "json: cannot unmarshal array into Go value of type struct { JSONRPC string \"json:\\\"jsonrpc\\\"\"; ID interface {} \"json:\\\"id\\\"\"; Method string \"json:\\\"method\\\"\"; Params json.RawMessage \"json:\\\"params\\\"\" }"   } }

Not sure if this is something introduced by a recent change.

Are you sure you've built the gnoland binary? This is the error you see with the master codebase now
It's working fine on my end, on the latest branch commit 👀