fix: concurrency issues between simulation, grpc queries and ABCI commit flow #213

p0mvn · 2022-04-29T17:34:47Z

Description

Fixes the problem in: osmosis-labs/osmosis#1356

Previously, queries were always directed to ABCI, getting eventually synchronized with the commit flow. In #137 , we allowed GRPC queries to be concurrent with the ABCI commit flow. However, integration tests were not configured correctly to test that update. Specifically, the GRPC client was never set up in the integration suite. As a result, the flow with concurrent GRPC queries was never tested.

Upon setting up the GRPC client correctly in testutil/network/util.go, 2 problems were exposed:

A number of data races across multiple GRPC queries and simulation unit tests
In some cases, the requested height was never propagated correctly by the GRPC call in the client. As a result, the GRPC server never read the requested height and always assumed the latest. According to the current design, height may come from grpc context or client context. Currently, if the caller provided height through grpc context, that would work. However, it wouldn't if they attempted to supply it via the client context, similar to how integration tests currently do. As a result, I added a function to choose the right height (the priority is described in the spec) and correctly pass it to either GRPC or ABCI query.

Also, I realized that a number of queries must never be concurrent with ABCI. Specifically, tendermint and simulation. Therefore, those are now always directed to the ABCI flow to avoid data races. This change should directly address the reported problem in osmosis-labs/osmosis#1356

I proceeded by writing table-driven tests for TestGRPCQuery in client and discovered that if a user request a height that does not exist, an error is returned. This is stemming from the fact that IAVL tree never returns an error from GetImmutable(< height >) when a non-existent height is requested. Instead it returns an empty tree with nil. This is contrary to what is stated in the following comment. This problem should be investigated further in a separate issue and potentially fixed. For now, I added an additional safeguard against non-existent heights here

Task Log

enable GRPC client in integration tests
fix simulation and Tendermint GRPC queries by directing it to the ABCI flow
correctly pass height metadata to fix a number of other GRPC query tests
add unit tests for the new selectHeight method
improve TestGRPCQuery integration tests with table-driven tests
fix querying non-existent heights.

client/grpc_query_test.go

testutil/network/network.go

alexanderbez · 2022-05-02T13:44:44Z

baseapp/abci.go

@@ -603,15 +603,19 @@ func (app *BaseApp) createQueryContext(height int64, prove bool) (sdk.Context, e
 			)
 	}

-	cacheMS, err := app.cms.CacheMultiStoreWithVersion(height)
-	if err != nil {
+	if height > app.LastBlockHeight() {


We should store app.LastBlockHeight() in a variable since we use it twice :)

baseapp/abci.go

alexanderbez · 2022-05-02T13:46:13Z

client/grpc_query.go

+
+// selectHeight returns the height chosen from client context and grpc context.
+// If exists, height extracted from grpcCtx takes precedence.
+func selectHeight(clientContext Context, grpcCtx gocontext.Context) (height int64, err error) {


can we avoid named return vars?

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

…mit flow (#213) * fix: simulation and grpc query concurrency with ABCI * table tests for selectHeight * add table-driven GRPC query tests and fix non-existent heights * clean up * clean up grpc query tests * Apply suggestions from code review * Update testutil/network/network.go * fix typo in abci.go * Update baseapp/abci.go Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com> * store last block height in a var * avoid returning named parameters Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com> (cherry picked from commit 18b0d2d)

…mit flow (#213) (#216)

ValarDragon · 2022-05-03T03:38:26Z

client/grpc_query.go

+// selectHeight returns the height chosen from client context and grpc context.
+// If exists, height extracted from grpcCtx takes precedence.
+func selectHeight(clientContext Context, grpcCtx gocontext.Context) (int64, error) {


Are we choosing 0 when neither is set? If so maybe lets set this more explicitly?

Should we be having input validation for heights > latest block height?

ValarDragon · 2022-05-03T03:38:26Z

client/grpc_query.go

+// selectHeight returns the height chosen from client context and grpc context.
+// If exists, height extracted from grpcCtx takes precedence.
+func selectHeight(clientContext Context, grpcCtx gocontext.Context) (int64, error) {


Are we choosing 0 when neither is set? If so maybe lets set this more explicitly?

Should we be having input validation for heights > latest block height?

Zero is guaranteed to be returned in that case by var height int64 which is being initialized to 0. There is a test to cover this case as well. How are you suggesting making this more explicit? With height := int64(0) or adding a relevant comment to the spec? Just making sure I understand what you mean

The validation for heights > latest block height is done at the server layer. It would be challenging to have that information here in client.

No need for height := int64(0) IMO.

github-actions bot added C:CLI C:x/auth C:x/authz and removed C:x/authz labels Apr 29, 2022

fix: simulation and grpc query concurrency with ABCI

6fd8642

p0mvn force-pushed the roman/simulate-tx branch from 5d920ae to 6fd8642 Compare April 30, 2022 18:58

p0mvn added 4 commits April 30, 2022 15:46

table tests for selectHeight

ffbc01a

add table-driven GRPC query tests and fix non-existent heights

2412f7d

clean up

cd30668

clean up grpc query tests

3a6d01a

p0mvn commented Apr 30, 2022

View reviewed changes

client/grpc_query_test.go Outdated Show resolved Hide resolved

p0mvn commented Apr 30, 2022

View reviewed changes

client/grpc_query_test.go Outdated Show resolved Hide resolved

Apply suggestions from code review

10f89a3

p0mvn commented Apr 30, 2022

View reviewed changes

testutil/network/network.go Outdated Show resolved Hide resolved

Update testutil/network/network.go

7caa9cd

p0mvn changed the title ~~fix: simulation concurrency race~~ fix: concurrency issues between simulation, grpc queries and ABCI commit flow Apr 30, 2022

fix typo in abci.go

b55cc3a

p0mvn marked this pull request as ready for review May 1, 2022 00:10

p0mvn requested a review from alexanderbez as a code owner May 1, 2022 00:10

p0mvn requested a review from ValarDragon May 1, 2022 00:10

p0mvn mentioned this pull request May 1, 2022

bug: concurrent commit and RPC Simulate call causes a data race in IAVL osmosis-labs/osmosis#1356

Closed

alexanderbez reviewed May 2, 2022

View reviewed changes

p0mvn and others added 3 commits May 2, 2022 06:52

Update baseapp/abci.go

0b54d4b

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>

store last block height in a var

798a3b0

avoid returning named parameters

2b138fa

alexanderbez approved these changes May 2, 2022

View reviewed changes

p0mvn added the A:backport/v0.45.0x-osmo-v7 label May 2, 2022

p0mvn merged commit 18b0d2d into osmosis-main May 2, 2022

p0mvn deleted the roman/simulate-tx branch May 2, 2022 18:13

mergify bot mentioned this pull request May 2, 2022

fix: concurrency issues between simulation, grpc queries and ABCI commit flow (backport #213) #216

Merged

alexanderbez pushed a commit that referenced this pull request May 2, 2022

fix: concurrency issues between simulation, grpc queries and ABCI com…

5831ca0

…mit flow (#213) (#216)

ValarDragon reviewed May 3, 2022

View reviewed changes

p0mvn mentioned this pull request May 5, 2022

fix: revert iterator #212

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: concurrency issues between simulation, grpc queries and ABCI commit flow #213

fix: concurrency issues between simulation, grpc queries and ABCI commit flow #213

p0mvn commented Apr 29, 2022 •

edited

Loading

alexanderbez May 2, 2022

alexanderbez May 2, 2022

ValarDragon May 3, 2022

ValarDragon May 3, 2022

p0mvn May 3, 2022

alexanderbez May 3, 2022

fix: concurrency issues between simulation, grpc queries and ABCI commit flow #213

fix: concurrency issues between simulation, grpc queries and ABCI commit flow #213

Conversation

p0mvn commented Apr 29, 2022 • edited Loading

Description

Task Log

alexanderbez May 2, 2022

Choose a reason for hiding this comment

alexanderbez May 2, 2022

Choose a reason for hiding this comment

ValarDragon May 3, 2022

Choose a reason for hiding this comment

ValarDragon May 3, 2022

Choose a reason for hiding this comment

p0mvn May 3, 2022

Choose a reason for hiding this comment

alexanderbez May 3, 2022

Choose a reason for hiding this comment

p0mvn commented Apr 29, 2022 •

edited

Loading