/cosmos/staking/validators/{validator}/delegators unstable when querying

[FloppyDisck]

Summary of Bug

This bug applies to any blockchain using the recent SDK (Terra and Cosmos). Querying any validator related information either takes an unexpectedly high amount of time (> 10 minutes) or ends in returning a 504 error.

Version

Cosmos SDK v0.42.6 (Gaia v5.0.2 / cosmoshub-4)

Steps to Reproduce

GET https://api.cosmos.network/cosmos/staking/validators/{validator}/delegators?pagination.limit=100

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[alexanderbez]

Yeah I think we've seen numerous reports of staking related endpoints being completely unusable. @robert-zaremba or @AmauryM can you lmk if we already have this captured in another issue? Also, can you summarize why this is? Is it because of gRPC and mutex contention and/or poor index design?

[amaury1093]

I don't think we have another issue for this, we can use this one.

Is it because of gRPC and mutex contention and/or poor index design?

I'd say poor index design (gut feeling). I think the current design forces us to loop through all delegations, unmarshal each of them, and check if each one's validator matches the requested one. We should do some benchmarks to verify this hypothesis though, and add new indices if needed.

Edit: see

cosmos-sdk/x/staking/keeper/grpc_query.go

Lines 102 to 103 in da87ab0

	pageRes, err := query.FilteredPaginate(valStore, req.Pagination, func(key []byte, value []byte, accumulate bool) (bool, error) {
	delegation, err := types.UnmarshalDelegation(k.cdc, value)

and following lines

[freak12techno]

I researched it and came to the conclusion that it iterates over ALL of the delegation entries, including the ones by other validators, then takes first N (N = limit) entries. If there are less delegators than N, it'll iterate over all the delegations. Here's a bit of my research: #15162

[alexanderbez]

Closing this one as it's a dup.