Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make store.Query use ics23-proofs #6315

Closed
wants to merge 6 commits into from
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ require (
github.com/99designs/keyring v1.1.5
github.com/bgentry/speakeasy v0.1.0
github.com/btcsuite/btcd v0.20.1-beta
github.com/confio/ics23-iavl v0.6.0
github.com/confio/ics23/go v0.0.0-20200325200809-9f53dd0c4212
github.com/cosmos/go-bip39 v0.0.0-20180819234021-555e2067c45d
github.com/cosmos/ledger-cosmos-go v0.11.1
Expand Down
5 changes: 5 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGX
github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa/go.mod h1:zn76sxSg3SzpJ0PPJaLDCu+Bu0Lg3sKTORVIj19EIF8=
github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd/go.mod h1:sE/e/2PUdi/liOCUjSTXgM1o87ZssimdTWN964YiIeI=
github.com/confio/ics23 v0.6.0 h1:bQsi55t2+xjW6EWDl83IBF1VWurplbUu+OT6pukeiEo=
github.com/confio/ics23-iavl v0.6.0 h1:vVRCuVaP38FCw1kTeEdFuGuiY+2vAGTBQoH7Zxkq/ws=
github.com/confio/ics23-iavl v0.6.0/go.mod h1:mmXAxD1vWoO0VP8YHu6mM1QHGv71NQqa1iSVm4HeKcY=
github.com/confio/ics23/go v0.0.0-20200323120010-7d9a00f0a2fa/go.mod h1:W1I3XC8d9N8OTu/ct5VJ84ylcOunZwMXsWkd27nvVts=
github.com/confio/ics23/go v0.0.0-20200325200809-9f53dd0c4212 h1:MgS8JP5m7fPl7kumRm+YyAe5le3JlwQ4n5T/JXvr36s=
github.com/confio/ics23/go v0.0.0-20200325200809-9f53dd0c4212/go.mod h1:W1I3XC8d9N8OTu/ct5VJ84ylcOunZwMXsWkd27nvVts=
github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
Expand Down Expand Up @@ -451,13 +454,15 @@ github.com/tendermint/crypto v0.0.0-20191022145703-50d29ede1e15/go.mod h1:z4YtwM
github.com/tendermint/go-amino v0.14.1/go.mod h1:i/UKE5Uocn+argJJBb12qTZsCDBcAYMbR92AaJVmKso=
github.com/tendermint/go-amino v0.15.1 h1:D2uk35eT4iTsvJd9jWIetzthE5C0/k2QmMFkCN+4JgQ=
github.com/tendermint/go-amino v0.15.1/go.mod h1:TQU0M1i/ImAo+tYpZi73AU3V/dKeCoMC9Sphe2ZwGME=
github.com/tendermint/iavl v0.13.2/go.mod h1:vE1u0XAGXYjHykd4BLp8p/yivrw2PF1TuoljBcsQoGA=
github.com/tendermint/iavl v0.13.3 h1:expgBDY1MX+6/3sqrIxGChbTNf9N9aTJ67SH4bPchCs=
github.com/tendermint/iavl v0.13.3/go.mod h1:2lE7GiWdSvc7kvT78ncIKmkOjCnp6JEnSb2O7B9htLw=
github.com/tendermint/tendermint v0.33.2 h1:NzvRMTuXJxqSsFed2J7uHmMU5N1CVzSpfi3nCc882KY=
github.com/tendermint/tendermint v0.33.2/go.mod h1:25DqB7YvV1tN3tHsjWoc2vFtlwICfrub9XO6UBO+4xk=
github.com/tendermint/tendermint v0.33.4 h1:NM3G9618yC5PaaxGrcAySc5ylc1PAANeIx42u2Re/jo=
github.com/tendermint/tendermint v0.33.4/go.mod h1:6NW9DVkvsvqmCY6wbRsOo66qGDhMXglRL70aXajvBEA=
github.com/tendermint/tm-db v0.4.1/go.mod h1:JsJ6qzYkCGiGwm5GHl/H5GLI9XLb6qZX7PRe425dHAY=
github.com/tendermint/tm-db v0.5.0/go.mod h1:lSq7q5WRR/njf1LnhiZ/lIJHk2S8Y1Zyq5oP/3o9C2U=
github.com/tendermint/tm-db v0.5.1 h1:H9HDq8UEA7Eeg13kdYckkgwwkQLBnJGgX4PgLJRhieY=
github.com/tendermint/tm-db v0.5.1/go.mod h1:g92zWjHpCYlEvQXvy9M168Su8V1IBEeawpXVVBaK4f4=
github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
Expand Down
109 changes: 109 additions & 0 deletions store/iavl/proof.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
package iavl

import (
"errors"
"fmt"

ics23 "github.com/confio/ics23/go"
"github.com/tendermint/tendermint/crypto/merkle"
)

const ProofOpIAVL = "iavlstore"

type IAVLOp struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, if it makes it easier, use a different proof type for existence and absence. So you don't have to type switch below. Makes more sense to this code, but not sure in how it is used externally.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm... if other people are in favor I may go for it, it's certainly doable. I agree it removes the type switch, but virtually everything else about how to proof is constructed is identical so I think having the switch statement here is fine

Key []byte
Proof *ics23.CommitmentProof
}

var _ merkle.ProofOperator = IAVLOp{}

func NewIAVLOp(key []byte, proof *ics23.CommitmentProof) IAVLOp {
return IAVLOp{
Key: key,
Proof: proof,
}
}

func IAVLOpDecoder(pop merkle.ProofOp) (merkle.ProofOperator, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This guy definitely warrants a godoc 👍

if pop.Type != ProofOpIAVL {
return nil, errors.New(fmt.Sprintf("unexpected ProofOp.Type; got %v, want %v", pop.Type, ProofOpIAVL))
}
var op IAVLOp
proof := &ics23.CommitmentProof{}
err := proof.Unmarshal(pop.Data)
if err != nil {
return nil, err
}
op.Proof = proof

// Get Key from proof for now
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the logic comparisons to ics23 types, are you sure the ok semantics are correct? It reads as if it should be ; ok { .. } and not ; !ok { ... }

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently op.Data just contains marshalled ics proof while the op has the key in a separate field. I extract key from proof here, though we may want to instead wrap the ics-proof in a proto struct containing proof and key and just directly unmarshal the proofoperator

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading this again, I see that the marshalled version is just the commitment proof and the key is only cached in memory when decoding Yeah, please ignore my comment about the newtype, I was really worrying about the serialized version.

if existProof, ok := op.Proof.Proof.(*ics23.CommitmentProof_Exist); !ok {
op.Key = existProof.Exist.Key
} else if nonexistProof, ok := op.Proof.Proof.(*ics23.CommitmentProof_Nonexist); !ok {
op.Key = nonexistProof.Nonexist.Key
} else {
return nil, errors.New("Proof type unsupported")
}
return op, nil
}

func (op IAVLOp) GetKey() []byte {
return op.Key
}

func (op IAVLOp) Run(args [][]byte) ([][]byte, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does Run do? Add a godoc here 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is super confusing. Not from @AdityaSripal but from whoever designed this ProofOperator interface:

https://github.com/tendermint/tendermint/blob/master/crypto/merkle/proof.go#L12-L18

It is totally unclear if it is checking absence or existence. And why it returns multiple roots. I think this ProofOp interface needs an overhaul as well, but I think he is implementing this properly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

switch len(args) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both Existence and Nonexistence proofs are being handled by the same IAVLop here

case 0:
// Args are nil, so we verify the absence of the key.
nonexistProof, ok := op.Proof.Proof.(*ics23.CommitmentProof_Nonexist)
if !ok {
return nil, errors.New("proof is not a nonexistence proof and args is nil")
}

root, err := nonexistProof.Nonexist.Left.Calculate()
AdityaSripal marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
return nil, errors.New("could not calculate root from nonexistence proof")
}

absent := ics23.VerifyNonMembership(ics23.IavlSpec, root, op.Proof, op.Key)
if !absent {
return nil, errors.New(fmt.Sprintf("proof did not verify absence of key: %s", string(op.Key)))
}

return [][]byte{root}, nil

case 1:
// Args is length 1, verify existence of key with value args[0]
existProof, ok := op.Proof.Proof.(*ics23.CommitmentProof_Exist)
if !ok {
return nil, errors.New("proof is not a existence proof and args is length 1")
}
// For subtree verification, we simply calculate the root from the proof and use it to prove
// against the value
root, err := existProof.Exist.Calculate()
if err != nil {
return nil, errors.New("could not calculate root from existence proof")
}

exists := ics23.VerifyMembership(ics23.IavlSpec, root, op.Proof, op.Key, args[0])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only iavl-specific code here is this ics23.IavlSpec. It would be good to somehow abstract the code out so we can use 98% the same code for simple merkle proofs.

Although that is for a future pr. It is fine as is now.

if !exists {
return nil, errors.New(fmt.Sprintf("proof did not verify existence of key %s with given value %x", op.Key, args[0]))
}

return [][]byte{root}, nil
default:
return nil, errors.New(fmt.Sprintf("args must be length 0 or 1, got: %d", len(args)))
}
}

func (op IAVLOp) ProofOp() merkle.ProofOp {
bz, err := op.Proof.Marshal()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note data just contains marshalled proof for now

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is quite nice.
A client (eg. js) can parse out the proof op type and data fields. type is just a string, and data is a protobuf message with mutli-langauge decoding support.

This is perfect to just store the protobuf data.

if err != nil {
panic(err.Error())
}
return merkle.ProofOp{
Type: ProofOpIAVL,
Key: op.Key,
Data: bz,
}
}
43 changes: 25 additions & 18 deletions store/iavl/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ import (
"io"
"sync"

ics23iavl "github.com/confio/ics23-iavl"
ics23 "github.com/confio/ics23/go"
"github.com/pkg/errors"
"github.com/tendermint/iavl"
abci "github.com/tendermint/tendermint/abci/types"
Expand Down Expand Up @@ -275,31 +277,36 @@ func (st *Store) Query(req abci.RequestQuery) (res abci.ResponseQuery) {
break
}

_, res.Value = tree.GetVersioned(key, res.Height)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note we get the value from res.Height, but below use the tree from the current height.
This will fail if the key changed between the querier and current heights.

if req.Prove {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd love to see this being implemented as follows:

Suggested change
if req.Prove {
if !req.Prove {
break
}

This would reduce the indentation level and make the code more readable

value, proof, err := tree.GetVersionedWithProof(key, res.Height)
if err != nil {
res.Log = err.Error()
break
}
if proof == nil {
// Proof == nil implies that the store is empty.
if value != nil {
panic("unexpected value for an empty proof")
var commitmentProof *ics23.CommitmentProof
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: group vars :)

var err error
if res.Value != nil {
// Only support query proof from MutableTree for now
mtree, ok := tree.(*iavl.MutableTree)
Copy link
Contributor

@alessio alessio Jun 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd love to see mtree declared outside of this block - mostly for readability's sake.

if !ok {
return sdkerrors.QueryResult(sdkerrors.Wrapf(sdkerrors.ErrUnknownRequest, "store must contain iavl.MutableTree to return proof"))
}
}
if value != nil {
// value was found
res.Value = value
res.Proof = &merkle.Proof{Ops: []merkle.ProofOp{iavl.NewValueOp(key, proof).ProofOp()}}
commitmentProof, err = ics23iavl.CreateMembershipProof(mtree, req.Data)
if err != nil {
panic(fmt.Sprintf("unexpected value for empty proof: %s", err.Error()))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why we both panic and return and error. It seems like we should always return an error.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should never happen, so I panic. Also believe this matches previous behavior

}
} else {
// Only support query proof from MutableTree for now
mtree, ok := tree.(*iavl.MutableTree)
if !ok {
return sdkerrors.QueryResult(sdkerrors.Wrapf(sdkerrors.ErrUnknownRequest, "store must contain iavl.MutableTree to return proof"))
}
// value wasn't found
res.Value = nil
res.Proof = &merkle.Proof{Ops: []merkle.ProofOp{iavl.NewAbsenceOp(key, proof).ProofOp()}}
commitmentProof, err = ics23iavl.CreateNonMembershipProof(mtree, req.Data)
if err != nil {
panic(fmt.Sprintf("unexpected empty absence proof: %s", err.Error()))
}
}
} else {
_, res.Value = tree.GetVersioned(key, res.Height)
op := NewIAVLOp(req.Data, commitmentProof)
res.Proof = &merkle.Proof{Ops: []merkle.ProofOp{op.ProofOp()}}
}

case "/subspace":
var KVs []types.KVPair

Expand Down