Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make store.Query use ics23-proofs #6315

Closed
wants to merge 6 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ require (
github.com/99designs/keyring v1.1.5
github.com/bgentry/speakeasy v0.1.0
github.com/btcsuite/btcd v0.20.1-beta
github.com/confio/ics23-iavl v0.6.0
github.com/confio/ics23/go v0.0.0-20200325200809-9f53dd0c4212
github.com/cosmos/go-bip39 v0.0.0-20180819234021-555e2067c45d
github.com/cosmos/ledger-cosmos-go v0.11.1
Expand Down
5 changes: 5 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGX
github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa/go.mod h1:zn76sxSg3SzpJ0PPJaLDCu+Bu0Lg3sKTORVIj19EIF8=
github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd/go.mod h1:sE/e/2PUdi/liOCUjSTXgM1o87ZssimdTWN964YiIeI=
github.com/confio/ics23 v0.6.0 h1:bQsi55t2+xjW6EWDl83IBF1VWurplbUu+OT6pukeiEo=
github.com/confio/ics23-iavl v0.6.0 h1:vVRCuVaP38FCw1kTeEdFuGuiY+2vAGTBQoH7Zxkq/ws=
github.com/confio/ics23-iavl v0.6.0/go.mod h1:mmXAxD1vWoO0VP8YHu6mM1QHGv71NQqa1iSVm4HeKcY=
github.com/confio/ics23/go v0.0.0-20200323120010-7d9a00f0a2fa/go.mod h1:W1I3XC8d9N8OTu/ct5VJ84ylcOunZwMXsWkd27nvVts=
github.com/confio/ics23/go v0.0.0-20200325200809-9f53dd0c4212 h1:MgS8JP5m7fPl7kumRm+YyAe5le3JlwQ4n5T/JXvr36s=
github.com/confio/ics23/go v0.0.0-20200325200809-9f53dd0c4212/go.mod h1:W1I3XC8d9N8OTu/ct5VJ84ylcOunZwMXsWkd27nvVts=
github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
Expand Down Expand Up @@ -451,13 +454,15 @@ github.com/tendermint/crypto v0.0.0-20191022145703-50d29ede1e15/go.mod h1:z4YtwM
github.com/tendermint/go-amino v0.14.1/go.mod h1:i/UKE5Uocn+argJJBb12qTZsCDBcAYMbR92AaJVmKso=
github.com/tendermint/go-amino v0.15.1 h1:D2uk35eT4iTsvJd9jWIetzthE5C0/k2QmMFkCN+4JgQ=
github.com/tendermint/go-amino v0.15.1/go.mod h1:TQU0M1i/ImAo+tYpZi73AU3V/dKeCoMC9Sphe2ZwGME=
github.com/tendermint/iavl v0.13.2/go.mod h1:vE1u0XAGXYjHykd4BLp8p/yivrw2PF1TuoljBcsQoGA=
github.com/tendermint/iavl v0.13.3 h1:expgBDY1MX+6/3sqrIxGChbTNf9N9aTJ67SH4bPchCs=
github.com/tendermint/iavl v0.13.3/go.mod h1:2lE7GiWdSvc7kvT78ncIKmkOjCnp6JEnSb2O7B9htLw=
github.com/tendermint/tendermint v0.33.2 h1:NzvRMTuXJxqSsFed2J7uHmMU5N1CVzSpfi3nCc882KY=
github.com/tendermint/tendermint v0.33.2/go.mod h1:25DqB7YvV1tN3tHsjWoc2vFtlwICfrub9XO6UBO+4xk=
github.com/tendermint/tendermint v0.33.4 h1:NM3G9618yC5PaaxGrcAySc5ylc1PAANeIx42u2Re/jo=
github.com/tendermint/tendermint v0.33.4/go.mod h1:6NW9DVkvsvqmCY6wbRsOo66qGDhMXglRL70aXajvBEA=
github.com/tendermint/tm-db v0.4.1/go.mod h1:JsJ6qzYkCGiGwm5GHl/H5GLI9XLb6qZX7PRe425dHAY=
github.com/tendermint/tm-db v0.5.0/go.mod h1:lSq7q5WRR/njf1LnhiZ/lIJHk2S8Y1Zyq5oP/3o9C2U=
github.com/tendermint/tm-db v0.5.1 h1:H9HDq8UEA7Eeg13kdYckkgwwkQLBnJGgX4PgLJRhieY=
github.com/tendermint/tm-db v0.5.1/go.mod h1:g92zWjHpCYlEvQXvy9M168Su8V1IBEeawpXVVBaK4f4=
github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
Expand Down
109 changes: 109 additions & 0 deletions store/iavl/proof.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
package iavl

import (
"errors"
"fmt"

ics23 "github.com/confio/ics23/go"
"github.com/tendermint/tendermint/crypto/merkle"
)

const ProofOpIAVL = "iavlstore"

type IAVLOp struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, if it makes it easier, use a different proof type for existence and absence. So you don't have to type switch below. Makes more sense to this code, but not sure in how it is used externally.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm... if other people are in favor I may go for it, it's certainly doable. I agree it removes the type switch, but virtually everything else about how to proof is constructed is identical so I think having the switch statement here is fine

Key []byte
Proof *ics23.CommitmentProof
}

var _ merkle.ProofOperator = IAVLOp{}

func NewIAVLOp(key []byte, proof *ics23.CommitmentProof) IAVLOp {
return IAVLOp{
Key: key,
Proof: proof,
}
}

func IAVLOpDecoder(pop merkle.ProofOp) (merkle.ProofOperator, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This guy definitely warrants a godoc 👍

if pop.Type != ProofOpIAVL {
return nil, errors.New(fmt.Sprintf("unexpected ProofOp.Type; got %v, want %v", pop.Type, ProofOpIAVL))
}
var op IAVLOp
proof := &ics23.CommitmentProof{}
err := proof.Unmarshal(pop.Data)
if err != nil {
return nil, err
}
op.Proof = proof

// Get Key from proof for now
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the logic comparisons to ics23 types, are you sure the ok semantics are correct? It reads as if it should be ; ok { .. } and not ; !ok { ... }

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently op.Data just contains marshalled ics proof while the op has the key in a separate field. I extract key from proof here, though we may want to instead wrap the ics-proof in a proto struct containing proof and key and just directly unmarshal the proofoperator

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading this again, I see that the marshalled version is just the commitment proof and the key is only cached in memory when decoding Yeah, please ignore my comment about the newtype, I was really worrying about the serialized version.

if existProof, ok := op.Proof.Proof.(*ics23.CommitmentProof_Exist); ok {
op.Key = existProof.Exist.Key
} else if nonexistProof, ok := op.Proof.Proof.(*ics23.CommitmentProof_Nonexist); ok {
op.Key = nonexistProof.Nonexist.Key
} else {
return nil, errors.New("Proof type unsupported")
}
return op, nil
}

func (op IAVLOp) GetKey() []byte {
return op.Key
}

func (op IAVLOp) Run(args [][]byte) ([][]byte, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does Run do? Add a godoc here 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is super confusing. Not from @AdityaSripal but from whoever designed this ProofOperator interface:

https://github.com/tendermint/tendermint/blob/master/crypto/merkle/proof.go#L12-L18

It is totally unclear if it is checking absence or existence. And why it returns multiple roots. I think this ProofOp interface needs an overhaul as well, but I think he is implementing this properly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

switch len(args) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both Existence and Nonexistence proofs are being handled by the same IAVLop here

case 0:
// Args are nil, so we verify the absence of the key.
nonexistProof, ok := op.Proof.Proof.(*ics23.CommitmentProof_Nonexist)
if !ok {
return nil, errors.New("proof is not a nonexistence proof and args is nil")
}

root, err := nonexistProof.Nonexist.Left.Calculate()
AdityaSripal marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
return nil, errors.New("could not calculate root from nonexistence proof")
}

absent := ics23.VerifyNonMembership(ics23.IavlSpec, root, op.Proof, op.Key)
if !absent {
return nil, errors.New(fmt.Sprintf("proof did not verify absence of key: %s", string(op.Key)))
}

return [][]byte{root}, nil

case 1:
// Args is length 1, verify existence of key with value args[0]
existProof, ok := op.Proof.Proof.(*ics23.CommitmentProof_Exist)
if !ok {
return nil, errors.New("proof is not a existence proof and args is length 1")
}
// For subtree verification, we simply calculate the root from the proof and use it to prove
// against the value
root, err := existProof.Exist.Calculate()
if err != nil {
return nil, errors.New("could not calculate root from existence proof")
}

exists := ics23.VerifyMembership(ics23.IavlSpec, root, op.Proof, op.Key, args[0])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only iavl-specific code here is this ics23.IavlSpec. It would be good to somehow abstract the code out so we can use 98% the same code for simple merkle proofs.

Although that is for a future pr. It is fine as is now.

if !exists {
return nil, errors.New(fmt.Sprintf("proof did not verify existence of key %s with given value %x", op.Key, args[0]))
}

return [][]byte{root}, nil
default:
return nil, errors.New(fmt.Sprintf("args must be length 0 or 1, got: %d", len(args)))
}
}

func (op IAVLOp) ProofOp() merkle.ProofOp {
bz, err := op.Proof.Marshal()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note data just contains marshalled proof for now

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is quite nice.
A client (eg. js) can parse out the proof op type and data fields. type is just a string, and data is a protobuf message with mutli-langauge decoding support.

This is perfect to just store the protobuf data.

if err != nil {
panic(err.Error())
}
return merkle.ProofOp{
Type: ProofOpIAVL,
Key: op.Key,
Data: bz,
}
}
55 changes: 34 additions & 21 deletions store/iavl/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ import (
"io"
"sync"

ics23iavl "github.com/confio/ics23-iavl"
ics23 "github.com/confio/ics23/go"
"github.com/pkg/errors"
"github.com/tendermint/iavl"
abci "github.com/tendermint/tendermint/abci/types"
Expand Down Expand Up @@ -275,31 +277,42 @@ func (st *Store) Query(req abci.RequestQuery) (res abci.ResponseQuery) {
break
}

if req.Prove {
value, proof, err := tree.GetVersionedWithProof(key, res.Height)
if err != nil {
res.Log = err.Error()
break
}
if proof == nil {
// Proof == nil implies that the store is empty.
if value != nil {
panic("unexpected value for an empty proof")
}
_, res.Value = tree.GetVersioned(key, res.Height)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note we get the value from res.Height, but below use the tree from the current height.
This will fail if the key changed between the querier and current heights.

if !req.Prove {
break
}
// Continue to prove existence/absence of value
var commitmentProof *ics23.CommitmentProof
var err error

// Must convert store.Tree to iavl.MutableTree to use in CreateProof
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can use another type in ics23iavl, please let me know and make a pr to simplify this logic

Thank you for handling this case, but it really should go in that library (once you get it working here, let's pull it over there)

var mtree *iavl.MutableTree
switch t := tree.(type) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what I am talking about. We need the tree from the requested height here to use for proofs.

case *iavl.MutableTree:
mtree = t
case *immutableTree:
mtree = &iavl.MutableTree{
ImmutableTree: t.ImmutableTree,
}
if value != nil {
// value was found
res.Value = value
res.Proof = &merkle.Proof{Ops: []merkle.ProofOp{iavl.NewValueOp(key, proof).ProofOp()}}
} else {
// value wasn't found
res.Value = nil
res.Proof = &merkle.Proof{Ops: []merkle.ProofOp{iavl.NewAbsenceOp(key, proof).ProofOp()}}
default:
return sdkerrors.QueryResult(sdkerrors.Wrapf(sdkerrors.ErrUnknownRequest, "store must contain iavl.MutableTree or iavl.ImmutableTree to return proof"))
}

if res.Value != nil {
// value was found
commitmentProof, err = ics23iavl.CreateMembershipProof(mtree, req.Data)
if err != nil {
panic(fmt.Sprintf("unexpected value for empty proof: %s", err.Error()))
}
} else {
_, res.Value = tree.GetVersioned(key, res.Height)
// value wasn't found
commitmentProof, err = ics23iavl.CreateNonMembershipProof(mtree, req.Data)
if err != nil {
panic(fmt.Sprintf("unexpected empty absence proof: %s", err.Error()))
}
}

op := NewIAVLOp(req.Data, commitmentProof)
res.Proof = &merkle.Proof{Ops: []merkle.ProofOp{op.ProofOp()}}
case "/subspace":
var KVs []types.KVPair

Expand Down