Use custom multihash and refactor plugin (#155)

* change plugin package

* update to new rsmt2d API and update go.mod

* Moved (pre)loading the plugin to init() of the plugin package itself

* minor clarification with regard to the skipped test

* refactor plugin to be more easily usable as a lib

* Use cid.Undef and remove obsolete comment

* use that lazyledger/go-ipfs fork

* use ll go-verifcid and go-ipfs fork in main ll-core package too

* update forks to latest version

* replace deps in plugin and reorg in ll-core

* bump ipfs related versions

* Register custom hasher from plugin code instead of in go-multihash fork
 - use hasher from nmt instead of redefining the logic in multihash or here
   (see celestiaorg/nmt#19)
 - remove multihash fork and use updated go-verifcid
 - minor renaming and refactoring

* update nmt and assert that the hashFunc is registered

* small test to validate verifcid allows our hasher

* actually skip test

* use remote instead of local go-ipfs repo

* clean up Makefile, remove set-taget, and add comment

 about building locally (in Readme) and
 remove ipfs dependency from ll-core module
 (this will be part of #152 instead)

* review feedback: rename to mustRegisterNamespacedCodec

* more review feedback: rename to mustRegisterNamespacedCodec and make sure the codec isn't already known

* update nmt too

* address nits: remove some unnecessary else statements

* Update p2p/ipld/README.md

Co-authored-by: John Adler <adlerjohn@users.noreply.github.com>

* s/defaulLength/defaultLength

* Update p2p/ipld/plugin/nodes/nodes.go

Co-authored-by: John Adler <adlerjohn@users.noreply.github.com>

* Deduplicate nmtHashSize

* implement Copy

* fix Tree implementation

* Address more review comments:
 - rename share to namespacedLeaf
 - fix doc comment
"When you realize you forgot to Load more https://www.youtube.com/watch?v=8yGfQak-q9M"

* Add clarifying comment to `DataSquareRowOrColumnRawInputParser` and rename constant

* minor doc improvement

Co-authored-by: John Adler <adlerjohn@users.noreply.github.com>

go.mod

@@ -17,8 +17,8 @@ require (
 	github.com/gorilla/websocket v1.4.2
 	github.com/gtank/merlin v0.1.1
 	github.com/hdevalence/ed25519consensus v0.0.0-20201207055737-7fde80a9d5ff
-	github.com/lazyledger/nmt v0.1.0
-	github.com/lazyledger/rsmt2d v0.0.0-20200626141417-ea94438fa457
+	github.com/lazyledger/nmt v0.2.0
+	github.com/lazyledger/rsmt2d v0.0.0-20201215203123-e5ec7910ddd4
 	github.com/libp2p/go-buffer-pool v0.0.2
 	github.com/minio/highwayhash v1.0.1
 	github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e // indirect

go.sum

@@ -21,12 +21,6 @@ github.com/ChainSafe/go-schnorrkel v0.0.0-20200405005733-88cbf1b4c40d/go.mod h1:
 github.com/DataDog/zstd v1.4.1 h1:3oxKN3wbHibqx897utPC2LTQU4J+IHWWJO+glkAkpFM=
 github.com/DataDog/zstd v1.4.1/go.mod h1:1jcaCB/ufaK+sKp1NBhlGmpz41jOoPQ35bpF36t7BBo=
 github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible/go.mod h1:r7JcOSlj0wfOMncg0iLm8Leh48TZaKVeNIfJntJ2wa0=
-github.com/NebulousLabs/errors v0.0.0-20181203160057-9f787ce8f69e h1:9UwdEr0AFI021vXG+hXDGH9ZHGuNtPCdJkRTqcIhJ3A=
-github.com/NebulousLabs/errors v0.0.0-20181203160057-9f787ce8f69e/go.mod h1:J7tUI9Fg4YuFLsqeLE5uIp93Fot9oBCw2vwZJvLmWso=
-github.com/NebulousLabs/fastrand v0.0.0-20181203155948-6fb6489aac4e h1:n+DcnTNkQnHlwpsrHoQtkrJIO7CBx029fw6oR4vIob4=
-github.com/NebulousLabs/fastrand v0.0.0-20181203155948-6fb6489aac4e/go.mod h1:Bdzq+51GR4/0DIhaICZEOm+OHvXGwwB2trKZ8B4Y6eQ=
-github.com/NebulousLabs/merkletree v0.0.0-20181203152040-08d5d54b07f5 h1:pk9SclNGplPbF6YDIDKMhHh9SaUWcoxPkMr7zdu1hfk=
-github.com/NebulousLabs/merkletree v0.0.0-20181203152040-08d5d54b07f5/go.mod h1:Cn056wBLKay+uIS9LJn7ymwhgC5mqbOtG6iOhEvyy4M=
 github.com/OneOfOne/xxhash v1.2.2 h1:KMrpdQIwFcEqXDklaen+P1axHaj9BSKzvpUUfnHldSE=
 github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
 github.com/Shopify/sarama v1.19.0/go.mod h1:FVkBWblsNy7DGZRfXLU0O9RCGt5g3g3yEuWXgklEdEo=
@@ -172,6 +166,7 @@ github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekf
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
 github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
+github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 h1:ZgQEtGgCBiWRM39fZuwSd1LwSqqSW0hOdXCYYDX0R3I=
 github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
@@ -254,6 +249,7 @@ github.com/hashicorp/go-uuid v1.0.1/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/b
 github.com/hashicorp/go-version v1.2.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
 github.com/hashicorp/go.net v0.0.1/go.mod h1:hjKkEWcCURg++eb33jQU7oqQcI9XDCnUzHA0oac0k90=
 github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
+github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU=
 github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
 github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
@@ -295,6 +291,7 @@ github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxv
 github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515 h1:T+h1c/A9Gawja4Y9mFVWj2vyii2bbUNDw3kt9VxK2EY=
 github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
+github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
@@ -303,10 +300,10 @@ github.com/lazyledger/go-leopard v0.0.0-20200604113236-298f93361181 h1:mUeCGuCgj
 github.com/lazyledger/go-leopard v0.0.0-20200604113236-298f93361181/go.mod h1:v1o1CRihQ9i7hizx23KK4aR79lxA6VDUIzUCfDva0XQ=
 github.com/lazyledger/merkletree v0.0.0-20201214195110-6901c4c3c75f h1:jbyPAH6o6hGte4RtZBaqWs2n4Fl6hS7qJGXX3qnjiy4=
 github.com/lazyledger/merkletree v0.0.0-20201214195110-6901c4c3c75f/go.mod h1:10PA0NlnYtB8HrtwIDQAyTKWp8TEZ0zBZCGlYC/7+QE=
-github.com/lazyledger/nmt v0.1.0 h1:GSpAmBc3B7cwJSYTjh/v+vPQCsNCqKzSklu8LzYxwaU=
-github.com/lazyledger/nmt v0.1.0/go.mod h1:tY7ypPX26Sbkt6F8EbPl3AT33B5N0BJe4OVPbq849YI=
-github.com/lazyledger/rsmt2d v0.0.0-20200626141417-ea94438fa457 h1:mXl0oKmdYegd8HaP++leltErwDnvcR2v+nttbRA8pk8=
-github.com/lazyledger/rsmt2d v0.0.0-20200626141417-ea94438fa457/go.mod h1:X2BD0VWh4t2V361JujYpGpYKwjsyp8vgr4iFvYo+Ojg=
+github.com/lazyledger/nmt v0.2.0 h1:jKpC+XMSc4P1asLvutSwyMlgUDnAvv1HAVyzal21MbA=
+github.com/lazyledger/nmt v0.2.0/go.mod h1:tY7ypPX26Sbkt6F8EbPl3AT33B5N0BJe4OVPbq849YI=
+github.com/lazyledger/rsmt2d v0.0.0-20201215203123-e5ec7910ddd4 h1:r2mI8AnDTOSwL9GWGjgHC4rwzaR56jRCo1GLpG2lwIw=
+github.com/lazyledger/rsmt2d v0.0.0-20201215203123-e5ec7910ddd4/go.mod h1:ORR2U7THCNr1fpUhwYqZN7QCFJ20iR2uiIWfXKz3KJ4=
 github.com/libp2p/go-buffer-pool v0.0.2 h1:QNK2iAFa8gjAe1SPz6mHSMuCcjs+X1wlHzeOSqcmlfs=
 github.com/libp2p/go-buffer-pool v0.0.2/go.mod h1:MvaB6xw5vOrDl8rYZGLFdKAuk/hRoRZd1Vi32+RXyFM=
 github.com/lightstep/lightstep-tracer-common/golang/gogo v0.0.0-20190605223551-bc2310a04743/go.mod h1:qklhhLq1aX+mtWk9cPHPzaBjWImj5ULL6C7HFJtXQMM=
@@ -372,6 +369,7 @@ github.com/op/go-logging v0.0.0-20160315200505-970db520ece7/go.mod h1:HzydrMdWEr
 github.com/opentracing-contrib/go-observer v0.0.0-20170622124052-a52f23424492/go.mod h1:Ngi6UdF0k5OKD5t5wlmGhe/EDKPoUM3BXZSSfIuJbis=
 github.com/opentracing/basictracer-go v1.0.0/go.mod h1:QfBfYuafItcjQuMwinw9GhYKwFXS9KnPs5lxoYwgW74=
 github.com/opentracing/opentracing-go v1.0.2/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
+github.com/opentracing/opentracing-go v1.1.0 h1:pWlfV3Bxv7k65HYwkikxat0+s3pV4bsqf19k25Ur8rU=
 github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
 github.com/openzipkin-contrib/zipkin-go-opentracing v0.4.5/go.mod h1:/wsWhb9smxSfWAKL3wpBW7V8scJMt8N8gnaMCS9E/cA=
 github.com/openzipkin/zipkin-go v0.1.6/go.mod h1:QgAqvLzwWbR/WpD4A3cGpPtJrZXNIiJc5AZX7/PBEpw=

p2p/ipld/README.md

@@ -4,15 +4,8 @@
 
 Usually, a simple `make install` should build and install the plugin locally.
 
-If you want to compile against a locally checked out version of IPFS, run:
-
-```sh
-cd plugin
-./set-target.sh REPLACE_WITH_YOUR_LOCAL_PATH/go-ipfs
-make install
-```
+If you want to compile against a locally checked out version of IPFS, change the `replace` directive in [go.mod](./plugin/go.mod).
 
 Running ipfs (e.g. `ipfs init` or `ipfs daemon`) loads the plugin automatically.
 
 Note, this plugin was only tested with IPFS [v0.7.0](https://github.com/ipfs/go-ipfs/releases/tag/v0.7.0).
-

p2p/ipld/plugin/Makefile

@@ -22,11 +22,6 @@ endif
 
 .PHONY: install build
 
-go.mod: FORCE
-	./set-target.sh $(IPFS_VERSION)
-
-FORCE:
-
 lazyledger-plugin.so: plugin.go go.mod
 	$(GOCC) build $(GOFLAGS) -buildmode=plugin -o "$@" "$<"
 	chmod +x "$@"

p2p/ipld/plugin/go.mod

@@ -8,8 +8,17 @@ require (
 	github.com/ipfs/go-ipfs v0.7.0
 	github.com/ipfs/go-ipfs-api v0.2.0
 	github.com/ipfs/go-ipld-format v0.2.0
-	github.com/lazyledger/nmt v0.1.0
+	github.com/ipfs/go-verifcid v0.0.1
+	github.com/lazyledger/nmt v0.2.0
 	// rsmt2d is only used in tests:
 	github.com/lazyledger/rsmt2d v0.0.0-20201215203123-e5ec7910ddd4
 	github.com/multiformats/go-multihash v0.0.14
 )
+
+replace (
+	// If you encounter:
+	//  plugin was built with a different version of package X
+	// overwrite below line with a local directory with the corresponding version of go-ipfs checked out
+	github.com/ipfs/go-ipfs => github.com/lazyledger/go-ipfs v0.7.0-lazypatch
+	github.com/ipfs/go-verifcid => github.com/lazyledger/go-verifcid v0.0.1-lazypatch
+)