dagstore: mmap-related panic when serving retrieval

[raulk]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

lotus miner/market - storage deal

Lotus Version

m1.3.3

Describe the Bug

goroutine 919 [running]:
runtime.throw(0x33563b1, 0x5)
	/usr/local/go/src/runtime/panic.go:1117 +0x72 fp=0xc001ff1328 sp=0xc001ff12f8 pc=0x58c112
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:741 +0x268 fp=0xc001ff1360 sp=0xc001ff1328 pc=0x5a3c08
runtime.memmove(0xc06c98e000, 0x7f9249b57bc9, 0x100026)
	/usr/local/go/src/runtime/memmove_amd64.s:392 +0x485 fp=0xc001ff1368 sp=0xc001ff1360 pc=0x5c7de5
golang.org/x/exp/mmap.(*ReaderAt).ReadAt(0xc04262da88, 0xc06c98e000, 0x100026, 0x100026, 0x39db57bc9, 0x7fe9f02f2918, 0x0, 0x100026)
	/home/acemgt/go/pkg/mod/golang.org/x/exp@v0.0.0-20210715201039-d37aa40e8013/mmap/mmap_unix.go:73 +0x185 fp=0xc001ff13c8 sp=0xc001ff1368 pc=0xd4cbc5
github.com/ipld/go-car/v2/internal/io.(*OffsetReadSeeker).Read(0xc05fa88220, 0xc06c98e000, 0x100026, 0x100026, 0xc001ff1468, 0x5a4dec, 0x100026)
	/home/acemgt/go/pkg/mod/github.com/ipld/go-car/v2@v2.0.0-beta1.0.20210721090610-5a9d1b217d25/internal/io/offset_read_seeker.go:32 +0x5a fp=0xc001ff1418 sp=0xc001ff13c8 pc=0xd3ce9a
io.ReadAtLeast(0x381c600, 0xc05fa88220, 0xc06c98e000, 0x100026, 0x100026, 0x100026, 0x20, 0x20, 0x3121c20)
	/usr/local/go/src/io/io.go:328 +0x87 fp=0xc001ff1478 sp=0xc001ff1418 pc=0x60d067
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
github.com/ipld/go-car/v2/internal/carv1/util.LdRead(0x381c600, 0xc05fa88220, 0xc0595a57a0, 0xc0378a0000, 0x0, 0xc001ff1548, 0x561e98)
	/home/acemgt/go/pkg/mod/github.com/ipld/go-car/v2@v2.0.0-beta1.0.20210721090610-5a9d1b217d25/internal/carv1/util/util.go:75 +0x1aa fp=0xc001ff14f0 sp=0xc001ff1478 pc=0xd4bf6a
github.com/ipld/go-car/v2/internal/carv1/util.ReadNode(0x381c600, 0xc05fa88220, 0x1, 0x1, 0x3841698, 0xc001ff1608, 0x56173b, 0xc0545da300, 0x40)
	/home/acemgt/go/pkg/mod/github.com/ipld/go-car/v2@v2.0.0-beta1.0.20210721090610-5a9d1b217d25/internal/carv1/util/util.go:19 +0x39 fp=0xc001ff1558 sp=0xc001ff14f0 pc=0xd4b9d9
github.com/ipld/go-car/v2/blockstore.(*ReadOnly).readBlock(0xc014a728c0, 0x39db57bc6, 0x12fc0, 0x1, 0xc001ff1620, 0xd4a22f, 0xc001ff1718, 0x487d, 0xc0564df416)
	/home/acemgt/go/pkg/mod/github.com/ipld/go-car/v2@v2.0.0-beta1.0.20210721090610-5a9d1b217d25/blockstore/readonly.go:165 +0x91 fp=0xc001ff15c0 sp=0xc001ff1558 pc=0xd64cb1
github.com/ipld/go-car/v2/blockstore.(*ReadOnly).Get.func1(0x39db57bc6, 0xc062bbb3b0)
	/home/acemgt/go/pkg/mod/github.com/ipld/go-car/v2@v2.0.0-beta1.0.20210721090610-5a9d1b217d25/blockstore/readonly.go:218 +0x7b fp=0xc001ff1668 sp=0xc001ff15c0 pc=0xd687fb
github.com/ipld/go-car/v2/index.(*singleWidthIndex).getAll(0xc001ff1718, 0xc0564df416, 0x20, 0x2a, 0xc0410d4540, 0x0, 0x7fea6d2e33c8)
	/home/acemgt/go/pkg/mod/github.com/ipld/go-car/v2@v2.0.0-beta1.0.20210721090610-5a9d1b217d25/index/indexsorted.go:96 +0x174 fp=0xc001ff16d8 sp=0xc001ff1668 pc=0xd48f74
github.com/ipld/go-car/v2/index.(*multiWidthIndex).GetAll(0xc02ab6cc60, 0xc05a8f0d50, 0x26, 0xc0410d4540, 0xc018ae0eb0, 0xc001ff1834)
	/home/acemgt/go/pkg/mod/github.com/ipld/go-car/v2@v2.0.0-beta1.0.20210721090610-5a9d1b217d25/index/indexsorted.go:129 +0x159 fp=0xc001ff1778 sp=0xc001ff16d8 pc=0xd494b9
github.com/ipld/go-car/v2/blockstore.(*ReadOnly).Get(0xc014a728c0, 0xc05a8f0d50, 0x26, 0x0, 0x0, 0x0, 0x0)
	/home/acemgt/go/pkg/mod/github.com/ipld/go-car/v2@v2.0.0-beta1.0.20210721090610-5a9d1b217d25/blockstore/readonly.go:217 +0x14a fp=0xc001ff1808 sp=0xc001ff1778 pc=0xd651ca
github.com/filecoin-project/lotus/markets/dagstore.(*ReadOnlyBlockstore).Get(0xc01dabf5c0, 0xc05a8f0d50, 0x26, 0x0, 0x384fb10, 0xc028041c60, 0x3616760)
	<autogenerated>:1 +0x50 fp=0xc001ff1850 sp=0xc001ff1808 pc=0x1c56950
github.com/filecoin-project/lotus/markets/dagstore.(*closableBlockstore).Get(0xc00c2863e0, 0xc05a8f0d50, 0x26, 0x0, 0x0, 0x92dc7d, 0xc000514db8)
	<autogenerated>:1 +0x50 fp=0xc001ff1898 sp=0xc001ff1850 pc=0x1c56eb0
github.com/filecoin-project/go-fil-markets/retrievalmarket/impl.(*lazyBlockstore).Get(0xc028041c60, 0xc05a8f0d50, 0x26, 0x3888cff, 0xc07580bd00, 0x3888d18, 0xc008a9b8e0)
	/home/acemgt/go/pkg/mod/github.com/filecoin-project/go-fil-markets@v1.6.3-0.20210729173742-a44f98a4f8c1/retrievalmarket/impl/lazyblockstore.go:53 +0x85 fp=0xc001ff18e0 sp=0xc001ff1898 pc=0x19bc565
github.com/ipfs/go-graphsync/storeutil.LoaderForBlockstore.func1(0x384fb10, 0xc01bb5d310, 0xc039099950, 0x6, 0x6, 0x3888c78, 0xc07580bde0, 0x3888d18, 0xc008a9b8e0, 0x0, ...)
	/home/acemgt/go/pkg/mod/github.com/ipfs/go-graphsync@v0.6.6/storeutil/storeutil.go:22 +0x5e fp=0xc001ff1940 sp=0xc001ff18e0 pc=0xdc797e
github.com/ipfs/go-graphsync/responsemanager/runtraversal.RunTraversal(0xc0347fe7c8, 0x386a7e0, 0xc0487afdc0, 0xc001ff1c48, 0x0, 0x2)
	/home/acemgt/go/pkg/mod/github.com/ipfs/go-graphsync@v0.6.6/responsemanager/runtraversal/runtraversal.go:40 +0x222 fp=0xc001ff1bb8 sp=0xc001ff1940 pc=0x1ad3202
github.com/ipfs/go-graphsync/responsemanager.(*queryExecutor).executeQuery(0xc0155ff000, 0xc008483bc0, 0x26, 0xc014e70090, 0x26, 0x38895d8, 0xc01b459d60, 0x8200000000, 0xc05eb68f60, 0x0, ...)
	/home/acemgt/go/pkg/mod/github.com/ipfs/go-graphsync@v0.6.6/responsemanager/queryexecutor.go:89 +0x172 fp=0xc001ff1cd8 sp=0xc001ff1bb8 pc=0x1ad4672
github.com/ipfs/go-graphsync/responsemanager.(*queryExecutor).processQueriesWorker(0xc0155ff000)
	/home/acemgt/go/pkg/mod/github.com/ipfs/go-graphsync@v0.6.6/responsemanager/queryexecutor.go:68 +0x5bb fp=0xc001ff1fd8 sp=0xc001ff1cd8 pc=0x1ad41fb
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc001ff1fe0 sp=0xc001ff1fd8 pc=0x5c6b81
created by github.com/ipfs/go-graphsync/responsemanager.(*ResponseManager).run
	/home/acemgt/go/pkg/mod/github.com/ipfs/go-graphsync@v0.6.6/responsemanager/responsemanager.go:305 +0x90

Logging Information

Above.

Repo Steps

N/A

[raulk]

Unfortunately go-car doesn't return an error when accessed after closed. It will go ahead and panic because the mmap was no longer active.

[raulk]

This happened in m1.3.3 because cancelling a graphsync request was racy. We would close the blockstore, but then graphsync would attempt to read from it again.

That was solved in filecoin-project/go-data-transfer#229 and integrated in m1.3.4, so this error should not be seen any longer in m1.3.4.

However, there's an underlying safety issue in go-car. It should detect when a blockstore was closed and be able to reject subsequent operations: ipld/go-car#205. This will be added and integrated very soon.