M1 Testing with MinerX.2 Discussion

[jennijuju]

For discussing anything around M1 testing.

[TippyFlitsUK]

I think I have both separate instances running but I'm getting errors with all the storage deal reporting tools... for example:

lotus-miner storage-deals list
ERROR: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference

LOTUS_MINER_PATH=/mnt/nvme0n1/.lotusmarket lotus-miner storage-deals list
ERROR: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference

Any ideas?

[raulk]

@TippyFlitsUK Hey! Thanks for reporting. Would you mind getting us the miner logs that show the panic stacktrace, please?

cc @nonsense

[TippyFlitsUK]

No problem @raulk...

lotus-miner storage-deals list

2021-07-25T18:10:21.959+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158	panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
github.com/filecoin-project/go-jsonrpc.doCall.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:965
runtime.panicmem
	/usr/local/go/src/runtime/panic.go:212
runtime.sigpanic
	/usr/local/go/src/runtime/signal_unix.go:734
github.com/filecoin-project/lotus/node/impl.(*StorageMinerAPI).MarketListIncompleteDeals
	/home/tippyflits/lotus/node/impl/storminer.go:460
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/lotus/metrics.proxy.func1
	/home/tippyflits/lotus/metrics/proxy.go:60
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc/auth.PermissionedProxy.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/auth/auth.go:62
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc.doCall
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:162
github.com/filecoin-project/go-jsonrpc.(*RPCServer).handle
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:259
2021-07-25T18:10:21.962+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:95	RPC Error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
2021-07-25T18:10:21.962+0100	WARN	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:101	rpc error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference

LOTUS_MINER_PATH=/mnt/nvme0n1/.lotusmarket lotus-miner storage-deals list

2021-07-25T18:12:23.187+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158	panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
github.com/filecoin-project/go-jsonrpc.doCall.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:965
runtime.panicmem
	/usr/local/go/src/runtime/panic.go:212
runtime.sigpanic
	/usr/local/go/src/runtime/signal_unix.go:734
github.com/filecoin-project/lotus/node/impl.(*StorageMinerAPI).MarketListIncompleteDeals
	/home/tippyflits/lotus/node/impl/storminer.go:460
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/lotus/metrics.proxy.func1
	/home/tippyflits/lotus/metrics/proxy.go:60
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc/auth.PermissionedProxy.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/auth/auth.go:62
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc.doCall
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:162
github.com/filecoin-project/go-jsonrpc.(*RPCServer).handle
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:259
2021-07-25T18:12:23.188+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:95	RPC Error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
2021-07-25T18:12:23.188+0100	WARN	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:101	rpc error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference

lotus-miner info

2021-07-25T18:14:38.706+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158	panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
github.com/filecoin-project/go-jsonrpc.doCall.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:965
runtime.panicmem
	/usr/local/go/src/runtime/panic.go:212
runtime.sigpanic
	/usr/local/go/src/runtime/signal_unix.go:734
github.com/filecoin-project/lotus/node/impl.(*StorageMinerAPI).MarketListIncompleteDeals
	/home/tippyflits/lotus/node/impl/storminer.go:460
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/lotus/metrics.proxy.func1
	/home/tippyflits/lotus/metrics/proxy.go:60
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc/auth.PermissionedProxy.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/auth/auth.go:62
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc.doCall
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:162
github.com/filecoin-project/go-jsonrpc.(*RPCServer).handle
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:259
2021-07-25T18:14:38.706+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:95	RPC Error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
2021-07-25T18:14:38.706+0100	WARN	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:101	rpc error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference

LOTUS_MINER_PATH=/mnt/nvme0n1/.lotusmarket lotus-miner info

2021-07-25T18:15:48.040+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158	panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
github.com/filecoin-project/go-jsonrpc.doCall.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:158
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:965
runtime.panicmem
	/usr/local/go/src/runtime/panic.go:212
runtime.sigpanic
	/usr/local/go/src/runtime/signal_unix.go:734
github.com/filecoin-project/lotus/node/impl.(*StorageMinerAPI).MarketListIncompleteDeals
	/home/tippyflits/lotus/node/impl/storminer.go:460
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/lotus/metrics.proxy.func1
	/home/tippyflits/lotus/metrics/proxy.go:60
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc/auth.PermissionedProxy.func1
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/auth/auth.go:62
github.com/filecoin-project/lotus/api.(*StorageMinerStruct).MarketListIncompleteDeals
	/home/tippyflits/lotus/api/proxy_gen.go:2963
reflect.Value.call
	/usr/local/go/src/reflect/value.go:476
reflect.Value.Call
	/usr/local/go/src/reflect/value.go:337
github.com/filecoin-project/go-jsonrpc.doCall
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:162
github.com/filecoin-project/go-jsonrpc.(*RPCServer).handle
	/home/tippyflits/go/pkg/mod/github.com/filecoin-project/go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:259
2021-07-25T18:15:48.041+0100	ERROR	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:95	RPC Error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference
2021-07-25T18:15:48.041+0100	WARN	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/server.go:101	rpc error: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference

There are multiple commands that no longer work since the separation of miner and markets although the miner does appear to be running without issue. I have completed window posts and sealed sectors successfully on this release. I am unable to see any information with regards to deals and data transfers as all command are broken.

Let me know of you need anything else buddy!! Many thanks!!

[tmyuu]

@TippyFlitsUK
You might already be aware of this but able to avoid errors by specifying an appropriate one according to the usage environment.

• MARKETS_API_INFO env variable
• LOTUS_MARKETS_PATH env variable
• --markets-api-url flag
• --markets-repo flag

#6852 (comment)

[TippyFlitsUK]

Many thanks @tmyuu!!

I have these in my environment variables and they make things a great deal easier.

Appreciate the shout-out!!

[nonsense]

@TippyFlitsUK would you please share your repos' config.toml files? I see you are running a markets instance, and a proving/sealing instance, so I am wondering if the configs for these two are correct.

On the markets instance, you're supposed to have:

[Subsystems]
  EnableMining = false
  EnableSealing = false
  EnableSectorStorage = false
  EnableStorageMarket = true
  SealerApiInfo = "....."
  SectorIndexApiInfo = "....."

On the proving/sealing instance, you're supposed to have:

[Subsystems]
  EnableMining = true
  EnableSealing = true
  EnableSectorStorage = true
  EnableStorageMarket = false
  SealerApiInfo = ""
  SectorIndexApiInfo = ""

---

I believe that if you have MINER_API_INFO environment variable set, even though you call LOTUS_MINER_PATH=/mnt/nvme0n1/.lotusmarket lotus-miner storage-deals list, you are actually targeting the node specified in the MINER_API_INFO. So you might need to unset it.

[nonsense]

@TippyFlitsUK could you please share the full log from the markets miner log. It appears that the API is not at all running, so maybe the log will have some details on why that is. Thank you!

[TippyFlitsUK]

Sure @nonsense.

market.log

[TippyFlitsUK]

FYI I have run the market instance with a deliberately incorrect API token in the market config.toml and it gives a different error (below).
To me it looks like the API connection is made (with the correct miner API token) but things go wrong from there??

2021-07-26T17:31:47.438+0100	INFO	p2pnode	lp2p/addrs.go:114	Swarm listening at: [/ip4/127.0.0.1/tcp/4567 /ip4/192.168.1.50/tcp/4567]
2021-07-26T17:31:47.440+0100	INFO	modules	modules/storageminer_svc.go:68	Connecting storage service to miner
2021-07-26T17:31:47.440+0100	INFO	modules	modules/storageminer_svc.go:31	Checking (svc) api version of ws://127.0.0.1:2345/rpc/v0
ERROR: creating node: starting node: could not build arguments for function "github.com/filecoin-project/lotus/node/modules".HandleDeals (/home/tippyflits/lotus/node/modules/storageminer.go:288): failed to build storagemarket.StorageProvider: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:14): failed to build *storedask.StoredAsk: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:14): failed to build storagemarket.StorageProviderNode: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:14): failed to build *sectorblocks.SectorBlocks: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:14): failed to build sectorblocks.SectorBuilder: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:14): failed to build modules.MinerStorageService: received non-nil error from function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:14): websocket: bad handshake

[bq1024]

I‘m getting the same errors, running markets instance and proving/sealing on the same machine.

$ lotus-miner info
Chain: [sync ok] [basefee 106.799 pFIL]
Miner: f0165400 (64 GiB sectors)
Power: 111 Ti / 8.14 Ei (0.0013%)
	Raw: 109.8 TiB / 8.134 EiB (0.0013%)
	Committed: 109.9 TiB
	Proving: 109.8 TiB
Projected average block win rate: 1.31/week (every 128h6m30s)
Projected block win with 99.9% probability every 884h54m50s
(projections DO NOT account for future network and miner growth)

ERROR: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference

[nonsense]

@boqian-nbai could you confirm that you have MINER_API_INFO disabled?

1. Could you query lotus-miner sectors list on the proving/sealing instance?
2. Could you query lotus-miner storage-deals list on the markets instance?
3. Could you query lotus-miner net id on the markets instance?

[stuberman]

I started my migration on Phase III - m1.3.2

May I suggest that during the process of upgrading, instructions should clearly advise storage providers to disable all online and offline deals.

I also highly suggest that as part of the DAG store migration that some type of progress indicator be included such as:

566 deals of 1447 total deals migrated....

[tmyuu]

I am testing m1.3.3 1.11.1-rc1+mainnet+git.3be268182.
fstmpxxxxx transferred from client collects in $ LOTUS_MARKETS_HOME. This seems to be deleted after AddPiece. is there way to change a directory in which market node temporarily stores data these file?

[tmyuu]

Oh sorry, my writing was bad.
For example, $LOTUS_MARKETS_PATH/datastore/fstmp/fstmpxxxxx is less disorderly than $LOTUS_MARKETS_PATH/fstmpxxxxx even temporaly, and I can mount a new disk in the fstmp directory. I thought it would be preferable because of reducing to write data to the disk mounted on home directory of market miner. It has been receiving deals frequently and may make more in the future.

[ribasushi]

Addressing this is in progress by @frrist here: #6853

[tmyuu]

That would be nice.
Thank you!

[neondragon]

This is great, I have the same use case (wanting to mount a separate volume for fstmp files, to limit how much space large transient files are able to consume).

[frrist]

#6853 is just waiting on review if someone wants to give it a look.

[neondragon]

Attaching storage to the markets process via CLI didn't work as expected (lotus-miner --call-on-markets storage attach /path/to/storage)

It attached the storage location to the miner process.

I'd guess (not tested) that LOTUS_MINER_PATH="~/.lotusmarkets" lotus-miner storage attach /path/to/storage would do what I was trying to do. It is just a little cumbersome/confusing at the moment. A command exists, yet the docs suggest to use the command sometimes, and edit storage.json at other times.

https://docs.filecoin.io/mine/lotus/dagstore/#cli-commands
"If possible, configure your markets node storage.json to point to storage paths that are shared with your mining/sealing/storage node."

[tmyuu]

I'm on m1.3.5

Daemon:  1.11.1-m1.3.5+mainnet+git.3ff8e256b+api1.3.1
Local: lotus version 1.11.1-m1.3.5+mainnet+git.3ff8e256b

Initializing shard fails for a sector in which unsealed copy does not exist.
--include-sealed option might not be working properly.

2021-08-13T02:55:26.358+0900    INFO    node    impl/storminer.go:684   DagstoreInitializeAll: skipping piece because it's sealed       {"piece_cid": "baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby", "error": null}

When I specify key that state is under ShardStateNew with lotus-miner dagstore initialize-shard and try it again, an error occurs.

2021-08-13T03:04:32.350+0900	DEBUG	advmgr	sector-storage/piece_provider.go:83	did not get storage reader;sector={Miner:104671 Number:535}, err:failed to read sector {{104671 535} 8} from remote(1): sector not found
2021-08-13T03:04:32.350+0900	DEBUG	advmgr	sector-storage/piece_provider.go:110	result of first tryReadUnsealedPiece: r=<nil>, err=failed to read sector {{104671 535} 8} from remote(1): sector not found
2021-08-13T03:04:32.350+0900	DEBUG	advmgr	sector-storage/piece_provider.go:113	no unsealed sector file with unsealed piece, sector={ID:{Miner:104671 Number:535} ProofType:8}, offset=66584576, size=66584576
2021-08-13T03:04:32.366+0900	ERROR	advmgr	sector-storage/piece_provider.go:133	failed to SectorsUnsealPiece: worker UnsealPiece call: storage call error 0: unsealing sector: acquire unsealed sector path (existing): allocate local sector for fetching: couldn't find a suitable path for a sector
2021-08-13T03:04:32.367+0900	WARN	dagstore	dagstore/miner_api.go:155	failed to unseal deal 1935634: failed to unseal piece from sector 535: unsealing piece: worker UnsealPiece call: storage call error 0: unsealing sector: acquire unsealed sector path (existing): allocate local sector for fetching: couldn't find a suitable path for a sector
2021-08-13T03:04:32.367+0900	WARN	dagstore/upgrader	mount/upgrader.go:137	failed to refetch	{"shard": "baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby", "error": "failed to fetch and copy underlying mount to transient file: failed to fetch from underlying mount: failed to fetch unsealed piece baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby: failed to unseal deal 1935634: failed to unseal piece from sector 535: unsealing piece: worker UnsealPiece call: storage call error 0: unsealing sector: acquire unsealed sector path (existing): allocate local sector for fetching: couldn't find a suitable path for a sector"}
2021-08-13T03:04:32.367+0900	WARN	dagstore	dagstore@v0.4.2/dagstore_async.go:107	initialize: failed to fetch from mount upgrader	{"shard": "baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby", "error": "mount fetch failed: failed to fetch and copy underlying mount to transient file: failed to fetch from underlying mount: failed to fetch unsealed piece baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby: failed to unseal deal 1935634: failed to unseal piece from sector 535: unsealing piece: worker UnsealPiece call: storage call error 0: unsealing sector: acquire unsealed sector path (existing): allocate local sector for fetching: couldn't find a suitable path for a sector"}
2021-08-13T03:04:32.367+0900	WARN	rpc	go-jsonrpc@v0.1.4-0.20210217175800-45ea43ac2bec/handler.go:279	error in RPC call to 'Filecoin.DagstoreInitializeShard': failed to acquire shard: failed to acquire shard: failed to acquire reader of mount on initialization: mount fetch failed: failed to fetch and copy underlying mount to transient file: failed to fetch from underlying mount: failed to fetch unsealed piece baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby: failed to unseal deal 1935634: failed to unseal piece from sector 535: unsealing piece: worker UnsealPiece call: storage call error 0: unsealing sector: acquire unsealed sector path (existing): allocate local sector for fetching: couldn't find a suitable path for a sector
2021-08-13T03:04:32.367+0900	INFO	dagstore	dagstore@v0.4.2/handlers.go:36	failure handler: recovering shard	{"key": "baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby", "from_error": "failed to acquire reader of mount on initialization: mount fetch failed: failed to fetch and copy underlying mount to transient file: failed to fetch from underlying mount: failed to fetch unsealed piece baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby: failed to unseal deal 1935634: failed to unseal piece from sector 535: unsealing piece: worker UnsealPiece call: storage call error 0: unsealing sector: acquire unsealed sector path (existing): allocate local sector for fetching: couldn't find a suitable path for a sector", "attempt": 0}
2021-08-13T03:04:32.367+0900	WARN	dagstore	dagstore@v0.4.2/dagstore_control.go:282	recovery: failed to drop index for shard	{"shard": "baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby", "error": "remove /usr/local/filecoin/.market/dagstore/index/baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby.full.idx: no such file or directory"}
2021-08-13T03:04:32.367+0900	WARN	dagstore	dagstore@v0.4.2/dagstore_async.go:107	initialize: failed to fetch from mount upgrader	{"shard": "baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby", "error": "mount fetch failed: failed to fetch and copy underlying mount to transient file: failed to fetch from underlying mount: failed to fetch unsealed piece baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby: failed to unseal deal 1935634: failed to unseal piece from sector 535: unsealing piece: worker UnsealPiece call: storage call error 0: unsealing sector: acquire unsealed sector path (existing): allocate local sector for fetching: couldn't find a suitable path for a sector"}
2021-08-13T03:04:32.368+0900	WARN	dagstore	dagstore@v0.4.2/handlers.go:54	failure handler: failed to recover shard	{"key": "baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby", "attempt": 1}
2021-08-13T03:04:32.368+0900	INFO	dagstore	dagstore@v0.4.2/handlers.go:32	failure handler: max attempts exceeded; skipping recovery	{"key": "baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby", "from_error": "failed to acquire reader of mount on initialization: mount fetch failed: failed to fetch and copy underlying mount to transient file: failed to fetch from underlying mount: failed to fetch unsealed piece baga6ea4seaqjw3ohy2zrhtxpnpomutvp2555xc5v23kneeidcaummwxsoe52mby: failed to unseal deal 1935634: failed to unseal piece from sector 535: unsealing piece: worker UnsealPiece call: storage call error 0: unsealing sector: acquire unsealed sector path (existing): allocate local sector for fetching: couldn't find a suitable path for a sector", "attempt": 1}

No recovery from error is expected for sectors without copies.
lotus-miner dagstore recover-shard doesn't work.

[tmyuu]

It seems that unseal started in lotus-worker after lotus-miner dagstore initialize-all was completed.

[stuberman]

This is a general discussion on m1 and the deal making function of FileCoin based on my experiences last night. For the first time (running m1.3.5) I was able to see the system at full load and it was not pretty. This ended up being an 'all-nighter' as I attempted to resolve the core issue of high deal flow. I also encountered what appears to be a memory leak, but that is not the core issue.

Context: Running a dedicated market node m1.3.5 with bidbot (automated offline deals) and using CID-Gravity to manage deal prices and capacity. The sealing system has been able to process and seal up to 768 GiB/day of 32GiB sectors when clients arrange for large batches (100+) of offline deals. Download speed will be as good as 32GiB in 4-5 minutes. My verified deal prices is set to zero and my unverified prices is 0.000000012.

Assumption: Storage providers, Filecoin and clients require the maximum efficiency for the deal making pipeline, in order to as affordably as possible ingest deals at near hardware and network capacity. While offline deals can be planned, online deals are very unpredictable, necessitating mechanisms to help client systems manage deal flow such that they send valid deals in a manner that will not cause them to fail or timeout or be rejected for other reasons.

Problem #1 - Batch Published Deals (or PreCommit or Commit batches)
Batches should not fail when a single deal is not valid or expires prior to sealing. Last night I received three 32GiB bidbot deals and three Estuary deals (0.5 or 1GiB each) that were downloaded over the course of three hours and queued up for Batch Publishing. Upon the expiration of the batch publish timer, all six deals were discarded due to a problem (deal cap) with the Estuary deals. Not only is this a waste of precious resources, but the loss of good deal data is unfortunate and not something our clients (especially those with large data sets under time pressure) will find acceptable.

Problem #2 - Deal throttling required
Last night I received more than 100 Estuary (verified) smallish deals and 44 (verified) 32GiB deals in the course of 12 hours.
Three bidbot deals made it through sealing and one Estuary deal (and one random online deal). The rest failed.
The bidbot deals use an automated offline deal mechanism to download and then import the deals into Lotus. Bidbot has some local parameters such as "Maximum running total bytes in the deals to bid for a period of time."
The new markets node also has some throttling parameters such as "SimultaneousTransfers" and "ParallelFetchLimit" (for storage).
I am not familiar with any throttling for Estuary storage providers or other systems such as the DealBot.

As I monitored the market node last night I saw more than 100 fstmp* files in my TMPDIR and more than 60 transfers showing active in "Receiving Channels" when running lotus-miner data-transfers list just within 6 hours. Under the best of conditions that deal load would take about 12 hours to process and seal (my MaxSealingSectorsForDeals setting is '18'). Consider if this deal load continued without stopping this would amount to 48 sectors per day, more than double my sealing capacity. The result would be a large backlog of downloaded deals that would ultimately expire and fail due to start-epoch math. (In this case, a memory leak caused the market node to crash as the deal load grew and grew or after numerous restarts as the deal processes tried to catch up. I deleted all fstmp files and lowered various parameters to restrict deal flows. For instance CID-Gravity allows me to limit the number of deals (not bytes) per client per hour (however it currently does not allow me to limit the deal flow across all clients in aggregate).

The Challenge
How can we throttle incoming deals from a variety of client systems (DealBot/bidbot/Estuary/preplanned offline deals/random online clients) such that we are not accepting too many deals and overloading the markets nodes? While we can choose to upgrade and expand our capacities, the issue of deal flow pipeline capacity management is something that needs a good mechanism to control and manage. I know there are many new clients with pebibytes of data anxiously awaiting storage providers to properly and seamlessly store this data.

Note
Since sectors =! deals the current state of sector management which does work also does not apply to this issue.
We also have to look at the network and market node lode of retrievals as that is now in scope as well.

@jennijuju @raulk @Stefaan-V @whyrusleeping @jsign @TippyFlitsUK @jimmylee

[jennijuju]

Yah totally. I’m not saying that’s the only thing needed - but one of the things.
Other things can be like per client base filtering control or 1 miner: N markets node and so on.

[raulk]

Thank you for highlighting the painpoints, and also making highly useful suggestions!

On the one hand, I'm stoked that it's now possible to truly stress the system, once the many deal-making impediments and bottlenecks were removed. Those were acting as implicit guardrails because they were severely limiting the throughput of deal-making.

I'm even more satisfied that all that stress testing has not affected your mining/sealing pipelines. In a previous version of the world, you would've seen your mining/sealing activity interrupted due to the crashes.

That said, I 100% agree that miners are currently not well-equipped to deal with the new magnitudes of scale the software is now able to achieve. That includes traffic management, observability, and more.

Note that there is only so much that a single markets node can achieve. Next step in our roadmap is to enable miners to scale their deal-making activity by operating N markets node, orchestrated via Kubernetes or the like, elastically growing and shrinking their markets farm in response to real-time activity.

However, before we proceeded with that, we wanted to find out the breaking points of a single node, which we seem to be discovering now (although, admittedly, shadowed by the memory leak, tracked at #7077), thanks to the useful feedback from miners like you.

I'll look into the suggestions and see if we can prioritise for implementation.

[SBudo]

I totally agree with all the points made above.
We also had some deals going down the drain while doing a publishing aggregation. This needs to be addressed as soon as possible (or could we all agree to reduce the burn fee associated with publishing?).
Furthermore, I would also add, for the smaller miners, that a mechanism to check that there is sufficient FIL on the worker address before accepting deals would be absolutely awesome!

[jennijuju]

a mechanism to check that there is sufficient FIL on the worker address before accepting deals would be absolutely awesome!
@SBudo mind creating a feature request ticket?

could we all agree to reduce the burn fee associated with publishing?
will create an enhancement issue in actor for analyzing the gas consumption deduction possibilities. (cc @ZenGround0 )

[jennijuju]

This is a great thread that can use more attention from the lotus storage provider community. So created a dedicated discussion #7084 and transferring the points mentioned here in separate comments, so we can discuss accordingly and turn them into tickets and act on them.later.