Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] No longer possible to start with LOTUS_ENABLE_CHAINSTORE_FALLBACK=1 #5975

Closed
ribasushi opened this issue Apr 6, 2021 · 7 comments · Fixed by #6003
Closed

[BUG] No longer possible to start with LOTUS_ENABLE_CHAINSTORE_FALLBACK=1 #5975

ribasushi opened this issue Apr 6, 2021 · 7 comments · Fixed by #6003
Labels
area/chain Area: Chain kind/bug Kind: Bug P2 P2: Should be resolved

Comments

@ribasushi
Copy link
Collaborator

Describe the bug
Running LOTUS_ENABLE_CHAINSTORE_FALLBACK=1 lotus daemon crashes on startup

Version (run lotus version):
Stock v1.6.0

Logs

Complete log from startup to crash 
2021-04-06T13:36:45.537Z	INFO	main	lotus/daemon.go:214	lotus repo: /srv/lotus/.lotus
2021-04-06T13:36:45.537Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-2-b62098629d07946e9028127e70295ed996fe3ed25b0f9f88eb610a0ab4385a3c.vk is ok
2021-04-06T13:36:45.537Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-5294475db5237a2e83c3e52fd6c2b03859a1831d45ed08c4f35dbf9a803165a9.vk is ok
2021-04-06T13:36:45.537Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0170db1f394b35d995252228ee359194b13199d259380541dc529fb0099096b0.vk is ok
2021-04-06T13:36:45.538Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-ecd683648512ab1765faa2a5f14bab48f676e633467f0aa8aad4b55dcb0652bb.vk is ok
2021-04-06T13:36:45.538Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-3ea05428c9d11689f23529cde32fd30aabd50f7d2c93657c1d3650bca3e8ea9e.vk is ok
2021-04-06T13:36:45.538Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.vk is ok
2021-04-06T13:36:45.538Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-6babf46ce344ae495d558e7770a585b2382d54f225af8ed0397b8be7c3fcd472.vk is ok
2021-04-06T13:36:45.538Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-0-0-sha256_hasher-032d3138d22506ec0082ed72b2dcba18df18477904e35bafee82b3793b06832f.vk is ok
2021-04-06T13:36:45.537Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-559e581f022bb4e4ec6e719e563bf0e026ad6de42e56c18714a2c692b1b88d7e.vk is ok
2021-04-06T13:36:45.538Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-7d739b8cf60f1b0709eeebee7730e297683552e4b69cab6984ec0285663c5781.vk is ok
2021-04-06T13:36:45.538Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-0cfb4f178bbb71cf2ecfcd42accce558b27199ab4fb59cb78f2483fe21ef36d9.vk is ok
2021-04-06T13:36:45.538Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-2-sha256_hasher-96f1b4a04c5c51e4759bbf224bbc2ef5a42c7100f16ec0637123f16a845ddfb2.vk is ok
2021-04-06T13:36:45.537Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-0-0-50c7368dea9593ed0989e70974d28024efa9d156d585b7eea1be22b2e753f331.vk is ok
2021-04-06T13:36:45.545Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-0-0377ded656c6f524f1618760bffe4e0a1c51d5a70c4509eedae8a27555733edc.vk is ok
2021-04-06T13:36:45.545Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:138	Parameter file /srv/lotus/fil_caches/v28-proof-of-spacetime-fallback-merkletree-poseidon_hasher-8-8-2-2627e4006b67f99cef990c0a47d5426cb7ab0a0ad58fc1061547bf2d28b09def.vk is ok
2021-04-06T13:36:45.545Z	INFO	build	go-paramfetch@v0.0.2-0.20200701152213-3e0f0afdc261/paramfetch.go:162	parameter and key-fetching complete
2021-04-06T13:36:45.550Z	INFO	modules	modules/core.go:64	memory limits initialized	{"max_mem_heap": 0, "total_system_mem": 267082211328, "effective_mem_limit": 267082211328}
2021-04-06T13:36:45.551Z	WARN	modules	modules/core.go:124	failed to initialize cgroup-driven watchdog; err: failed to load memory cgroup stats: open /sys/fs/cgroup/memory/user.slice/user-1001.slice/session-5.scope/memory.memsw.usage_in_bytes: no such file or directory
2021-04-06T13:36:45.551Z	WARN	modules	modules/core.go:125	trying a system-driven watchdog
2021-04-06T13:36:45.551Z	INFO	watchdog	go-watchdog@v1.0.1/watermarks.go:16	initialized watermark watchdog policy; watermarks: []; thresholds: [133541105664 160249326796 186957547929 227019879628 240373990195 247051045478 253728100761]
2021-04-06T13:36:45.551Z	INFO	watchdog	go-watchdog@v1.0.1/watchdog.go:457	initialized heap profile capture; threshold: 0.900000; max captures: 10; dir: /srv/lotus/.lotus/heapprof
2021-04-06T13:36:45.551Z	INFO	modules	modules/core.go:130	initialized system-driven watchdog
2021-04-06T13:36:45.559Z	INFO	badger	v2@v2.2007.2/levels.go:183	All 0 tables opened in 0s

2021-04-06T13:36:45.559Z	INFO	badger	v2@v2.2007.2/value.go:1158	Replaying file id: 0 at offset: 0

2021-04-06T13:36:45.559Z	INFO	badger	v2@v2.2007.2/value.go:1178	Replay took: 2.314µs

2021-04-06T13:36:45.561Z	INFO	badger	v2@v2.2007.2/levels.go:183	All 1 tables opened in 0s

2021-04-06T13:36:45.561Z	INFO	badger	v2@v2.2007.2/value.go:1158	Replaying file id: 0 at offset: 49876

2021-04-06T13:36:45.561Z	INFO	badger	v2@v2.2007.2/value.go:1178	Replay took: 18.757µs

2021-04-06T13:36:45.561Z	INFO	backupds	backupds/log.go:123	opening log	{"file": "/srv/lotus/.lotus/kvlog/metadata/1617462230.log.cbor"}
2021-04-06T13:36:45.976Z	INFO	backupds	backupds/log.go:174	log opened	{"file": "/srv/lotus/.lotus/kvlog/metadata/1617462230.log.cbor", "openCount": 2, "logValues": 85043}
2021-04-06T13:36:45.977Z	INFO	backupds	backupds/log.go:199	new log head	{"loghead": "1617462230.log.cbor;f1ccf2fe-ce23-4a0f-955b-c4f6882a2db0;1617716205"}
runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0x402e9e0420 stack=[0x402e9e0000, 0x404e9e0000]
fatal error: stack overflow

runtime stack:
runtime.throw(0x2a2243d, 0xe)
	/usr/local/go/src/runtime/panic.go:1116 +0x54
runtime.newstack()
	/usr/local/go/src/runtime/stack.go:1067 +0x600
runtime.morestack()
	/usr/local/go/src/runtime/asm_arm64.s:308 +0x70

goroutine 1 [running]:
go.uber.org/dig.shallowCheckDependencies.func1(0x2e75020, 0x4001207440, 0xbdcb3fd31f4609b7)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:743 +0x43c fp=0x402e9e0420 sp=0x402e9e0420 pc=0x11a8b3c
go.uber.org/dig.paramVisitorFunc.Visit(0x400e429830, 0x2e75020, 0x4001207440, 0x0, 0x0)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:102 +0x38 fp=0x402e9e0450 sp=0x402e9e0420 pc=0x11a1f78
go.uber.org/dig.walkParam(0x2e75020, 0x4001207440, 0x2e434e0, 0x400e429830)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:117 +0x48 fp=0x402e9e0570 sp=0x402e9e0450 pc=0x11a2008
go.uber.org/dig.walkParam(0x2e74fa0, 0x400e429800, 0x2e434e0, 0x400e429830)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:131 +0x110 fp=0x402e9e0690 sp=0x402e9e0570 pc=0x11a20d0
go.uber.org/dig.shallowCheckDependencies(0x2e97ec0, 0x40005da3c0, 0x2e74fa0, 0x400e429800, 0x0, 0x40005da301)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:743 +0xc0 fp=0x402e9e06d0 sp=0x402e9e0690 pc=0x119dfc0
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:694 +0x60 fp=0x402e9e0850 sp=0x402e9e06d0 pc=0x119d7a0
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e0980 sp=0x402e9e0850 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13780, 0x1, 0x400e4297d0, 0x402e9e0a98, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e0a20 sp=0x402e9e0980 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e0b18, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e0aa0 sp=0x402e9e0a20 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e0c20 sp=0x402e9e0aa0 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e0d50 sp=0x402e9e0c20 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13700, 0x1, 0x400e429770, 0x402e9e0e68, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e0df0 sp=0x402e9e0d50 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e0ee8, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e0e70 sp=0x402e9e0df0 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e0ff0 sp=0x402e9e0e70 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e1120 sp=0x402e9e0ff0 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13660, 0x1, 0x400e429710, 0x402e9e1238, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e11c0 sp=0x402e9e1120 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e12b8, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e1240 sp=0x402e9e11c0 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e13c0 sp=0x402e9e1240 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e14f0 sp=0x402e9e13c0 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a135e0, 0x1, 0x400e4296b0, 0x402e9e1608, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e1590 sp=0x402e9e14f0 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e1688, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e1610 sp=0x402e9e1590 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e1790 sp=0x402e9e1610 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e18c0 sp=0x402e9e1790 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13540, 0x1, 0x400e429650, 0x402e9e19d8, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e1960 sp=0x402e9e18c0 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e1a58, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e19e0 sp=0x402e9e1960 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e1b60 sp=0x402e9e19e0 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e1c90 sp=0x402e9e1b60 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a134c0, 0x1, 0x400e4295f0, 0x402e9e1da8, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e1d30 sp=0x402e9e1c90 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e1e28, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e1db0 sp=0x402e9e1d30 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e1f30 sp=0x402e9e1db0 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e2060 sp=0x402e9e1f30 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13420, 0x1, 0x400e429590, 0x402e9e2178, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e2100 sp=0x402e9e2060 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e21f8, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e2180 sp=0x402e9e2100 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e2300 sp=0x402e9e2180 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e2430 sp=0x402e9e2300 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a133a0, 0x1, 0x400e429530, 0x402e9e2548, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e24d0 sp=0x402e9e2430 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e25c8, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e2550 sp=0x402e9e24d0 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e26d0 sp=0x402e9e2550 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e2800 sp=0x402e9e26d0 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13300, 0x1, 0x400e4294d0, 0x402e9e2918, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e28a0 sp=0x402e9e2800 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e2998, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e2920 sp=0x402e9e28a0 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e2aa0 sp=0x402e9e2920 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e2bd0 sp=0x402e9e2aa0 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13280, 0x1, 0x400e429470, 0x402e9e2ce8, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e2c70 sp=0x402e9e2bd0 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e2d68, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e2cf0 sp=0x402e9e2c70 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e2e70 sp=0x402e9e2cf0 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e2fa0 sp=0x402e9e2e70 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a131e0, 0x1, 0x400e429410, 0x402e9e30b8, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e3040 sp=0x402e9e2fa0 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e3138, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e30c0 sp=0x402e9e3040 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e3240 sp=0x402e9e30c0 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e3370 sp=0x402e9e3240 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13160, 0x1, 0x400e4293b0, 0x402e9e3488, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e3410 sp=0x402e9e3370 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e3508, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e3490 sp=0x402e9e3410 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e3610 sp=0x402e9e3490 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e3740 sp=0x402e9e3610 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a130c0, 0x1, 0x400e429350, 0x402e9e3858, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e37e0 sp=0x402e9e3740 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e38d8, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e3860 sp=0x402e9e37e0 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e39e0 sp=0x402e9e3860 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e3b10 sp=0x402e9e39e0 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a13040, 0x1, 0x400e4292f0, 0x402e9e3c28, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e3bb0 sp=0x402e9e3b10 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e3ca8, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e3c30 sp=0x402e9e3bb0 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e3db0 sp=0x402e9e3c30 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e3ee0 sp=0x402e9e3db0 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12fa0, 0x1, 0x400e429290, 0x402e9e3ff8, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e3f80 sp=0x402e9e3ee0 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e4078, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e4000 sp=0x402e9e3f80 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e4180 sp=0x402e9e4000 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e42b0 sp=0x402e9e4180 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12f20, 0x1, 0x400e429230, 0x402e9e43c8, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e4350 sp=0x402e9e42b0 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e4448, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e43d0 sp=0x402e9e4350 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e4550 sp=0x402e9e43d0 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e4680 sp=0x402e9e4550 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12e80, 0x1, 0x400e4291d0, 0x402e9e4798, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e4720 sp=0x402e9e4680 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e4818, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e47a0 sp=0x402e9e4720 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e4920 sp=0x402e9e47a0 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e4a50 sp=0x402e9e4920 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12e00, 0x1, 0x400e429170, 0x402e9e4b68, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e4af0 sp=0x402e9e4a50 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e4be8, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e4b70 sp=0x402e9e4af0 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e4cf0 sp=0x402e9e4b70 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e4e20 sp=0x402e9e4cf0 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12d60, 0x1, 0x400e429110, 0x402e9e4f38, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e4ec0 sp=0x402e9e4e20 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e4fb8, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e4f40 sp=0x402e9e4ec0 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e50c0 sp=0x402e9e4f40 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e51f0 sp=0x402e9e50c0 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12ce0, 0x1, 0x400e4290b0, 0x402e9e5308, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e5290 sp=0x402e9e51f0 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e5388, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e5310 sp=0x402e9e5290 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e5490 sp=0x402e9e5310 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e55c0 sp=0x402e9e5490 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12c40, 0x1, 0x400e429050, 0x402e9e56d8, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e5660 sp=0x402e9e55c0 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e5758, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e56e0 sp=0x402e9e5660 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e5860 sp=0x402e9e56e0 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e5990 sp=0x402e9e5860 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12bc0, 0x1, 0x400e428ff0, 0x402e9e5aa8, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e5a30 sp=0x402e9e5990 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e5b28, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e5ab0 sp=0x402e9e5a30 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e5c30 sp=0x402e9e5ab0 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e5d60 sp=0x402e9e5c30 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12b20, 0x1, 0x400e428f90, 0x402e9e5e78, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e5e00 sp=0x402e9e5d60 pc=0x11ac734
go.uber.org/dig.paramList.BuildList(0x2ead000, 0x25f8160, 0x4001065890, 0x1, 0x1, 0x2e97ec0, 0x40005da3c0, 0x119b79c, 0x402e9e5ef8, 0x119b7c8, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:201 +0x98 fp=0x402e9e5e80 sp=0x402e9e5e00 pc=0x11a2808
go.uber.org/dig.(*node).Call(0x40000f1360, 0x2e97ec0, 0x40005da3c0, 0x2ead000, 0x27d1820)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/dig.go:701 +0xcc fp=0x402e9e6000 sp=0x402e9e5e80 pc=0x119d80c
go.uber.org/dig.paramSingle.Build(0x0, 0x0, 0x0, 0x2ead000, 0x27d1820, 0x2e97ec0, 0x40005da3c0, 0x25f8160, 0x4001065890, 0x1, ...)
	/srv/lotus/go/pkg/mod/go.uber.org/dig@v1.10.0/param.go:245 +0x210 fp=0x402e9e6130 sp=0x402e9e6000 pc=0x11a2c20
go.uber.org/dig.(*paramSingle).Build(0x4001207440, 0x2e97ec0, 0x40005da3c0, 0x4012a12aa0, 0x1, 0x400e428f30, 0x402e9e6248, 0x119d7a0)
	<autogenerated>:1 +0x84 fp=0x402e9e61d0 sp=0x402e9e6130 pc=0x11ac734
...additional frames elided...

goroutine 6 [select]:
go.opencensus.io/stats/view.(*worker).start(0x4000184300)
	/srv/lotus/go/pkg/mod/go.opencensus.io@v0.22.5/stats/view/worker.go:276 +0xb4
created by go.opencensus.io/stats/view.init.0
	/srv/lotus/go/pkg/mod/go.opencensus.io@v0.22.5/stats/view/worker.go:34 +0x70

goroutine 66 [select]:
github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0x40002ba0c0)
	/srv/lotus/go/pkg/mod/github.com/ipfs/go-log@v1.0.4/writer/writer.go:71 +0xd0
created by github.com/ipfs/go-log/writer.NewMirrorWriter
	/srv/lotus/go/pkg/mod/github.com/ipfs/go-log@v1.0.4/writer/writer.go:36 +0xac

goroutine 98 [select]:
github.com/libp2p/go-libp2p-connmgr.(*decayer).process(0x40014a1180)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-connmgr@v0.2.4/decay.go:165 +0x1a4
created by github.com/libp2p/go-libp2p-connmgr.NewDecayer
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-connmgr@v0.2.4/decay.go:95 +0x1d8

goroutine 69 [chan receive]:
github.com/filecoin-project/lotus/cli/util.ReqContext.func1(0x4001048b40, 0x40010641f0)
	/srv/lotus/lotus/cli/util/api.go:268 +0x2c
created by github.com/filecoin-project/lotus/cli/util.ReqContext
	/srv/lotus/lotus/cli/util/api.go:267 +0xe0

goroutine 71 [syscall]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:147 +0xc4
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x20
created by os/signal.Notify.func1.1
	/usr/local/go/src/os/signal/signal.go:150 +0x44

goroutine 354 [IO wait]:
internal/poll.runtime_pollWait(0xffff80a8e3b8, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:222 +0x44
internal/poll.(*pollDesc).wait(0x4000760018, 0x72, 0x0, 0x10, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x38
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).ReadFrom(0x4000760000, 0x400075c030, 0x10, 0x10, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:215 +0x15c
net.(*netFD).readFrom(0x4000760000, 0x400075c030, 0x10, 0x10, 0x4000760000, 0x2c6cfe0, 0x40010a4d38, 0x682f24, 0x4000760000)
	/usr/local/go/src/net/fd_posix.go:61 +0x44
net.(*UDPConn).readFrom(0x4000618010, 0x400075c030, 0x10, 0x10, 0x2, 0x2, 0x2, 0x0)
	/usr/local/go/src/net/udpsock_posix.go:47 +0x44
net.(*UDPConn).ReadFromUDP(0x4000618010, 0x400075c030, 0x10, 0x10, 0x2, 0x0, 0x0, 0x85623f3b)
	/usr/local/go/src/net/udpsock.go:106 +0x48
github.com/jackpal/go-nat-pmp.(*network).call(0x40005ea060, 0x400075c000, 0x2, 0x2, 0x0, 0x400075c030, 0x10, 0x10, 0x0, 0x0)
	/srv/lotus/go/pkg/mod/github.com/jackpal/go-nat-pmp@v1.0.2/network.go:54 +0x208
github.com/jackpal/go-nat-pmp.(*Client).rpc(0x40005ea0c0, 0x400075c000, 0x2, 0x2, 0xc, 0x0, 0x0, 0x1, 0x4000129f48, 0x141b878)
	/srv/lotus/go/pkg/mod/github.com/jackpal/go-nat-pmp@v1.0.2/natpmp.go:108 +0x54
github.com/jackpal/go-nat-pmp.(*Client).GetExternalAddress(0x40005ea0c0, 0x40005ea0c0, 0x40019e2480, 0x0)
	/srv/lotus/go/pkg/mod/github.com/jackpal/go-nat-pmp@v1.0.2/natpmp.go:58 +0x64
github.com/libp2p/go-nat.discoverNATPMPWithAddr.func1(0x40014ca240, 0x400e8ac038, 0x4, 0x6c8)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/natpmp.go:44 +0xd0
created by github.com/libp2p/go-nat.discoverNATPMPWithAddr
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/natpmp.go:41 +0x6c

goroutine 179 [chan receive]:
github.com/dgraph-io/badger/v2.(*valueLog).waitOnGC(0x40014815d0, 0x4001056020)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1851 +0x58
created by github.com/dgraph-io/badger/v2.Open
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:415 +0xd08

goroutine 93 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0x400122c680)
	/srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x1a8
created by github.com/syndtr/goleveldb/leveldb.openDB
	/srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x440

goroutine 91 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0x400122c680)
	/srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0x88
created by github.com/syndtr/goleveldb/leveldb.openDB
	/srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x318

goroutine 92 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0x400122c680)
	/srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0xb4
created by github.com/syndtr/goleveldb/leveldb.openDB
	/srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x338

goroutine 33 [select]:
github.com/libp2p/go-libp2p-peerstore/pstoremem.(*memoryAddrBook).background(0x4000301b00)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.2.6/pstoremem/addr_book.go:93 +0xe4
created by github.com/libp2p/go-libp2p-peerstore/pstoremem.NewAddrBook
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.2.6/pstoremem/addr_book.go:83 +0x164

goroutine 32 [select]:
github.com/raulk/go-watchdog.pollingWatchdog(0x2e3fdc0, 0x400120af00, 0x12a05f200, 0x3e2f572000, 0x4000735410)
	/srv/lotus/go/pkg/mod/github.com/raulk/go-watchdog@v1.0.1/watchdog.go:320 +0x1ac
created by github.com/raulk/go-watchdog.SystemDriven
	/srv/lotus/go/pkg/mod/github.com/raulk/go-watchdog@v1.0.1/watchdog.go:268 +0x1a0

goroutine 99 [select]:
github.com/libp2p/go-libp2p-connmgr.(*BasicConnMgr).background(0x4000302d00)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-connmgr@v0.2.4/connmgr.go:246 +0x108
created by github.com/libp2p/go-libp2p-connmgr.NewConnManager
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-connmgr@v0.2.4/connmgr.go:140 +0x25c

goroutine 101 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0x40006e3880)
	/srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0xe0
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
	/srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x154

goroutine 94 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0x400122c680)
	/srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0xe4
created by github.com/syndtr/goleveldb/leveldb.openDB
	/srv/lotus/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x460

goroutine 156 [select]:
github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0x400123fe00, 0x400000fb80)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:212 +0x1dc
created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:70 +0x78

goroutine 157 [select]:
github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0x400123fe30, 0x400000fb80)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:212 +0x1dc
created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:70 +0x78

goroutine 158 [select]:
github.com/dgraph-io/badger/v2.(*DB).updateSize(0x4001481400, 0x400000fc60)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:1200 +0x108
created by github.com/dgraph-io/badger/v2.Open
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:360 +0x724

goroutine 162 [select]:
github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0x4001030070, 0x0, 0x40014942c0)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:390 +0x1fc
created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:371 +0x80

goroutine 163 [select]:
github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0x4001030070, 0x1, 0x40014942c0)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:390 +0x1fc
created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:371 +0x80

goroutine 164 [chan receive]:
github.com/dgraph-io/badger/v2.(*DB).flushMemtable(0x4001481400, 0x40014942e0, 0x0, 0x0)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:1107 +0x78
github.com/dgraph-io/badger/v2.Open.func5(0x40001320e8)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:377 +0x34
created by github.com/dgraph-io/badger/v2.Open
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:376 +0xf50

goroutine 178 [select]:
github.com/dgraph-io/badger/v2.(*DB).doWrites(0x4001481400, 0x4001056000)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:862 +0x2fc
created by github.com/dgraph-io/badger/v2.Open
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:411 +0xb44

goroutine 166 [select]:
github.com/dgraph-io/badger/v2.(*valueLog).flushDiscardStats(0x40014815d0)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1948 +0x108
created by github.com/dgraph-io/badger/v2.(*valueLog).open
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1114 +0x50

goroutine 180 [select]:
github.com/dgraph-io/badger/v2.(*publisher).listenForUpdates(0x400123fec0, 0x4001056040)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/publisher.go:67 +0xfc
created by github.com/dgraph-io/badger/v2.Open
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:419 +0xc04

goroutine 102 [select]:
github.com/ipfs/go-ds-badger2.(*Datastore).periodicGC(0x4001962120)
	/srv/lotus/go/pkg/mod/github.com/ipfs/go-ds-badger2@v0.1.1-0.20200708190120-187fc06f714e/datastore.go:172 +0xc0
created by github.com/ipfs/go-ds-badger2.NewDatastore
	/srv/lotus/go/pkg/mod/github.com/ipfs/go-ds-badger2@v0.1.1-0.20200708190120-187fc06f714e/datastore.go:160 +0x4e4

goroutine 228 [select]:
github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0x40011588a0, 0x40006fc540)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:212 +0x1dc
created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:70 +0x78

goroutine 229 [select]:
github.com/dgraph-io/badger/v2/y.(*WaterMark).process(0x40011588d0, 0x40006fc540)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:212 +0x1dc
created by github.com/dgraph-io/badger/v2/y.(*WaterMark).Init
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/y/watermark.go:70 +0x78

goroutine 230 [select]:
github.com/dgraph-io/badger/v2.(*DB).updateSize(0x4000700c00, 0x40006fc5e0)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:1200 +0x108
created by github.com/dgraph-io/badger/v2.Open
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:360 +0x724

goroutine 232 [select]:
github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0x40000fe8c0, 0x0, 0x400074e5e0)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:390 +0x1fc
created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:371 +0x80

goroutine 233 [select]:
github.com/dgraph-io/badger/v2.(*levelsController).runCompactor(0x40000fe8c0, 0x1, 0x400074e5e0)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:390 +0x1fc
created by github.com/dgraph-io/badger/v2.(*levelsController).startCompact
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/levels.go:371 +0x80

goroutine 234 [chan receive]:
github.com/dgraph-io/badger/v2.(*DB).flushMemtable(0x4000700c00, 0x400074e600, 0x0, 0x0)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:1107 +0x78
github.com/dgraph-io/badger/v2.Open.func5(0x40002da010)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:377 +0x34
created by github.com/dgraph-io/badger/v2.Open
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:376 +0xf50

goroutine 237 [select]:
github.com/dgraph-io/badger/v2.(*DB).doWrites(0x4000700c00, 0x400074e760)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:862 +0x2fc
created by github.com/dgraph-io/badger/v2.Open
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:411 +0xb44

goroutine 236 [select]:
github.com/dgraph-io/badger/v2.(*valueLog).flushDiscardStats(0x4000700dd0)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1948 +0x108
created by github.com/dgraph-io/badger/v2.(*valueLog).open
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1114 +0x50

goroutine 238 [chan receive]:
github.com/dgraph-io/badger/v2.(*valueLog).waitOnGC(0x4000700dd0, 0x400074e780)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/value.go:1851 +0x58
created by github.com/dgraph-io/badger/v2.Open
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:415 +0xd08

goroutine 239 [select]:
github.com/dgraph-io/badger/v2.(*publisher).listenForUpdates(0x4001158960, 0x400074e7a0)
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/publisher.go:67 +0xfc
created by github.com/dgraph-io/badger/v2.Open
	/srv/lotus/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.2007.2/db.go:419 +0xc04

goroutine 240 [select]:
github.com/ipfs/go-ds-badger2.(*Datastore).periodicGC(0x40019f9380)
	/srv/lotus/go/pkg/mod/github.com/ipfs/go-ds-badger2@v0.1.1-0.20200708190120-187fc06f714e/datastore.go:172 +0xc0
created by github.com/ipfs/go-ds-badger2.NewDatastore
	/srv/lotus/go/pkg/mod/github.com/ipfs/go-ds-badger2@v0.1.1-0.20200708190120-187fc06f714e/datastore.go:160 +0x4e4

goroutine 289 [select]:
github.com/libp2p/go-libp2p/p2p/protocol/identify.(*ObservedAddrManager).worker(0x4001f17980, 0x2e7df60, 0x4000641a40)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p@v0.12.0/p2p/protocol/identify/obsaddr.go:236 +0x1a8
created by github.com/libp2p/go-libp2p/p2p/protocol/identify.NewObservedAddrManager
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p@v0.12.0/p2p/protocol/identify/obsaddr.go:127 +0x16c

goroutine 290 [select]:
github.com/filecoin-project/lotus/lib/backupds.(*Datastore).runLog(0x4000640a00, 0x4001f18008)
	/srv/lotus/lotus/lib/backupds/log.go:76 +0xc4
created by github.com/filecoin-project/lotus/lib/backupds.(*Datastore).startLog
	/srv/lotus/lotus/lib/backupds/log.go:68 +0x3b8

goroutine 318 [select]:
github.com/libp2p/go-nat.discoverNATPMP.func1(0x400e88b620, 0x400e8ac038, 0x4, 0x6c8, 0x2e7dfe0, 0x400e88b560)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/natpmp.go:28 +0xdc
created by github.com/libp2p/go-nat.discoverNATPMP
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/natpmp.go:23 +0xa0

goroutine 288 [select]:
github.com/libp2p/go-libp2p-swarm.(*DialBackoff).background(0x4000703790, 0x2e7df60, 0x4000640cc0)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.3.1/swarm_dial.go:126 +0xd8
created by github.com/libp2p/go-libp2p-swarm.(*DialBackoff).init
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.3.1/swarm_dial.go:119 +0x54

goroutine 315 [select]:
github.com/libp2p/go-nat.DiscoverNATs.func1(0x40000d94a0, 0x2e7dfe0, 0x400e88b560)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/nat.go:54 +0x1c0
created by github.com/libp2p/go-nat.DiscoverNATs
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/nat.go:42 +0x60

goroutine 286 [select]:
github.com/jbenet/goprocess/context.CloseAfterContext.func1(0x2e7df60, 0x4000640c80, 0x2e9bf40, 0x4001f17800)
	/srv/lotus/go/pkg/mod/github.com/jbenet/goprocess@v0.1.4/context/context.go:65 +0xa0
created by github.com/jbenet/goprocess/context.CloseAfterContext
	/srv/lotus/go/pkg/mod/github.com/jbenet/goprocess@v0.1.4/context/context.go:64 +0xa4

goroutine 316 [IO wait]:
internal/poll.runtime_pollWait(0xffff80a8e670, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:222 +0x44
internal/poll.(*pollDesc).wait(0x4000138218, 0x72, 0x800, 0x800, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x38
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).ReadFrom(0x4000138200, 0x4000c3c000, 0x800, 0x800, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:215 +0x15c
net.(*netFD).readFrom(0x4000138200, 0x4000c3c000, 0x800, 0x800, 0x4000c3c000, 0xffff800c52d0, 0x4000138201, 0x400e98e000, 0x1)
	/usr/local/go/src/net/fd_posix.go:61 +0x44
net.(*UDPConn).readFrom(0x40002da1b0, 0x4000c3c000, 0x800, 0x800, 0x400e9849b8, 0x13f84dc, 0x400e9849b8, 0x13f8694)
	/usr/local/go/src/net/udpsock_posix.go:47 +0x44
net.(*UDPConn).ReadFrom(0x40002da1b0, 0x4000c3c000, 0x800, 0x800, 0x2e659a0, 0x4001058150, 0x87, 0x0, 0x0)
	/usr/local/go/src/net/udpsock.go:118 +0x48
github.com/huin/goupnp/httpu.(*HTTPUClient).Do(0x40006fdd00, 0x400e988000, 0x7d2b7500, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0)
	/srv/lotus/go/pkg/mod/github.com/huin/goupnp@v1.0.0/httpu/httpu.go:107 +0x4bc
github.com/huin/goupnp/ssdp.SSDPRawSearch(0x40006fdd00, 0x2b0d083, 0x31, 0x2, 0x3, 0x4001030a10, 0x0, 0x0, 0x0, 0x0)
	/srv/lotus/go/pkg/mod/github.com/huin/goupnp@v1.0.0/ssdp/ssdp.go:58 +0x3b4
github.com/huin/goupnp.DiscoverDevices(0x2b0d083, 0x31, 0x0, 0x0, 0x0, 0x0, 0x0)
	/srv/lotus/go/pkg/mod/github.com/huin/goupnp@v1.0.0/goupnp.go:66 +0x88
github.com/libp2p/go-nat.discoverUPNP_IG1.func1(0x40000d9500, 0x2e7dfe0, 0x400e88b560)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/upnp.go:27 +0x5c
created by github.com/libp2p/go-nat.discoverUPNP_IG1
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/upnp.go:23 +0x60

goroutine 317 [IO wait]:
internal/poll.runtime_pollWait(0xffff80a8e588, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:222 +0x44
internal/poll.(*pollDesc).wait(0x4000184018, 0x72, 0x800, 0x800, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x38
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).ReadFrom(0x4000184000, 0x400e8ce000, 0x800, 0x800, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:215 +0x15c
net.(*netFD).readFrom(0x4000184000, 0x400e8ce000, 0x800, 0x800, 0x400e8ce000, 0xffff48159170, 0x4000184001, 0x400dabe000, 0x1)
	/usr/local/go/src/net/fd_posix.go:61 +0x44
net.(*UDPConn).readFrom(0x40005fa080, 0x400e8ce000, 0x800, 0x800, 0x400e9809b8, 0x13f84dc, 0x400e9809b8, 0x13f8694)
	/usr/local/go/src/net/udpsock_posix.go:47 +0x44
net.(*UDPConn).ReadFrom(0x40005fa080, 0x400e8ce000, 0x800, 0x800, 0x2e659a0, 0x400daaf020, 0x87, 0x0, 0x0)
	/usr/local/go/src/net/udpsock.go:118 +0x48
github.com/huin/goupnp/httpu.(*HTTPUClient).Do(0x4001494460, 0x400dab6000, 0x7d2b7500, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0)
	/srv/lotus/go/pkg/mod/github.com/huin/goupnp@v1.0.0/httpu/httpu.go:107 +0x4bc
github.com/huin/goupnp/ssdp.SSDPRawSearch(0x4001494460, 0x2b0d0b4, 0x31, 0x2, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0)
	/srv/lotus/go/pkg/mod/github.com/huin/goupnp@v1.0.0/ssdp/ssdp.go:58 +0x3b4
github.com/huin/goupnp.DiscoverDevices(0x2b0d0b4, 0x31, 0x0, 0x0, 0x0, 0x0, 0x0)
	/srv/lotus/go/pkg/mod/github.com/huin/goupnp@v1.0.0/goupnp.go:66 +0x88
github.com/libp2p/go-nat.discoverUPNP_IG2.func1(0x40000d9560, 0x2e7dfe0, 0x400e88b560)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/upnp.go:84 +0x5c
created by github.com/libp2p/go-nat.discoverUPNP_IG2
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/upnp.go:80 +0x60

goroutine 338 [select]:
github.com/libp2p/go-libp2p/p2p/protocol/identify.(*IDService).loop(0x400033d6c0)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p@v0.12.0/p2p/protocol/identify/id.go:203 +0x39c
created by github.com/libp2p/go-libp2p/p2p/protocol/identify.NewIDService
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p@v0.12.0/p2p/protocol/identify/id.go:150 +0x250

goroutine 314 [chan receive]:
github.com/libp2p/go-nat.DiscoverGateway(0x0, 0x0, 0x0, 0x0)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/nat.go:93 +0xcc
github.com/libp2p/go-libp2p-nat.DiscoverNAT.func1(0x40000d93e0, 0x40006a2800, 0x40006a2810)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-nat@v0.0.6/nat.go:42 +0x48
created by github.com/libp2p/go-libp2p-nat.DiscoverNAT
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-nat@v0.0.6/nat.go:39 +0x98

goroutine 341 [select]:
github.com/libp2p/go-libp2p-nat.DiscoverNAT(0x2e7df60, 0x400e872480, 0x2e9bf40, 0x4001f17a40, 0x2e7df60)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-nat@v0.0.6/nat.go:45 +0x100
github.com/libp2p/go-libp2p/p2p/host/basic.(*natManager).start.func1(0x2e9bf40, 0x4001f17a40)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p@v0.12.0/p2p/host/basic/natmgr.go:89 +0x90
github.com/jbenet/goprocess.(*process).Go.func1(0x400110d050, 0x4001f17a40, 0x4001f17aa0)
	/srv/lotus/go/pkg/mod/github.com/jbenet/goprocess@v0.1.4/impl-mutex.go:134 +0x3c
created by github.com/jbenet/goprocess.(*process).Go
	/srv/lotus/go/pkg/mod/github.com/jbenet/goprocess@v0.1.4/impl-mutex.go:133 +0x1bc

goroutine 319 [IO wait]:
internal/poll.runtime_pollWait(0xffff80a8e4a0, 0x72, 0x40006a2d10)
	/usr/local/go/src/runtime/netpoll.go:222 +0x44
internal/poll.(*pollDesc).wait(0x40014a1798, 0x72, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x38
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).RawRead(0x40014a1780, 0x400e8776e0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:533 +0xe0
net.(*rawConn).Read(0x4001f181c0, 0x400e8776e0, 0x1, 0x1)
	/usr/local/go/src/net/rawconn.go:43 +0x60
golang.org/x/net/internal/socket.(*Conn).recvMsg(0x4001056c60, 0x40014c9c60, 0x0, 0x0, 0x40014c9c68)
	/srv/lotus/go/pkg/mod/golang.org/x/net@v0.0.0-20201021035429-f5854403a974/internal/socket/rawconn_msg.go:32 +0x1ac
golang.org/x/net/internal/socket.(*Conn).RecvMsg(...)
	/srv/lotus/go/pkg/mod/golang.org/x/net@v0.0.0-20201021035429-f5854403a974/internal/socket/socket.go:247
golang.org/x/net/ipv4.(*payloadHandler).ReadFrom(0x4001f070a0, 0x400e8be000, 0xffff, 0xffff, 0x0, 0x0, 0x560b400, 0x2329, 0x400e86cb20, 0x4)
	/srv/lotus/go/pkg/mod/golang.org/x/net@v0.0.0-20201021035429-f5854403a974/ipv4/payload_cmsg.go:31 +0x178
github.com/koron/go-ssdp.(*multicastConn).readPackets(0x400e8775c0, 0x12a05f200, 0x40014c9dc0, 0x96, 0x2e659a0)
	/srv/lotus/go/pkg/mod/github.com/koron/go-ssdp@v0.0.0-20191105050749-2e1c40ed0b5d/multicast.go:103 +0x7c
github.com/koron/go-ssdp.Search(0x29d9a02, 0x8, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/srv/lotus/go/pkg/mod/github.com/koron/go-ssdp@v0.0.0-20191105050749-2e1c40ed0b5d/search.go:100 +0x1a8
github.com/libp2p/go-nat.discoverUPNP_GenIGDev.func1(0x400e88b680, 0x2e7dfe0, 0x400e88b560)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/upnp.go:154 +0x70
created by github.com/libp2p/go-nat.discoverUPNP_GenIGDev
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-nat@v0.0.5/upnp.go:151 +0x64

goroutine 342 [select]:
github.com/libp2p/go-libp2p-peerstore/pstoremem.(*memoryAddrBook).background(0x4000301200)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.2.6/pstoremem/addr_book.go:93 +0xe4
created by github.com/libp2p/go-libp2p-peerstore/pstoremem.NewAddrBook
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-peerstore@v0.2.6/pstoremem/addr_book.go:83 +0x164

goroutine 343 [select]:
github.com/jbenet/goprocess/context.CloseAfterContext.func1(0x2e7df60, 0x4000640cc0, 0x2e9bf40, 0x4001f17c80)
	/srv/lotus/go/pkg/mod/github.com/jbenet/goprocess@v0.1.4/context/context.go:65 +0xa0
created by github.com/jbenet/goprocess/context.CloseAfterContext
	/srv/lotus/go/pkg/mod/github.com/jbenet/goprocess@v0.1.4/context/context.go:64 +0xa4

goroutine 345 [select]:
github.com/libp2p/go-libp2p-swarm.(*DialBackoff).background(0x400130ab90, 0x2e7df60, 0x4000641c40)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.3.1/swarm_dial.go:126 +0xd8
created by github.com/libp2p/go-libp2p-swarm.(*DialBackoff).init
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-swarm@v0.3.1/swarm_dial.go:119 +0x54

goroutine 346 [select]:
github.com/libp2p/go-libp2p-autonat.(*autoNATService).background(0x40005dadc0, 0x2e7df60, 0x4000641e80)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-autonat@v0.4.0/svc.go:217 +0x184
created by github.com/libp2p/go-libp2p-autonat.(*autoNATService).Enable
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-autonat@v0.4.0/svc.go:197 +0xd8

goroutine 349 [select]:
github.com/libp2p/go-libp2p-autonat.(*AmbientAutoNAT).background(0x400033dba0)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-autonat@v0.4.0/autonat.go:175 +0x1fc
created by github.com/libp2p/go-libp2p-autonat.New
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p-autonat@v0.4.0/autonat.go:125 +0x590

goroutine 350 [select]:
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).background(0x40002ca2c0)
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p@v0.12.0/p2p/host/basic/basic_host.go:520 +0x1cc
created by github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).Start
	/srv/lotus/go/pkg/mod/github.com/libp2p/go-libp2p@v0.12.0/p2p/host/basic/basic_host.go:368 +0x58
@ribasushi
Copy link
Collaborator Author

Commenting out this line: https://github.com/filecoin-project/lotus/blob/v1.6.0/node/builder.go#L704

Gets us to:

...
2021/04/06 13:42:21 [Fx] PROVIDE	fx.Shutdowner <= go.uber.org/fx.(*App).shutdowner-fm()
2021/04/06 13:42:21 [Fx] PROVIDE	fx.DotGraph <= go.uber.org/fx.(*App).dotGraph-fm()
2021/04/06 13:42:21 [Fx] INVOKE		github.com/filecoin-project/lotus/node/modules.MemoryWatchdog()
2021/04/06 13:42:21 [Fx] Error during "github.com/filecoin-project/lotus/node/modules.MemoryWatchdog()" invoke: cycle detected in dependency graph: dtypes.ChainBlockstore provided by "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_arm64.s:12)
	depends on dtypes.ChainBlockstore provided by "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_arm64.s:12)
ERROR: initializing node: starting node: cycle detected in dependency graph: dtypes.ChainBlockstore provided by "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_arm64.s:12)
	depends on dtypes.ChainBlockstore provided by "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_arm64.s:12)

I assume something splitstore-related?

@jennijuju jennijuju added area/chain Area: Chain and removed hint/needs-triaging labels Apr 6, 2021
@jennijuju
Copy link
Member

@vyzo @raulk thoughts on ^ ?

@vyzo
Copy link
Contributor

vyzo commented Apr 6, 2021

DI seems to fall into a loop; @magik6k might be the only person who really understands how the DI contraptions work.

@raulk
Copy link
Member

raulk commented Apr 6, 2021

@ribasushi does this node have the splitstore enabled?

@ribasushi
Copy link
Collaborator Author

@raulk No, stock config.

@firesWu
Copy link
Contributor

firesWu commented Apr 8, 2021

same error

@jennijuju jennijuju added the P2 P2: Should be resolved label Apr 9, 2021
@jennijuju
Copy link
Member

DI seems to fall into a loop; @magik6k might be the only person who really understands how the DI contraptions work.

@magik6k gentle ping^

@magik6k magik6k mentioned this issue Apr 9, 2021
Merged
@rickiey rickiey mentioned this issue Apr 19, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/chain Area: Chain kind/bug Kind: Bug P2 P2: Should be resolved
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix fallback chainstore filecoin-project/lotus
5 participants
@vyzo @ribasushi @raulk @firesWu @jennijuju