Keep failing at runPost - sanity checked failed

[jennijuju]

Lotus version 0.5.7

miner id: t09964

Issue: The miner is claiming that they were attacked once which lead sectors to be put in fault stage. They were hoping to recover them, however they were not able to submit PoSt proof successfully ever since. They are running 73 workers and 867/868 sectors(~40TiB power) were in fault in ddl1.

Error messages

{
"level": "error",
"ts": "2020-09-01T18:28:45.061+0800",
"logger": "storageminer",
"caller": "storage/wdpost_run.go:61",
"msg": "runPost failed: running post failed:\n github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPost\n /home/ps/code/community/lotus/storage/wdpost_run.go:410\n - sanity check failed\n github.com/filecoin-project/filecoin-ffi.GenerateWindowPoSt\n \t/home/ps/code/community/lotus/extern/filecoin-ffi/proofs.go:579\n github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).GenerateWindowPoSt\n \t/home/ps/code/community/lotus/extern/sector-storage/ffiwrapper/verifier_cgo.go:41\n github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPost\n \t/home/ps/code/community/lotus/storage/wdpost_run.go:408\n github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).dPost.func1\n \t/home/ps/code/community/lotus/storage/wdpost_run.go:49\n runtime.goexit\n \t/usr/lib/go-1.14/src/runtime/asm_amd64.s:1373"
}

2. ```bash
{"level":"error","ts":"2020-09-02T18:29:10.504+0800","logger":"storageminer","caller":"storage/wdpost_run.go:61","msg":"runPost failed: running post failed:\n    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPost\n        /home/ps/code/community/lotus/storage/wdpost_run.go:410\n  - sanity check failed\n    github.com/filecoin-project/filecoin-ffi.GenerateWindowPoSt\n    \t/home/ps/code/community/lotus/extern/filecoin-ffi/proofs.go:579\n    github.com/filecoin-project/lotus/extern/sector-storage/ffiwrapper.(*Sealer).GenerateWindowPoSt\n    \t/home/ps/code/community/lotus/extern/sector-storage/ffiwrapper/verifier_cgo.go:41\n    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).runPost\n    \t/home/ps/code/community/lotus/storage/wdpost_run.go:408\n    github.com/filecoin-project/lotus/storage.(*WindowPoStScheduler).dPost.func1\n    \t/home/ps/code/community/lotus/storage/wdpost_run.go:49\n    runtime.goexit\n    \t/usr/lib/go-1.14/src/runtime/asm_amd64.s:1373"}

Note: I asked the miner for output of lotus-miner sealing jobs and `lotus-miner sectors status --log . Let me know if anything else can be helpful for debugging.

[axzmin]

How solved this problem?

[jennijuju]

@arajasek can you forward this ticket to the right person so that we can get this investigated? i saw 5-10 miners report the same issue everyday

[jennijuju]

#3710