Test failures/flakiness

[anorth]

Describe the bug
I am having real trouble getting the tests to pass, locally and on CI. I can't quite trace it down and not sure if it's just me.

Locally, the typical failure on master is:

--- FAIL: TestInspectConfig (15.40s)
    require.go:794:
        	Error Trace:	basic.go:109
        	            				inspector_daemon_test.go:31
        	Error:      	Received unexpected error:
        	            	context deadline exceeded
        	Test:       	TestInspectConfig
    log_writer.go:44: >>>> start-dump
    log_writer.go:44: ---- command     go-filecoin miner update-peerid --from t1nhudgskmseowv7rp4e6scxsmlt3qoysvpn73tuy --gas-price 300 --gas-limit 300 t2udhmwzokf4hq7oe4zj4g3faah7oz4dnbxtxaqoy QmNu3uJ5uRVGfauaTjBvUcMsSJEneL4PaTVWqTwChYHJwf --enc=json
    log_writer.go:44: ---- exit-code   0
    log_writer.go:44: ---- error       nil
    log_writer.go:44: ---- stdout
    log_writer.go:44: {"Cid":{"/":"zDPWYqFCwyN3H9vjLrvZDYYZVL4HHE8RAGSTJDxc51nQtmer6i81"},"GasUsed":"0","Preview":false}
    log_writer.go:44: ---- stderr
    log_writer.go:44: <<<< end-dump
    log_writer.go:44: <nil>
    inspector_daemon_test.go:28:
        	Error Trace:	inspector_daemon_test.go:28
        	            				panic.go:406
        	            				testing.go:609
        	            				basic.go:109
        	            				inspector_daemon_test.go:31
        	Error:      	Received unexpected error:
        	            	error killing daemon /var/folders/qn/3bnp3bys3rdgwhmg0zn0dwd00000gn/T/TestInspectConfig459344688/1: open /var/folders/qn/3bnp3bys3rdgwhmg0zn0dwd00000gn/T/TestInspectConfig459344688/1/daemon.pid: no such file or directory
        	Test:       	TestInspectConfig
--- FAIL: TestSelfDialStorageGoodError (12.82s)
    require.go:794:
        	Error Trace:	client_daemon_test.go:267
        	Error:      	Received unexpected error:
        	            	filecoin command: [go-filecoin miner create --gas-price 1 --gas-limit 300 10 1 --enc=json], exited with non-zero exitcode: 1
        	Test:       	TestSelfDialStorageGoodError
    log_writer.go:44: >>>> start-dump
    log_writer.go:44: ---- command     go-filecoin mining once --enc=json
    log_writer.go:44: ---- exit-code   0
    log_writer.go:44: ---- error       nil
    log_writer.go:44: ---- stdout
    log_writer.go:44: {"/":"zDPWYqFD5CkeMChZWpRFLUChDGtEjfHgTSREMCnYCNR1WhoAevdD"}
    log_writer.go:44: ---- stderr
    log_writer.go:44: <<<< end-dump
    log_writer.go:44: >>>> start-dump
    log_writer.go:44: ---- command     go-filecoin miner create --gas-price 1 --gas-limit 300 10 1 --enc=json
    log_writer.go:44: ---- exit-code   1
    log_writer.go:44: ---- error       nil
    log_writer.go:44: ---- stdout
    log_writer.go:44: ---- stderr
    log_writer.go:44: Error: Could not create miner. Please consult the documentation to setup your wallet and genesis block correctly: no actor at address t1hrxvsfcp2vjbi2y3zmrrzb57ipsvbrf5tr36yvi: actor not found
    log_writer.go:44: <<<< end-dump
FAIL
FAIL	github.com/filecoin-project/go-filecoin/commands	91.271s

• TestInspectConfig fails a lot for me locally, but I don't observe it on CI.
• An actor not found message is common in both

Here are some recent failing CI builds on master:

• 15813, TestBlockDaemon "address already in use"
• 15872 TestPaymentChannelLs "actor not found

At least some builds on CI pass, though, unlike my local environment.

Here are more on a branch (which does have changes from master, but a downstream branch scored a green build at some subsequent point)

• 15857 TestPaymentChannelCreateSuccess "actor not found"
• 15869 TestVoucherPersistenceAndPayments "badger ... writes are blocked"
• 15877 TestChainLs "address already in use"

To Reproduce
Locally, go run ./build/*.go best, or inspect/rebuild CI.

Expected behavior
Tests pass reliably

Version information

• Go: go version go1.12.1 darwin/amd64
• Rust: error: couldn't read version: No such file or directory (os error 2)
• Cargo: cargo 1.35.0-nightly (0e35bd8af 2019-03-13)
• Commit: 83faad9

• Version: commit: 83faad9

Additional context
I have been doing some work moving directories around, including in IPTB plugin. The local failures above are on master, though.

If I run a focussed test, like go run ./build/*.go best -run TestInspectConfig, it passes.

[anorth]

I just scored a green build on the CI branch with all the failures linked above: https://circleci.com/gh/filecoin-project/go-filecoin/15895

[rosalinekarr]

@frrist, you and I are the only ones that have touched TestInspectConfig afaik. do you have any idea what could cause this? i can't think of anything.