Validator experiencing a panic

[piersy]

Validator raised this issue on discord - https://discord.com/channels/600834479145353243/601046166620078100/1262163359428775967

Expected Behavior

No panics

Actual Behavior

A panic

Steps to reproduce the behavior

Using one of the following images:
us.gcr.io/celo-org/geth:1.8.2/1.8.4

Running a backup validator with the following args:

      --verbosity 3 
      --syncmode full 
      --etherbase $CELO_VALIDATOR_SIGNER_ADDRESS 
      --nodiscover 
      --nousb
      --mine 
      --proxy.proxied 
      --proxy.proxyenodeurlpairs=enode://$PROXY_ENODE02@$PROXY_EXTERNAL_IP02:30503\;enode://$PROXY_ENODE02@$PROXY_EXTERNAL_IP02:30303 
      --unlock=$CELO_VALIDATOR_SIGNER_ADDRESS 
      --password /root/.celo/.password 
      --celostats=stakely.io@stats-server.celo.org
      --light.serve 0
      --datadir /root/.celo

Syncing on top of a downloaded datadir, the latest at the 14th July.

Backtrace

 WARN [07-14|20:52:41.199] Error in retrieving exchange rate.  Will do comparison of two values without exchange rate conversion. currency1=0x2F25deB3848C207fc8E0c34035B3Ba7fC157602B err1="numerator zero: Exchange rate returned from the network is zero" currency2=nil err2=nil
WARN [07-14|20:52:41.199] Error in retrieving exchange rate.  Will do comparison of two values without exchange rate conversion. currency1=nil err1=nil currency2=0x2F25deB3848C207fc8E0c34035B3Ba7fC157602B err2="numerator zero: Exchange rate returned from the network is zero"
 ERROR[07-14|20:52:41.200] Error invoking evm function: can't unpack result to=0x2F25deB3848C207fc8E0c34035B3Ba7fC157602B method=balanceOf err="abi: attempting to unmarshall an empty string while arguments are expected" maxgas=100,000
 ERROR[07-14|20:52:41.200] GetBalanceOf evm invocation error        err="abi: attempting to unmarshall an empty string while arguments are expected"
 panic: runtime error: invalid memory address or nil pointer dereference
 [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5ce2a5]
 
 goroutine 159592 [running]:
 math/big.(*Int).Cmp(0x170b6a0?, 0xc0a9709080?)
       math/big/int.go:324 +0x25
 github.com/celo-org/celo-blockchain/core.(*txList).Filter(0xc0993563c0, 0xc0544cb480, 0xc083631ab8, 0x2faf080)
       github.com/celo-org/celo-blockchain/core/tx_list.go:422 +0x316
 github.com/celo-org/celo-blockchain/core.(*TxPool).promoteExecutables(0xc00035b440, {0xc088356468, 0x1, 0x928b11?})
       github.com/celo-org/celo-blockchain/core/tx_pool.go:1474 +0x62d
 github.com/celo-org/celo-blockchain/core.(*TxPool).runReorg(0xc00035b440, 0xc0305cca80, 0x0, 0xc0544cb060, 0xc0b02c9e60)
       github.com/celo-org/celo-blockchain/core/tx_pool.go:1283 +0x585
 created by github.com/celo-org/celo-blockchain/core.(*TxPool).scheduleReorgLoop
       github.com/celo-org/celo-blockchain/core/tx_pool.go:1196 +0x1b3

Chain/Network: Mainnet

[jamescowling]

I've also been running into this issue repeatedly when syncing from the latest chain backup. One time I was able to rewind past the unknown canonical block #26588289 via debug.setHead("0x195B480") and have the chain recover but half a dozen other attempts have ended up in that panic.

Edit: I managed to get my validator to sync finally after a few rounds of restarting from a snapshot and rolling back to version 1.8.0. Seems it self-resolves eventually and I'm now running fine back on 1.8.4.