versiondb lastest version don't match iavl latest version

[windycrypto]

Describe the bug

recently I reboot our server, then cronosd reports the following error on starting:

"versiondb lastest version 12063409 don't match iavl latest version 12063413"

I guess maybe cronosd doesn't stop gracefully when I reboot the server, maybe I should terminate cronosd before reboot the server. This is not 100% reproducible, a normal restart of the cronosd doesn't got this issue.
While despite that, could we have a self-healing approach to auto fix the mismatched version instead of failed to start, like rollback iavl version to match versiondb?

cronosd version: v1.0.14

[yihuang]

yes, this is known issue with ungraceful shutdown, for now there are some tricks you can do to recover fast, for example for rocksdb:

$ ldb put "s/latest" $(printf "0x%X" 12063409) --value_hex --db=/path/to/application.db

ldb is a tool builtin rocksdb repo, you can built yourself, or if you use nix, we also provide it in cronos repo: nix profile install github:crypto-org-chain/cronos#rocksdb.tools.