Support Index generation on the fly during shard acquisition

[aarshkshah1992]

Lotus Miners will delete the dagstore repo if they have to rollback for some reason. This means that retrieval deals will fail for the (storage deal with upgraded miner -> miner rolls-back and deletes the dagstore repo -> miner upgrades and we do a retrieval) scenario because AcquireShard will fail to find an Index. I think we should allow index generation on the fly during shard acquisition for scenarios such as these where the underlying Mount returns valid data but the Index has gone missing. We can have an option to disable this if needed.

I think a lot of clients will face scenarios such as these.

[raulk]

Isn't lotus executing the migration again on the second upgrade? If not, that should be fixed.

[aarshkshah1992]

@raulk

• But the shard has already been registered. Even if we execute the migration again, we still wont re-index it (assuming the datastore we pass to DAGStore will already have the shard registered) unless we ship a cli/instructions on how to clear up the datastore. Actually, I just thought about this and think that we should clear up the existing DAGStore data in Badger before starting a migration as the migration will build up the enitre state again.

• However, it's not inconceivable that miners might delete Index files (even if we keep aside the discussion of upgrade/rollback) to reclaim space but still expect retrievals to work. Future non-Lotus dagstore clients too will need this I think.