imdb instance sampling is non-deterministic

[yifanmai]

The reason for this is that the instance order is dependent on the order of files returned by os.listdir() here, which is arbitrary. This bug already existed in v0.1.0.

This is a subtle bug because running multiple times on the same download folder usually results in the same order due to scenario download caching (the order seems to be based on some OS-dependent order). But deleting the scenario download folder and re-running it causes the order to change.

The long term fix is to use sorted(os.listdir()) instead.

[yifanmai]

code has the same problem. We should audit the other scenarios as well.

[yifanmai]

ice has the same problem.

[yifanmai]

Unfortunately we recently migrated the scenarios between disks and this caused the os.listdir to be shuffled, so now we don't have access to the original os.listdir order. I might be able to reconstruct the original order based on the published evaluations, but it'll be tricky.