wrong test acc because redundant data in ddp mode

[xiadingZ]

If I have 499 videos to test set, in ddp model, It will load 512 videos to test, maybe by copying videos to match the batch size. But it will cause wrong test accuracy. Now I need to save each videos' predictions and calculate acc by my own. Is there any way to solve this problem?

[Borda]

@xiadingZ mind sharing some code example or Colab or how do you know that your test loaded more videos, what is you batch size?

[xiadingZ]

I write a test_flist to load video data, it only has 499 line video info. And in test_epoch_end, I write all video's prediction score to disk, with video id. Then I load these predictions, it has 512 videos, some of videos have same id. my batch size is 8, use ddp mode on multi-gpu

this ls my test dataloader

        dataset = VideoDataset(self.hparams, mode='val', transform=transform)
        return DataLoader(dataset, batch_size=self.batch_size,
                          num_workers=self.num_workers, pin_memory=True)

[SkafteNicki]

Duplicate of #2398
The short answer is that DistributedSampler adds additional samples to even the load over all processes. This will cause a slight bias in the metric value. Currently, you would need to run the testing on single gpu, until we support uneven inputs in ddp (#3325)

[Borda]

closing in favor of #2398 so pls continue the thread there 🐰