ViT pooling refactor #2209
ViT pooling refactor #2209
Conversation
Most of the CNNs have a max global pooling option. I would like to extend ViT to have this option.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
Exciting! Looking forward to testing the new 'avgmax' do you have a reference to where this was proposed? |
|
@fcossio I forget the exact origin, I think my use was influenced by the Dual-Path-Net mean-max pooling, though they used it at test time. https://arxiv.org/abs/1707.01629 I've been using a form of it since before timm was a thing, it was useful in some satellite imagery situations in my early Kaggle adventures. Been one of the 2d pooling options in timm for a long long time https://github.com/huggingface/pytorch-image-models/blob/main/timm/layers/adaptive_avgmax_pool.py Seems to work pretty well here too. When I was testing the max pool on resisc (satellite), avgmax was actually doing better than max or avg. I was never a huge fan of max because it sparsifies the gradient so much. |
|
I believe fastai added an option for the combined avg + max pooling in a similar time frame for convnets. |
Subsume #2207 .. testing.