16 bit Images?

[kanlions]

If I have 16 bit images what type of transformation should be done in the data transform section?

Describe the solution you'd like
I just want to know how to use TIMM with 16 bit images

Describe alternatives you've considered
I have considered dividing it by 65535 but I am not sure whether doing that is enough since the TIMM calls a model specific config so not sure how to handle it

Additional context
Any guidance is appreciated

[rwightman]

@kanlions if you have 16-bit images you should be able to use them with timm train scripts with few to no changes if you disable the prefetcher (prefetcher has uint8 hard coded). The typical normalization should work but you may need to set a custom mean/std ... also you'd be relying on the torchvision ToTensor to do the right thing re 16-bit -> float tensor. That could depend on the specifics of your 16-bit format and you may need to scale yourself to get it centered at 0 w/ roughly std-dev of 1.0.

[K-D-Gallagher]

Hi @rwightman, I am also trying to use the timm train scripts with 16 bit images. I'm having a couple of issues and points of confusion

1. Disabling the prefetcher is leading to an error:

File "train.py", line 870 in train_one_epoch
input, target = input.to(device), target.to(device)
AttributeError: 'list' object has no attribute 'to'

It seems like input is a list of three batches (each element of the list is torch.Size([2000,3,96,96]) which is my batch size). I'm not sure how to move this list to the GPU and I'm also not sure why input is a list of three batches.

2. Where is normalization being applied when the prefetched is disabled?
3. Is there a way to hard-code the prefetcher to load images as 16 bit instead of 8 bit?

Thank you!

[K-D-Gallagher]

Actually, I think the issue is deeper than just inactivating the prefetcher - PIL is used for image opening and augmentation and PIL simply does not support 16 bit. Changing how the images are read in to accommodate 16 bit is pretty simple (skimage io or cv2) and the prefetcher dtypes need modified as well, but then all of the augmentation would seemingly need to be replaced as well because it requires PIL images

[luckyhug]

I think it would be great if we could process numpy arrays with all different channels same as PIL images?