Replies: 3 comments 7 replies
-
|
@rwightman quick question - at which point or where in BITS does data sharding occur? Per my understanding each process should get a part of the data (this is based on HF accelerate) - I am not sure how this is dealt with in BITS? |
Beta Was this translation helpful? Give feedback.
-
|
Never mind about this one - found it! https://github.com/rwightman/pytorch-image-models/blob/bits_and_tpu/timm/data/loader.py#L81 |
Beta Was this translation helpful? Give feedback.
-
|
I'm actually using the TFDS + IterableDataset for all my TPU training right now, so that sharding code is here https://github.com/rwightman/pytorch-image-models/blob/bits_and_tpu/timm/data/parsers/parser_tfds.py#L103-L156 |
Beta Was this translation helpful? Give feedback.
-
|
Got it! Thanks. :) |
Beta Was this translation helpful? Give feedback.
-
|
Also, is there a way to use your version of ImageNet data? Believe it would save me some $$ to kick off training using your version and not host my own new dataset. |
Beta Was this translation helpful? Give feedback.
-
|
That would request setting up the bucket as 'requester pays' and, I don't think that requester pays works with the TF data GS reading code (needs an extra field in the HTTP requests).... The storage costs for imagenet aren't that significant, about $5 per month. The transfer costs (egress charge is free in same region, so you just pay for the op/sec) were about $35 with ~6 instances going reading full tilt for the month. The only cost issue I've had were cross continent transfer and that was REALLY expensive. Hence there is no way I'm not enabling requester pays if I share the bucket :) Scaling up to imagenet21k and openimages and beyond (video) is going to be a budgetary challenge though... |
Beta Was this translation helpful? Give feedback.
-
|
No worries - thanks Ross! Didn't realize it was $5/mo. Happy to host the dataset in my own bucket. :) |
Beta Was this translation helpful? Give feedback.
-
|
@amaarora BTW, re the timm bits vs accelerate comparison, a few corrections. First timm bits is a collection of multiple 'bits', the idea being you can use up to whatever level of abstraction you want. At it's lowest level the DeviceEnv is a thinner abstraction than accelerate and doesn't lock you in to any other design decisions re prepare. It's a very low level device wrapper that covers most PyTorch + CPU vs PyTorch + CUDA vs Pytorch XLA + TPU/GPU/CPU differences for common float32/(b)float16/AMP and distributed scenarios. I decided to let some hardware detail leak into the Updater classes to keep that logic clean and performant. The coupling of the grad scaler with clipping and the grad synchronization you need to do in PyTorch XLA vs Pytorch w/ DDP made it hard to do nicely w/ just device env. Also, the monster that is DeepSpeed (one massive Engine class that includes every model, optimizer, detail imageinable) makes it really difficult to use without making DeviceEnv cover 'too much'. I feel I can probably do that reasonably at the Updater level w/ some Deepspeed hacks but we shall see. One can use just
There is absolutely no need to use the timm bits provided train or eval loops/fns. My train script will use them but they won't cover every use case and I will encourage users to write their own when they need to vs me trying to add interfaces, hooks, and fns to cover all foreseable use cases. |
Beta Was this translation helpful? Give feedback.
-
|
Also the current train loop, train functions (including the setup) are transitional from the previous timm variants... quite a bit will change there |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @rwightman for sharing! Will update the comparison to include these points in it. |
Beta Was this translation helpful? Give feedback.
-
Starting a new discussion post to discuss things related to timm bits. Will continue to add comments/post in this discussion post to keep it as a central place in case anybody else wants to read it in the future too.
Below, comments aren't really answers but rather further questions as I go through the source code.
Beta Was this translation helpful? Give feedback.
All reactions