Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Support for multiple train loaders #1959

Merged
merged 38 commits into from
Jan 4, 2021
Merged

Add Support for multiple train loaders #1959

merged 38 commits into from
Jan 4, 2021

Commits on Dec 21, 2020

  1. add support for wrong dtype in apply_func

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    98aafa1 View commit details
    Browse the repository at this point in the history

  2. apply loader resetting to possible collection of loaders

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    f7e2405 View commit details
    Browse the repository at this point in the history

  3. add combined loader iter class

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    dc1f0d0 View commit details
    Browse the repository at this point in the history

  4. integrate combined loader iter to training loop

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    4ab644f View commit details
    Browse the repository at this point in the history

  5. fix imports

    @Borda
    Justus Schock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    6b8f317 View commit details
    Browse the repository at this point in the history

  6. fix imports

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    04ce0ef View commit details
    Browse the repository at this point in the history

  7. finish supporters

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    aa9b153 View commit details
    Browse the repository at this point in the history

  8. add tests for supporters

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    9525024 View commit details
    Browse the repository at this point in the history

  9. add test for model with multiple loaders

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    dab57af View commit details
    Browse the repository at this point in the history

  10. fix trainer integration

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    2f867dc View commit details
    Browse the repository at this point in the history

  11. fix instance check

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    0ae527b View commit details
    Browse the repository at this point in the history

  12. Train loaders (#4032) 

    * patch for issues discussed in #1959, encapsulating underlying datastructures returned from train_dataloader

* update data_loading.py to it uses patch discussed in #1959
    @christofer-f @Borda
    christofer-f authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    608f503 View commit details
    Browse the repository at this point in the history

  13. rename class

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    a1ced87 View commit details
    Browse the repository at this point in the history

  14. Separate CombinedLoaderIterator into two classes, and update related … 

    …tests. (#4606)

* Fix the bugs after rebasing.

* Add custom get_len for apply_to_collection

* Refactor MultiIterator to be as CombinedLoaderIterator

* To get the right num_training_batches. Call the wrapper for multi trainloader in data_loading.py, instead of training_loop.py

* Reload _loader_iters when calling __iter__

* Don't transform DataLoader to CombinedLoaderIterator when it's along

* Updates test_fit_multiple_train_loaders for testing num_training_batches

* Seperate CombinedLoaderIterator into CombinedLoaderIterator and CombinedDataLoader. Add CombinedDataset for unified DataLoader format.

* Initialize CombinedDataLoader before calculating num_training_batches. Also updating self._worker_check for multiple loaders

* Update tests for supporters

* Update tests for multiple trainloaders. Add tests about few_workers for multiple loaders.

* Fix pep8 issues

* Add tests for train_loader_patch.py

* Add descriptions to multiple_trainloader_mode

* Remove unused variables

* Add docstrings and typing

* Add more tests for better converage

* Remove unused commented codes

* Add sampler property

* Remove extract_dataset

* Update typing
    @ylsung @Borda
    ylsung authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    eea6aae View commit details
    Browse the repository at this point in the history

  15. pep8

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    32aeb70 View commit details
    Browse the repository at this point in the history

  16. Update train_loader_patch.py

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    c4482c4 View commit details
    Browse the repository at this point in the history

  17. Apply suggestions from code review 

    Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
    @justusschock @Borda
    justusschock and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    1a531a5 View commit details
    Browse the repository at this point in the history

  18. Update pytorch_lightning/trainer/supporters.py 

    Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
    @justusschock @Borda
    justusschock and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    a5d3652 View commit details
    Browse the repository at this point in the history

  19. reviewer comments

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    dc94a59 View commit details
    Browse the repository at this point in the history

  20. fix stupid import

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    3c85cfb View commit details
    Browse the repository at this point in the history

  21. add docs

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    c091414 View commit details
    Browse the repository at this point in the history

  22. add back line separator

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    ebb6277 View commit details
    Browse the repository at this point in the history

  23. fix line sep

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    e4e50ab View commit details
    Browse the repository at this point in the history

  24. pep8

    @justusschock @Borda
    justusschock authored and Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    0227a68 View commit details
    Browse the repository at this point in the history

  25. Apply suggestions from code review

    @Borda
    Borda committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    644a490 View commit details
    Browse the repository at this point in the history

  26. fix

    @rohitgr7
    rohitgr7 committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    552e6a6 View commit details
    Browse the repository at this point in the history

  27. fix

    @rohitgr7
    rohitgr7 committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    3ab1907 View commit details
    Browse the repository at this point in the history

Commits on Dec 31, 2020

  1. Apply suggestions from code review 

    Co-authored-by: Adrian Wälchli <aedu.waelchli@gmail.com>
    @Borda @awaelchli
    Borda and awaelchli committed Dec 31, 2020
    Configuration menu
    Copy the full SHA
    a2d017f View commit details
    Browse the repository at this point in the history

  2. Apply suggestions from code review 

    Co-authored-by: Nicki Skafte <skaftenicki@gmail.com>
    @Borda @SkafteNicki
    Borda and SkafteNicki committed Dec 31, 2020
    Configuration menu
    Copy the full SHA
    6af5c90 View commit details
    Browse the repository at this point in the history

  3. flake8

    @Borda
    Borda committed Dec 31, 2020
    Configuration menu
    Copy the full SHA
    e138c7a View commit details
    Browse the repository at this point in the history

  4. chlog

    @Borda
    Borda committed Dec 31, 2020
    Configuration menu
    Copy the full SHA
    9265651 View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2021

  1. Update pytorch_lightning/trainer/supporters.py 

    Co-authored-by: chaton <thomas@grid.ai>
    @justusschock @tchaton
    justusschock and tchaton committed Jan 4, 2021
    Configuration menu
    Copy the full SHA
    7669a40 View commit details
    Browse the repository at this point in the history

  2. add missing test

    @justusschock
    justusschock committed Jan 4, 2021
    Configuration menu
    Copy the full SHA
    8e9bd3d View commit details
    Browse the repository at this point in the history

  3. fix dataset length

    @justusschock
    justusschock committed Jan 4, 2021
    Configuration menu
    Copy the full SHA
    854756b View commit details
    Browse the repository at this point in the history

  4. Update supporters.py

    @justusschock
    justusschock committed Jan 4, 2021
    Configuration menu
    Copy the full SHA
    55037d4 View commit details
    Browse the repository at this point in the history

  5. remove unused patch

    @justusschock
    justusschock committed Jan 4, 2021
    Configuration menu
    Copy the full SHA
    9d42c7e View commit details
    Browse the repository at this point in the history

  6. remove tests of otherwise unused patch

    @justusschock
    justusschock committed Jan 4, 2021
    Configuration menu
    Copy the full SHA
    232c7ce View commit details
    Browse the repository at this point in the history

  7. Merge branch 'release/1.2-dev' into train_loaders

    @tchaton
    tchaton committed Jan 4, 2021
    Configuration menu
    Copy the full SHA
    59651c6 View commit details
    Browse the repository at this point in the history