Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FSDP integration #6152

Closed
wants to merge 77 commits into from
Closed

FSDP integration #6152

wants to merge 77 commits into from

Commits on Feb 23, 2021

  1. Add initial FSDP integration

    SeanNaren committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    78f1eb4 View commit details
    Browse the repository at this point in the history

  2. Fix error in refactor

    SeanNaren committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    c36e00a View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2021

  1. update

    @tchaton
    tchaton committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    59dbb83 View commit details
    Browse the repository at this point in the history

  2. Revert "update" 

    This reverts commit 59dbb83
    SeanNaren committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    19a1440 View commit details
    Browse the repository at this point in the history

  3. Address reviews

    SeanNaren committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    3b38615 View commit details
    Browse the repository at this point in the history

  4. Fix doc string

    SeanNaren committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    5ff06ab View commit details
    Browse the repository at this point in the history

  5. Even moar code review

    SeanNaren committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    36434f0 View commit details
    Browse the repository at this point in the history

  6. Add deprecation

    SeanNaren committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    c61a190 View commit details
    Browse the repository at this point in the history

Commits on Feb 25, 2021

  1. Merge branch 'master' into feat/fsdp

    SeanNaren committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    1c4f011 View commit details
    Browse the repository at this point in the history

  2. Fix name of test

    SeanNaren committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    02599e6 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2021

  1. Integrate nesting, fix bugs across implementation

    SeanNaren committed Mar 1, 2021
    Configuration menu
    Copy the full SHA
    e79977a View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2021

  1. Merge branch 'master' into feat/fsdp

    SeanNaren committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    d15d4b5 View commit details
    Browse the repository at this point in the history

  2. Formatting types

    SeanNaren committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    ebf1818 View commit details
    Browse the repository at this point in the history

  3. Add additional tests for accelerator model

    SeanNaren committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    290e8fd View commit details
    Browse the repository at this point in the history

  4. Fix import

    SeanNaren committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    5c5f762 View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2021

  1. Few test fixes, expose params

    SeanNaren committed Mar 3, 2021
    Configuration menu
    Copy the full SHA
    d28438b View commit details
    Browse the repository at this point in the history

  2. Allow training_type_plugin to delay optimizer configure

    SeanNaren committed Mar 3, 2021
    Configuration menu
    Copy the full SHA
    ab591a8 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'feat/fsdp_2n' into feat/fsdp

    SeanNaren committed Mar 3, 2021
    Configuration menu
    Copy the full SHA
    23ccdb8 View commit details
    Browse the repository at this point in the history

  4. Add missing references to trainer, add a CPU accelerator based test

    SeanNaren committed Mar 3, 2021
    Configuration menu
    Copy the full SHA
    a60f2c0 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2021

  1. Merge branch 'feat/fsdp_2n' into feat/fsdp

    SeanNaren committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    3d4e6df View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2021

  1. Update for latest API changes to fairscale

    SeanNaren committed Mar 9, 2021
    Configuration menu
    Copy the full SHA
    516bd04 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2021

  1. Add base hook for model parallel

    SeanNaren committed Mar 23, 2021
    Configuration menu
    Copy the full SHA
    9f8864f View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2021

  1. fix callback signature

    @kaushikb11
    kaushikb11 committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    eac5344 View commit details
    Browse the repository at this point in the history

  2. Simplify hook

    SeanNaren committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    32df0cb View commit details
    Browse the repository at this point in the history

  3. Add hook logic

    SeanNaren committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    282a133 View commit details
    Browse the repository at this point in the history

  4. add tests

    @kaushikb11
    kaushikb11 committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    7a94e72 View commit details
    Browse the repository at this point in the history

  5. add property setter

    @kaushikb11
    kaushikb11 committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    8091481 View commit details
    Browse the repository at this point in the history

  6. add logic for being called once

    @kaushikb11
    kaushikb11 committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    633fc77 View commit details
    Browse the repository at this point in the history

  7. Update changelog

    @kaushikb11
    kaushikb11 committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    c99a36f View commit details
    Browse the repository at this point in the history

  8. Merge branch 'master' into feat/model_parallel_hook

    @kaushikb11
    kaushikb11 committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    a68c8d7 View commit details
    Browse the repository at this point in the history

  9. Fix

    @kaushikb11
    kaushikb11 committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    9529a22 View commit details
    Browse the repository at this point in the history

  10. fix return type

    @kaushikb11
    kaushikb11 committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    3c1c782 View commit details
    Browse the repository at this point in the history

  11. Merge branch 'master' into feat/fsdp 

    # Conflicts:
#	pytorch_lightning/accelerators/accelerator.py
#	pytorch_lightning/plugins/__init__.py
#	pytorch_lightning/plugins/precision/__init__.py
#	pytorch_lightning/plugins/precision/sharded_native_amp.py
#	pytorch_lightning/plugins/training_type/rpc_sequential.py
#	pytorch_lightning/plugins/training_type/training_type_plugin.py
#	pytorch_lightning/trainer/connectors/accelerator_connector.py
#	pytorch_lightning/trainer/trainer.py
#	requirements/extra.txt
#	tests/accelerators/test_cpu.py
#	tests/trainer/properties/test_get_model.py
    SeanNaren committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    7daba43 View commit details
    Browse the repository at this point in the history

  12. Fix property name

    SeanNaren committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    87ec222 View commit details
    Browse the repository at this point in the history

  13. Merge branch 'feat/model_parallel_hook' into feat/fsdp 

    # Conflicts:
#	pytorch_lightning/core/hooks.py
    SeanNaren committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    966b2e5 View commit details
    Browse the repository at this point in the history

  14. Updaet wrapper, use latest fixes for hooks

    SeanNaren committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    5f6e039 View commit details
    Browse the repository at this point in the history

  15. Swap hook order

    SeanNaren committed Mar 25, 2021
    Configuration menu
    Copy the full SHA
    b512e72 View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2021

  1. Merge branch 'master' into feat/fsdp

    SeanNaren committed Mar 29, 2021
    Configuration menu
    Copy the full SHA
    8ba82df View commit details
    Browse the repository at this point in the history

  2. Small changes

    SeanNaren committed Mar 29, 2021
    Configuration menu
    Copy the full SHA
    1e5ca37 View commit details
    Browse the repository at this point in the history

  3. Fixes

    SeanNaren committed Mar 29, 2021
    Configuration menu
    Copy the full SHA
    936dc1a View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2021

  1. Remove activation checkpointing

    SeanNaren committed Apr 1, 2021
    Configuration menu
    Copy the full SHA
    a6de18e View commit details
    Browse the repository at this point in the history

  2. Turn off auto wrap by default

    SeanNaren committed Apr 1, 2021
    Configuration menu
    Copy the full SHA
    8684f94 View commit details
    Browse the repository at this point in the history

Commits on Apr 7, 2021

  1. Move to trainer.model

    SeanNaren committed Apr 7, 2021
    Configuration menu
    Copy the full SHA
    76091ae View commit details
    Browse the repository at this point in the history

  2. fix reference

    SeanNaren committed Apr 7, 2021
    Configuration menu
    Copy the full SHA
    226d498 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into feat/fsdp

    SeanNaren committed Apr 7, 2021
    Configuration menu
    Copy the full SHA
    cd63c10 View commit details
    Browse the repository at this point in the history

  4. Remove flag

    SeanNaren committed Apr 7, 2021
    Configuration menu
    Copy the full SHA
    b881e2f View commit details
    Browse the repository at this point in the history

  5. Fix imports

    SeanNaren committed Apr 7, 2021
    Configuration menu
    Copy the full SHA
    e8959be View commit details
    Browse the repository at this point in the history

  6. Fix versions, update docs

    SeanNaren committed Apr 7, 2021
    Configuration menu
    Copy the full SHA
    52478ac View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2021

  1. Fix clip gradients

    SeanNaren committed Apr 8, 2021
    Configuration menu
    Copy the full SHA
    b7f1896 View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2021

  1. Merge branch 'master' into feat/fsdp

    SeanNaren committed Apr 10, 2021
    Configuration menu
    Copy the full SHA
    a62f8d8 View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2021

  1. Merge branch 'master' into feat/fsdp

    SeanNaren committed Apr 14, 2021
    Configuration menu
    Copy the full SHA
    69c33f1 View commit details
    Browse the repository at this point in the history

  2. Fixes

    SeanNaren committed Apr 14, 2021
    Configuration menu
    Copy the full SHA
    9fa26c0 View commit details
    Browse the repository at this point in the history

  3. pull

    SeanNaren committed Apr 14, 2021
    Configuration menu
    Copy the full SHA
    56f23ce View commit details
    Browse the repository at this point in the history

  4. Few changes across the board

    SeanNaren committed Apr 14, 2021
    Configuration menu
    Copy the full SHA
    9ca3f0c View commit details
    Browse the repository at this point in the history

  5. Fix imports

    SeanNaren committed Apr 14, 2021
    Configuration menu
    Copy the full SHA
    b53ba36 View commit details
    Browse the repository at this point in the history

  6. Set none

    SeanNaren committed Apr 14, 2021
    Configuration menu
    Copy the full SHA
    0da5249 View commit details
    Browse the repository at this point in the history

  7. Swap to warnings

    SeanNaren committed Apr 14, 2021
    Configuration menu
    Copy the full SHA
    90c6479 View commit details
    Browse the repository at this point in the history

  8. Remove fairscale from container

    SeanNaren committed Apr 14, 2021
    Configuration menu
    Copy the full SHA
    69d8178 View commit details
    Browse the repository at this point in the history

  9. pull

    SeanNaren committed Apr 14, 2021
    Configuration menu
    Copy the full SHA
    a459d10 View commit details
    Browse the repository at this point in the history

  10. Update dockers/base-cuda/Dockerfile

    Sean Naren committed Apr 14, 2021
    Configuration menu
    Copy the full SHA
    a7842d9 View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2021

  1. Add defaults, add test to ensure nested wrapper is set correctly

    SeanNaren committed Apr 15, 2021
    Configuration menu
    Copy the full SHA
    48ee83f View commit details
    Browse the repository at this point in the history

  2. Remove deprecation as this will be removed completely

    SeanNaren committed Apr 15, 2021
    Configuration menu
    Copy the full SHA
    57a696c View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2021

  1. Check for nested FSDP wrappers, and omit wrapping algorithm

    SeanNaren committed Apr 16, 2021
    Configuration menu
    Copy the full SHA
    36889b8 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into feat/fsdp 

    # Conflicts:
#	pytorch_lightning/plugins/precision/sharded_native_amp.py
#	pytorch_lightning/plugins/training_type/ddp.py
#	requirements/extra.txt
    SeanNaren committed Apr 16, 2021
    Configuration menu
    Copy the full SHA
    89b8cb5 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2021

  1. Update pytorch_lightning/trainer/connectors/accelerator_connector.py 

    Co-authored-by: ananthsub <ananth.subramaniam@gmail.com>
    @ananthsub
    Sean Naren and ananthsub committed Apr 21, 2021
    Configuration menu
    Copy the full SHA
    0c1d2de View commit details
    Browse the repository at this point in the history

  2. Address code review points

    SeanNaren committed Apr 21, 2021
    Configuration menu
    Copy the full SHA
    592bb28 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2021

  1. Merge branch 'master' into feat/fsdp

    Sean Naren committed Apr 26, 2021
    Configuration menu
    Copy the full SHA
    4e230c9 View commit details
    Browse the repository at this point in the history

  2. Add back missing model that was removed from clipping signature

    SeanNaren committed Apr 26, 2021
    Configuration menu
    Copy the full SHA
    ca8e586 View commit details
    Browse the repository at this point in the history

  3. Do not pass model through, accelerator does it

    SeanNaren committed Apr 26, 2021
    Configuration menu
    Copy the full SHA
    54f501d View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2021

  1. Merge branch 'master' into feat/fsdp

    Sean Naren committed Apr 27, 2021
    Configuration menu
    Copy the full SHA
    02925cc View commit details
    Browse the repository at this point in the history

  2. Fix merge

    SeanNaren committed Apr 27, 2021
    Configuration menu
    Copy the full SHA
    b67f1a9 View commit details
    Browse the repository at this point in the history

  3. Fix imports

    SeanNaren committed Apr 27, 2021
    Configuration menu
    Copy the full SHA
    132eb64 View commit details
    Browse the repository at this point in the history

  4. Changes to precision plugin

    SeanNaren committed Apr 27, 2021
    Configuration menu
    Copy the full SHA
    e6ce3cf View commit details
    Browse the repository at this point in the history

  5. Require 2 GPU for multi gpu test

    SeanNaren committed Apr 27, 2021
    Configuration menu
    Copy the full SHA
    01153af View commit details
    Browse the repository at this point in the history

Commits on May 2, 2021

  1. Merge branch 'master' into feat/fsdp

    Sean Naren committed May 2, 2021
    Configuration menu
    Copy the full SHA
    6cfe57d View commit details
    Browse the repository at this point in the history

Commits on May 4, 2021

  1. Use callback in test, swap to DynamicLossScaler from fairscale to tes… 

    …t it out
    SeanNaren committed May 4, 2021
    Configuration menu
    Copy the full SHA
    efa81ab View commit details
    Browse the repository at this point in the history

  2. Disable loss scaler for now

    SeanNaren committed May 4, 2021
    Configuration menu
    Copy the full SHA
    78d52b5 View commit details
    Browse the repository at this point in the history