feature: support PyTorch 1.7.1 training, inference and data parallel #2185

eitansela · 2021-03-03T11:15:27Z

Issue #, if available:

PyTorch 1.7.1 DLC for training and inference are availible but not supported with SageMaker.

ValueError: Unsupported pytorch version: 1.7.1 You may need to upgrade your SDK version (pip install -U sagemaker) for newer pytorch versions. Supported pytorch version(s): 0.4.0, 1.0.0, 1.1.0, 1.2.0, 1.3.1, 1.4.0, 1.5.0, 1.6.0, 0.4, 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6.

Description of changes:

Added "1.7.1" version for training and inference sections of the json.

Testing done:

Training and inference on cifar10 dataset with 1.7.1, using this code.

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2021-03-03T18:39:38Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 869e6e6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T18:39:58Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 869e6e6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T18:40:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 869e6e6
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T18:47:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 869e6e6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ChoiByungWook · 2021-03-03T18:48:43Z

src/sagemaker/image_uri_config/pytorch.json

@@ -54,7 +54,8 @@
            "1.3": "1.3.1",
            "1.4": "1.4.0",
            "1.5": "1.5.0",
-            "1.6": "1.6.0"
+            "1.6": "1.6.0",
+            "1.7": "1.7.0"


Should this alias be to 1.7.1?

Yes. Fixed.

Pushed code with the fix for both inference and training version_aliases.

…n-sdk

ChoiByungWook

Awesome! Thank you so much!

I think we might have to update: https://github.com/aws/sagemaker-python-sdk/blob/master/tests/conftest.py#L172-L185

As it doesn't account for py36.

We can follow similar logic shown here: https://github.com/aws/sagemaker-python-sdk/blob/master/tests/conftest.py#L139-L156

sagemaker-bot · 2021-03-03T19:12:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 869e6e6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

eitansela · 2021-03-03T19:16:52Z

As i can see in https://github.com/aws/deep-learning-containers/blob/master/available_images.md PyTorch 1.7.1 is both 3.6 (py36) for training and inference. So if you can please explain more?

…

On Wed, Mar 3, 2021 at 9:04 PM Dan ***@***.***> wrote: ***@***.**** commented on this pull request. Awesome! Thank you so much! I think we might have to update: https://github.com/aws/sagemaker-python-sdk/blob/master/tests/conftest.py#L172-L185 As it doesn't account for py36. We can follow similar logic shown here: https://github.com/aws/sagemaker-python-sdk/blob/master/tests/conftest.py#L139-L156 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2185 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEICFBROMDET7YPLNZWXPMDTB2B4VANCNFSM4YRA5GYA> .

ChoiByungWook · 2021-03-03T19:40:48Z

As i can see in https://github.com/aws/deep-learning-containers/blob/master/available_images.md PyTorch 1.7.1 is both 3.6 (py36) for training and inference. So if you can please explain more?
…
On Wed, Mar 3, 2021 at 9:04 PM Dan @.***> wrote: @ChoiByungWook commented on this pull request. Awesome! Thank you so much! I think we might have to update: https://github.com/aws/sagemaker-python-sdk/blob/master/tests/conftest.py#L172-L185 As it doesn't account for py36. We can follow similar logic shown here: https://github.com/aws/sagemaker-python-sdk/blob/master/tests/conftest.py#L139-L156 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2185 (review)>, or unsubscribe <github.com/notifications/unsubscribe-auth/AEICFBROMDET7YPLNZWXPMDTB2B4VANCNFSM4YRA5GYA> .

Spoke offline.

sagemaker-bot · 2021-03-03T19:44:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 5f88d4a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T20:18:11Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 5f88d4a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T20:23:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 5f88d4a
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T20:24:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 5f88d4a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ChoiByungWook · 2021-03-03T20:26:26Z

Integ test for SageMaker Data Parallel is failing, as it uses the latest_pytorch_version in the conftest, however there is a conditional that maps to supported versions. These should ideally be decoupled.

For this PR I'll add in the changes introduced in: #2039

sagemaker-bot · 2021-03-03T20:31:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 5f88d4a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T20:41:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: caf6580
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T21:05:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: caf6580
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T21:10:08Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: caf6580
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T21:10:16Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: caf6580
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T21:15:30Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: caf6580
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T21:41:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: caf6580
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ChoiByungWook · 2021-03-03T22:11:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr

Commit ID: caf6580

Result: FAILED

Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Integration test failing due to: pytorch/vision#1938

Test was updated based on: #986

sagemaker-bot · 2021-03-03T23:12:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: aa54102
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T23:36:18Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: aa54102
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T23:40:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: aa54102
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T23:43:03Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: aa54102
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-03-03T23:49:56Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: aa54102
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

eitansela added 2 commits March 3, 2021 13:07

feature: support PyTorch 1.7 training and inference

b0a7f19

Update pytorch.json

2cc6f07

eitansela changed the title ~~feature: support PyTorch 1.7 training and inference~~ feature: support PyTorch 1.7.1 training and inference Mar 3, 2021

Merge branch 'master' into master

869e6e6

ChoiByungWook reviewed Mar 3, 2021

View reviewed changes

eitansela added 3 commits March 3, 2021 20:52

Update pytorch.json

1d7006c

Merge branch 'master' of https://github.com/eitansela/sagemaker-pytho…

6b326ab

…n-sdk

Update pytorch.json

5005984

ChoiByungWook reviewed Mar 3, 2021

View reviewed changes

Update conftest.py

5f88d4a

ChoiByungWook self-assigned this Mar 3, 2021

ChoiByungWook mentioned this pull request Mar 3, 2021

Add PT 1.7.1 support to smddp #2039

Closed

7 tasks

ChoiByungWook previously approved these changes Mar 3, 2021

View reviewed changes

update data parallel to support pytorch 1.7.1

caf6580

ChoiByungWook dismissed their stale review via caf6580 March 3, 2021 20:30

ChoiByungWook changed the title ~~feature: support PyTorch 1.7.1 training and inference~~ feature: support PyTorch 1.7.1 training, inference and data parallel Mar 3, 2021

ChoiByungWook previously approved these changes Mar 3, 2021

View reviewed changes

update data parallel integ test to use local mnist dataset

aa54102

ChoiByungWook dismissed their stale review via aa54102 March 3, 2021 23:00

ChoiByungWook approved these changes Mar 3, 2021

View reviewed changes

ChoiByungWook merged commit c13fc6d into aws:master Mar 3, 2021

ChoiByungWook mentioned this pull request Mar 4, 2021

feature: add support for TensorFlow 2.4.1 for training, inference and… #2188

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: support PyTorch 1.7.1 training, inference and data parallel #2185

feature: support PyTorch 1.7.1 training, inference and data parallel #2185

eitansela commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

ChoiByungWook Mar 3, 2021

eitansela Mar 3, 2021

eitansela Mar 3, 2021

ChoiByungWook left a comment

sagemaker-bot commented Mar 3, 2021

eitansela commented Mar 3, 2021 via email

ChoiByungWook commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

ChoiByungWook commented Mar 3, 2021 •

edited

Loading

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

ChoiByungWook commented Mar 3, 2021 •

edited

Loading

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

feature: support PyTorch 1.7.1 training, inference and data parallel #2185

feature: support PyTorch 1.7.1 training, inference and data parallel #2185

Conversation

eitansela commented Mar 3, 2021

Merge Checklist

General

Tests

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

ChoiByungWook Mar 3, 2021

Choose a reason for hiding this comment

eitansela Mar 3, 2021

Choose a reason for hiding this comment

eitansela Mar 3, 2021

Choose a reason for hiding this comment

ChoiByungWook left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

eitansela commented Mar 3, 2021 via email

ChoiByungWook commented Mar 3, 2021

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

ChoiByungWook commented Mar 3, 2021 • edited Loading

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

ChoiByungWook commented Mar 3, 2021 • edited Loading

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Mar 3, 2021

AWS CodeBuild CI Report

ChoiByungWook commented Mar 3, 2021 •

edited

Loading

ChoiByungWook commented Mar 3, 2021 •

edited

Loading