Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix for Cuda 10.1 /Cuda 10.2 related torch package installation issue in Docker. #642

Merged
merged 12 commits into from
Nov 19, 2020
Merged

Fix for Cuda 10.1 /Cuda 10.2 related torch package installation issue in Docker. #642

merged 12 commits into from
Nov 19, 2020

Conversation

harshbafna
Copy link
Contributor

Description

Fixes Cuda 10.1 /Cuda 10.2 related torch package installation issue in Docker.

Fixes #640

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • This change requires a documentation update

Feature/Issue validation/testing

Verbose logs for docker image creation with Cuda 10.1 and Cuda 10.2 based torch packages :

docker cuda 10.2.txt
docker cuda 10.1.txt

Checklist:

  • Have you made corresponding changes to the documentation?

@harshbafna harshbafna self-assigned this Aug 24, 2020
@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: 12bcdec
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: 12bcdec
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: d9e423b
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: d9e423b
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@harshbafna harshbafna changed the title Fix for Cuda 10.1 /Cuda 10.2 related torch package installation issue in Docker. [WIP] Fix for Cuda 10.1 /Cuda 10.2 related torch package installation issue in Docker. Aug 24, 2020
@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: e36f76f
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: e36f76f
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@harshbafna harshbafna changed the title [WIP] Fix for Cuda 10.1 /Cuda 10.2 related torch package installation issue in Docker. Fix for Cuda 10.1 /Cuda 10.2 related torch package installation issue in Docker. Aug 24, 2020
@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: f4b6dc9
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: f4b6dc9
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: 972a6a0
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: 972a6a0
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

chauhang
chauhang previously requested changes Sep 14, 2020
View reviewed changes
docker/Dockerfile Outdated
RUN export USE_CUDA=1

RUN pip install --no-cache-dir torch torchvision torchtext torchserve torch-model-archiver
RUN if [ "$BASE_IMAGE" != "${BASE_IMAGE#*cuda:10.1}" ]; then \
pip install --no-cache-dir torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html; \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@harshbafna Shouldn't this be in the else part based on the check?

Also please have the torchvision + cuda version a variable so that we don't have to keep changing the Dockerfile with each release

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@harshbafna ping

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@harshbafna Any update on this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chauhang: This has been taken care of. @jeremiahschung has implemented these changes. Could you please review it once again?

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: e1819bb
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: e1819bb
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: 7c38aab
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jeremiahschung
jeremiahschung reviewed Oct 13, 2020
View reviewed changes
docker/README.md Outdated Show resolved Hide resolved
jeremiahschung
jeremiahschung reviewed Oct 13, 2020
View reviewed changes
docker/README.md Outdated Show resolved Hide resolved
harshbafna and others added 2 commits October 14, 2020 13:08
@harshbafna @jeremiahschung
Added support for multiple cuda versions.
5c297b7 
Co-authored-by: jeremiahschung <70922646+jeremiahschung@users.noreply.github.com>
@harshbafna
Merge branch 'master' into issue_640
7849fbc
@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: 5c297b7
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: 7849fbc
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: 7849fbc
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: 5c297b7
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jeremiahschung
jeremiahschung reviewed Oct 14, 2020
View reviewed changes
docker/README.md Outdated Show resolved Hide resolved
@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: c88e103
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: c88e103
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@maaquib maaquib requested a review from chauhang October 26, 2020 18:06
@chauhang
Copy link
Contributor

@harshbafna What is the reason for pinning the version to PyTorch 1.6 and TorchVision to 0.7?

@jeremiahschung
Copy link
Collaborator

@harshbafna What is the reason for pinning the version to PyTorch 1.6 and TorchVision to 0.7?

@chauhang The updated code (here) fetches the latest version numbers of Torch and TorchVision from pip and combines them with the CUDA version specified by the user.

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: a7bcda3
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

maaquib
maaquib requested changes Nov 19, 2020
View reviewed changes
Copy link
Collaborator

@maaquib maaquib left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

DOCKER_BUILDKIT=1 docker build --no-cache --file Dockerfile --build-arg BASE_IMAGE=nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04 -t torchserve:latest .

docker run --gpus=all --rm -it -p 8080:8080 -p 8081:8081 torchserve:latest
...
Number of GPUs: 1
Number of CPUs: 64
...

DOCKER_BUILDKIT=1 docker build --no-cache --file Dockerfile --build-arg BASE_IMAGE=nvidia/cuda:10.1-cudnn7-runtime-ubuntu18.04 --build-arg CUDA_VERSION=cu101 -t torchserve:latest .

docker run --rm --gpus=all -it -p 8080:8080 -p 8081:8081 torchserve:latest
...
Number of GPUs: 1
Number of CPUs: 64
...

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: a7bcda3
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@maaquib maaquib merged commit 48a1f46 into master Nov 19, 2020
@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-cpu
  • Commit ID: a7bcda3
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-neo-ci-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: torch-serve-build-gpu
  • Commit ID: a7bcda3
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@harshbafna harshbafna deleted the issue_640 branch December 11, 2020 06:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jeremiahschung jeremiahschung jeremiahschung left review comments

@dhaniram-kshirsagar dhaniram-kshirsagar dhaniram-kshirsagar left review comments

@dhanainme dhanainme dhanainme approved these changes

@maaquib maaquib maaquib approved these changes

@chauhang chauhang Awaiting requested review from chauhang

Assignees

@harshbafna harshbafna

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Cuda not available in pytorch/torchserve:0.2.0-cuda10.1-cudnn7-runtime Docker image
7 participants
@harshbafna @sagemaker-neo-ci-bot @chauhang @jeremiahschung @maaquib @dhaniram-kshirsagar @dhanainme