Dockerfile and accompanying documentation #970

bhagemeier · 2022-04-28T14:42:08Z

The Dockerfile provides some flexibility in selecting which version of HeAT should be inside the Docker image. Also, one can choose whether to install from source or from PyPI.

Description

Provide a Dockerfile and short README about how to use it.

Issue/s resolved: #897

Changes proposed:

add docker/{Dockerfile, README.md}

Type of change

Repository structure extension, no code change.

Memory requirements

n/a

Performance

n/a

Due Diligence

All split configurations tested
Multiple dtypes tested in relevant functions
Documentation updated (if needed)
Updated changelog.md under the title "Pending Additions"

Does this change modify the behaviour of other functions? If so, which?

no

skip ci

ghost · 2022-04-28T14:43:26Z

👇 Click on the image for a new way to code review

Legend

codecov · 2022-04-28T14:44:54Z

Codecov Report

Merging #970 (072d504) into main (6ddc295) will decrease coverage by 0.09%.
The diff coverage is n/a.

❗ Current head 072d504 differs from pull request most recent head 2cd4785. Consider uploading reports for the commit 2cd4785 to get more accurate results

@@            Coverage Diff             @@
##             main     #970      +/-   ##
==========================================
- Coverage   91.85%   91.77%   -0.09%     
==========================================
  Files          74       72       -2     
  Lines       10712    10485     -227     
==========================================
- Hits         9840     9623     -217     
+ Misses        872      862      -10

Flag	Coverage Δ
unit	`91.77% <ø> (-0.09%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 7 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

coquelin77 · 2022-10-26T08:37:08Z

another container to try: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch

docker/Dockerfile

coquelin77

looks good so far. only the mpi4py suggestion jumps out to me

ClaudiaComito · 2022-11-07T09:10:27Z

@Mystic-Slice @shahpratham @JedrzejMosiezny if you happen to have time, can you try this out as well? Thanks a lot!

ClaudiaComito

Thanks @bhagemeier for providing this. My main problem with this PR is that I don't understand / can't follow the README.

ClaudiaComito · 2022-11-14T04:32:13Z

docker/README.md

+The [Dockerfile](./Dockerfile) guiding the build of the Docker image is located in this
+directory. It is typically most convenient to `cd` over here and run the Docker build as:
+
+	$ docker build .


Instructions to run the container needed here

ClaudiaComito · 2022-11-14T04:33:31Z

docker/README.md

+The resulting image (ID) should then be tagged for subsequent upload (push) to a
+repository, for example:
+
+	$ docker tag ea0a1040bf8a ghcr.io/helmholtz-analytics/heat:1.2.0_torch1.11_cuda11.5_py3.9
+	$ docker push ghcr.io/helmholtz-analytics/heat:1.2.0_torch1.11_cuda11.5_py3.9
+
+Please ensure that you push the same tag that you just created.


Is this for the devs or the users? I'm not sure what the use case is.

For me it is also unclear whether I have to run this when I just want to use Heat in containerized version or whether this just explains what had to be done in order to create the container.

ClaudiaComito · 2022-11-14T04:40:26Z

docker/README.md

+## Building for HPC
+
+With HeAT being a native HPC library, one would naturally want to build the container
+image also for HPC systems, such as the ones available at [Juelich Supercomputing Centre
+(JSC)](https://www.fz-juelich.de/jsc/ "Juelich Supercomputing Centre").
+
+HPC centres may run a choice of Apptainer or Singularity, which may incur limitations to
+the flexibility of building images. For instance, the Singularity Image Builder (SIB)
+does not work with the arguments mentioned above, such that these will have to be
+avoided.
+
+However, SIB is capable of using just about any available Docker image from any
+registry, such that a specific Singularity image can be built by simply referencing the
+available image. SIB is thus used as a conversion tool.
+
+A simple `Dockerfile` (in addition to the one above) to be used with SIB could look like
+this:
+
+	FROM ghcr.io/helmholtz-analytics/heat:1.2.0_torch1.11_cuda11.5_py3.9
+


I don't get it. I would like to start containerized Heat from source on a specific branch. What am I supposed to do? Should I load some modules first?

FROM ghcr.io/helmholtz-analytics/heat:1.2.0_torch1.11_cuda11.5_py3.9 -bash: FROM: command not found

JuanPedroGHM · 2023-01-24T13:40:19Z

Added a working multi-node example on the Docker readme and a singularity definition file. Both worked on HoreKa. Would be good if someone could confirm that it works on other systems as well.

The Dockerfile provides some flexibility in selecting which version of HeAT should be inside the Docker image. Also, one can choose whether to install from source or from PyPI.

Some code sections had a mix of spaces and tabs, which have now been convertd into tabs.

Use pytorch 1.11 Fix problem with CUDA package repo keys

NVidia images come with support for HPC systems desirable for our uses. They work a little differently internally and required some changes. The tzdata configuration configures the CET/CEST timezone, which seems to be required when installing additional packages. There is an issue with pip caches in the image, which led to the final cache purge to fail in the PyPI release based build. This is fixed through a final invocation of true.

… docker readme

for more information, see https://pre-commit.ci

mrfh92 · 2023-04-18T13:42:56Z

docker/README.md

+Dockerfile. This method does not support build arguments, so version, branch and type of installation have to
+changed in the definition file.
+
+    $ singularity build heat_1.2.0_torch.11_cuda11.5_py3.9.sif heat-singularity-image.def


I tried this one on CARO, but got

FATAL: You must be the root user, however you can use --remote or --fakeroot to build from a Singularity recipe file

Option --remote however yields

FATAL: Unable to submit build job: no authentication token, log in with `singularity remote login`

and option --fakeroot yields

FATAL: could not use fakeroot: no mapping entry found in /etc/subuid for hopp_fa

Is this just caused by the configuration on our cluster or can you add some hints here how to resolve this problem in general?

docker/README.md

mrfh92 · 2023-04-18T13:46:16Z

docker/README.md

+The resulting image (ID) should then be tagged for subsequent upload (push) to a
+repository, for example:
+
+	$ docker tag ea0a1040bf8a ghcr.io/helmholtz-analytics/heat:1.2.0_torch1.11_cuda11.5_py3.9
+	$ docker push ghcr.io/helmholtz-analytics/heat:1.2.0_torch1.11_cuda11.5_py3.9
+
+Please ensure that you push the same tag that you just created.


For me it is also unclear whether I have to run this when I just want to use Heat in containerized version or whether this just explains what had to be done in order to create the container.

mrfh92 · 2023-04-18T13:47:13Z

docker/README.md

+A simple `Dockerfile` (in addition to the one above) to be used with SIB could look like
+this:
+
+	FROM ghcr.io/helmholtz-analytics/heat:1.2.0_torch1.11_cuda11.5_py3.9


Do we need to replace the Dockerfile from above (both have the same name)?

mrfh92 · 2023-04-18T13:48:16Z

docker/README.md

+
+The invocation to build the image would be:
+
+	$ sib upload ./Dockerfile heat_1.2.0_torch.11_cuda11.5_py3.9


Maybe I am just a bit confused (and could not try out the commands because sib is not available on CARO), but where do I upload and from where do I download here?

mrfh92 · 2023-04-18T13:49:11Z

docker/README.md

+	$ sib build --recipe-name heat_1.2.0_torch.11_cuda11.5_py3.9
+	$ sib download --recipe-name heat_1.2.0_torch.11_cuda11.5_py3.9
+
+### Apptainer (formerly singularity)


If this is the variant most likely to be used in HPC environments, we maybe should put this paragraph at the beginning and could shift docker and sib to an "expert" section?

docker/README.md

mrfh92 · 2023-04-24T07:06:13Z

When I run

singularity build heat_1.2.0_torch.11_cuda11.5_py3.9.sif heat-singularity-image.def

i get the following error:

INFO:    User not listed in /etc/subuid, trying root-mapped namespace
INFO:    The %post section will be run under fakeroot
FATAL:   Unable to build from heat-singularity-image.def: while parsing definition: heat-singularity-image.def: failed to parse deffile header: header key cat heat-nvidia.def had no val

Is there missing some file heat-nvidia.def?

github-actions · 2023-04-24T16:09:02Z

Thank you for the PR!

.github/workflows/docker.yml

docker/README.md

mrfh92 · 2023-05-08T12:18:42Z

I tried it out. The build was no problem on my workstation, however running resulted in the following problem:

hopp_fa@sc-030122l:~/heat/docker$ singularity run --nv heat.sif /bin/bash
INFO:    underlay of /usr/bin/nvidia-smi required more than 50 (492) bind mounts

=============
== PyTorch ==
=============

NVIDIA Release 22.05 (build 37432893)
PyTorch Version 1.12.0a0+8a1a93a

Container image Copyright (c) 2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Copyright (c) 2014-2022 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

NOTE: CUDA Forward Compatibility mode ENABLED.
  Using CUDA 11.7 driver version 515.43.04 with kernel driver version 470.182.03.
  See https://docs.nvidia.com/deploy/cuda-compatibility/ for details.

Apptainer> python
Python 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import heat 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/heat/__init__.py", line 5, in <module>
    from .core import *
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/heat/core/__init__.py", line 5, in <module>
    from .arithmetics import *
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/heat/core/arithmetics.py", line 7, in <module>
    import torch
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/torch/__init__.py", line 751, in <module>
    from .functional import *  # noqa: F403
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/torch/functional.py", line 8, in <module>
    import torch.nn.functional as F
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/torch/nn/__init__.py", line 1, in <module>
    from .modules import *  # noqa: F403
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/torch/nn/modules/__init__.py", line 2, in <module>
    from .linear import Identity, Linear, Bilinear, LazyLinear
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 6, in <module>
    from .. import functional as F
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 18, in <module>
    from .._jit_internal import boolean_dispatch, _overload, BroadcastingList1, BroadcastingList2, BroadcastingList3
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/torch/_jit_internal.py", line 24, in <module>
    import torch.distributed.rpc
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/torch/distributed/__init__.py", line 55, in <module>
    from .distributed_c10d import *  # noqa: F403
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 188, in <module>
    reduce_op = _reduce_op()
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 176, in __init__
    for k, v in ReduceOp.__members__.items():
AttributeError: type object 'torch._C._distributed_c10d.ReduceOp' has no attribute '__members__'
>>> import heat as ht
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/heat/__init__.py", line 5, in <module>
    from .core import *
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/heat/core/__init__.py", line 5, in <module>
    from .arithmetics import *
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/heat/core/arithmetics.py", line 7, in <module>
    import torch
  File "/home/hopp_fa/.local/lib/python3.8/site-packages/torch/__init__.py", line 231, in <module>
    __all__ += [name for name in dir(_C)
NameError: name '_C' is not defined
>>>

Co-authored-by: Claudia Comito <39374113+ClaudiaComito@users.noreply.github.com>

github-actions · 2023-05-08T13:08:32Z

Thank you for the PR!

github-actions · 2023-05-08T13:13:09Z

Thank you for the PR!

mrfh92 · 2023-05-08T13:15:03Z

I tested on our cluster as well. I needed to modify the above script a bit:

#!/bin/bash
#SBATCH --time 0:10:00
#SBATCH --nodes 2
#SBATCH --tasks-per-node 2

module load singularity # make singularity available (may be different on each system) 
srun --mpi="pmi2" singularity exec <path_to_your_heat_image>/heat.sif bash -c "cd ~/heat/examples/lasso; python demo.py"

I dont know what --bind /scratch is supposed to do, but it produced errors on my system. Without it, everything is fine...

github-actions · 2023-05-22T08:00:46Z

Thank you for the PR!

JuanPedroGHM · 2023-05-30T08:20:51Z

I tested on our cluster as well. I needed to modify the above script a bit:
#!/bin/bash
#SBATCH --time 0:10:00
#SBATCH --nodes 2
#SBATCH --tasks-per-node 2

module load singularity # make singularity available (may be different on each system) 
srun --mpi="pmi2" singularity exec <path_to_your_heat_image>/heat.sif bash -c "cd ~/heat/examples/lasso; python demo.py"
I dont know what --bind /scratch is supposed to do, but it produced errors on my system. Without it, everything is fine...

I removed the flag from the docs, missed it when creating the template.

github-actions · 2023-05-30T08:23:15Z

Thank you for the PR!

github-actions · 2023-05-31T04:28:33Z

Thank you for the PR!

github-actions · 2023-06-07T08:38:36Z

Thank you for the PR!

ClaudiaComito

Thanks @bhagemeier @JuanPedroGHM @coquelin77 ! Merging

github-actions · 2023-06-20T12:04:06Z

Thank you for the PR!

ClaudiaComito added the PR talk label May 11, 2022

ClaudiaComito removed the PR talk label Jun 1, 2022

bhagemeier marked this pull request as ready for review October 5, 2022 13:58

bhagemeier requested review from ClaudiaComito and mtar October 5, 2022 13:59

ClaudiaComito added the PR talk label Oct 24, 2022

coquelin77 reviewed Oct 26, 2022

View reviewed changes

docker/Dockerfile Show resolved Hide resolved

bhagemeier requested a review from coquelin77 October 26, 2022 08:42

coquelin77 reviewed Oct 26, 2022

View reviewed changes

ClaudiaComito added the v1.2.1 label Oct 28, 2022

ClaudiaComito requested changes Nov 14, 2022

View reviewed changes

ClaudiaComito removed the v1.2.1 label Jan 14, 2023

bhagemeier and others added 9 commits March 3, 2023 10:08

Dockerfile and accompanying documentation

3664934

The Dockerfile provides some flexibility in selecting which version of HeAT should be inside the Docker image. Also, one can choose whether to install from source or from PyPI.

README.md describing containerization

6dc85a8

Fix indentation in README.md

56211be

Some code sections had a mix of spaces and tabs, which have now been convertd into tabs.

Docker support

aea60ec

Use pytorch 1.11 Fix problem with CUDA package repo keys

Ensure mpi4py installation from source

00025f9

Provide sample file for Singularity

7d870a7

feat: singularity definition file and slurm multi-node example in the…

351f9c1

… docker readme

docs: quick_start.md has a docker section with link to docker readme

64b474f

JuanPedroGHM force-pushed the features/897-containerization branch from 6a69b56 to 64b474f Compare March 3, 2023 09:11

[pre-commit.ci] auto fixes from pre-commit.com hooks

e7e0716

for more information, see https://pre-commit.ci

ClaudiaComito added this to the 1.3.0 milestone Mar 29, 2023

Merge branch 'main' into features/897-containerization

dace7c6

mrfh92 reviewed Apr 18, 2023

View reviewed changes

ci: docker cleanup

072d504

ci: build docker action, updated docs

5fbd116