Adding ml.net centos6 and ubuntu 16.04 container for openmp support #94

singlis · 2019-03-07T00:47:29Z

Updates the docker containers for centos and ubuntu to build clang with openmp support.

Fixes issue #93.

with openmp support. Fixes issue dotnet#93.

MichaelSimons · 2019-03-07T01:33:50Z

@singlis - Have you considered added these ML specific components into a ML specific image that is built on top of the core images? The primary benefit of this type of layering is it keeps the core image more manageable from a size perspective as well as tracking who depends on the various special components.

shauheen · 2019-03-07T05:59:07Z

Thanks @MichaelSimons for the suggestion. However OpenMP is not really ML specific. I agree with you completely that any specific requirement should be processed as you are suggesting.

MichaelSimons · 2019-03-07T12:58:28Z

@singlis - By ML specific I mean it is only being used by the ML product. What other teams are going to be using OpenMP?

eerhardt · 2019-03-07T15:39:56Z

One issue with making a new docker image is the way we are building clang in the centos images. The new docker image would need to duplicate the clang build logic, add the openmp specific stuff into it, and re-make-install the whole clang again.

For the ubuntu images, it looks like it would be easy. @MichaelSimons, where would you suggest a new dockerfile be created?

singlis · 2019-03-07T17:48:47Z

Thank you everyone for the replies.
@MichaelSimons - yes it is specific to ML.net as I don't know of other teams that have a need for OpenMP. However it is a feature in Clang that I am enabling, so I tend to agree with Shauheen.

Eric also brings up a good issue with centos. @MichaelSimons - do you know if OpenMP can be side-loaded after clang is installed? I can try downloading only openmp and see if that can be compiled.

singlis · 2019-03-08T03:03:24Z

I confirmed on centos images that I am able to add OpenMP without having re-install Clang.

singlis · 2019-03-08T03:11:08Z

@MichaelSimons Let me know how you would like to proceed. I can create an image specifically for ML.net based on the centos 7 and ubuntu 16.04 images -- but where should they be placed? Should they go in this repository?

If you feel that this should be part of the dot net images, it maybe worth updating all images to include these changes so thats its a consistent change (its the main reason why I made this a WIP).

MichaelSimons · 2019-03-08T03:35:01Z

Since nobody is dependent on it outside of ML, my preference would be to put them in a ML specific image. I am alright with including them in this repo. Just put them in a directory underneath the os folder e.g. src/centos/6/ML/Dockerfile. If there is a more generic name other that would apply other than ML that would be great.

- Reverted changes to centos and ubuntu images

singlis · 2019-03-08T21:43:49Z

src/centos/7/mlnet/Dockerfile

@@ -0,0 +1,26 @@
+FROM mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7


@MichaelSimons - can you confirm if this is correct?
Also, how do I test this?
And should these dockerfiles be a part of the build?

You can test via .\build.ps1 -DockerfilePath "centos/7*"
Yes these should be part of the build.

You will need to add these to the manifest.json at the root of the repo in order for them to get built.

See How to modify or create a new image for details on this.

singlis · 2019-03-11T18:00:56Z

@MichaelSimons I have created ml.net containers for centos7 and ubuntu16.04. Can you review when you have a chance? Thank you very much for your help!

singlis · 2019-03-11T21:03:56Z

opensuse is failing due to the opensuse docker container 42.1 is no longer available. I can fix this in my PR if needed.

MichaelSimons

LGTM - just one minor comment.

No need to address the failing opensuse CI with your PR. I will log a separate issue for it.

MichaelSimons · 2019-03-11T21:39:32Z

src/centos/7/mlnet/Dockerfile

+RUN yum install -y \
+        perl-Data-Dumper
+
+RUN wget http://releases.llvm.org/3.9.1/openmp-3.9.1.src.tar.xz && \


A comment here would be valuable IMO just calling out the purpose of the layer - install OpenMP. This was called out in the Ubuntu Dockerfile and that was more obvious why not here as well😸.

Thanks Michael - since the opensuse issue causes the build to fail, its blocking this PR from going in.

...or maybe I dont have permissions to merge? I will update the PR per your comment.

Not everyone has permission to merge. I will merge once all legs are green except for opensuse.

MichaelSimons · 2019-03-12T19:38:08Z

FYI - the centos build leg is failing it is a network access issue. It is being investigated as time permits. We are busy with release activities today.

Step 3/3 : RUN wget ftp://sourceware.org/pub/binutils/snapshots/binutils-2.29.1.tar.xz &&     wget http://releases.llvm.org/3.9.1/cfe-3.9.1.src.tar.xz &&     wget http://releases.llvm.org/3.9.1/llvm-3.9.1.src.tar.xz &&     wget http://releases.llvm.org/3.9.1/lldb-3.9.1.src.tar.xz &&     wget http://releases.llvm.org/3.9.1/compiler-rt-3.9.1.src.tar.xz &&         tar -xf binutils-2.29.1.tar.xz &&     tar -xf llvm-3.9.1.src.tar.xz &&     mkdir llvm-3.9.1.src/tools/clang &&     mkdir llvm-3.9.1.src/tools/lldb &&     mkdir llvm-3.9.1.src/projects/compiler-rt &&     tar -xf cfe-3.9.1.src.tar.xz --strip 1 -C llvm-3.9.1.src/tools/clang &&     tar -xf lldb-3.9.1.src.tar.xz --strip 1 -C llvm-3.9.1.src/tools/lldb &&     tar -xf compiler-rt-3.9.1.src.tar.xz --strip 1 -C llvm-3.9.1.src/projects/compiler-rt &&     rm binutils-2.29.1.tar.xz &&     rm cfe-3.9.1.src.tar.xz &&     rm lldb-3.9.1.src.tar.xz &&     rm llvm-3.9.1.src.tar.xz &&     rm compiler-rt-3.9.1.src.tar.xz &&         mkdir llvmbuild &&     cd llvmbuild &&     cmake3         -DCMAKE_BUILD_TYPE=Release         -DLLVM_LIBDIR_SUFFIX=64        -DLLVM_ENABLE_EH=1         -DLLVM_ENABLE_RTTI=1         -DLLVM_BINUTILS_INCDIR=../binutils-2.29.1/include         ../llvm-3.9.1.src     &&     make -j $(($(getconf _NPROCESSORS_ONLN)+1)) &&     make install &&     cd .. &&     rm -r llvmbuild &&     rm -r llvm-3.9.1.src &&     rm -r binutils-2.29.1
 ---> Running in c4bf4a986bb2
--2019-03-12 16:20:10--  ftp:&#x2F;&#x2F;sourceware.org&#x2F;pub&#x2F;binutils&#x2F;snapshots&#x2F;binutils-2.29.1.tar.xz‌
           => 'binutils-2.29.1.tar.xz'
Resolving sourceware.org (sourceware.org)... ‌209.132.180.131‌
Connecting to sourceware.org (sourceware.org)|209.132.180.131|:21... ‌failed: Connection timed out.‌
Retrying.

The ubuntu build passed and produced the following images:

mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-207e097-20190312152303
mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-debpkg-cfdd435-20190312152303
mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-coredeps-d770b8b-20190312152303
mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-crossdeps-cfdd435-20190312152303
mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-arm64-cfdd435-20190312152303
mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-arm64-alpine-cfdd435-20190312152303
mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-cfdd435-20190312152303
mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-mlnet-207e097-20190312152303

MichaelSimons · 2019-03-13T01:14:53Z

The following centos/7 build issue was resolved.

mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-df8e152-20190312232803
mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-df8e152-20190312232803

singlis · 2019-03-13T01:53:58Z

Thanks @MichaelSimons - does it take some time for the docker images to become available? I updated my PR for ML.Net to reference the new docker containers, but it fails to find them. Here is the pull request for reference:
dotnet/machinelearning#2867

MichaelSimons · 2019-03-13T01:56:18Z

Your references are not correct, all new images from this repo are now being published to mcr. Note the fully qualified tag names I mentioned earlier - e.g. mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-df8e152-20190312232803

- Updates the docker containers for centos and ubuntu to build clang

975d5ba

with openmp support. Fixes issue dotnet#93.

singlis mentioned this pull request Mar 7, 2019

Updating MKL dotnet/machinelearning#2867

Merged

- Moved openmp support into its own docker file for ml.net.

03b06e8

- Reverted changes to centos and ubuntu images

singlis commented Mar 8, 2019

View reviewed changes

- Added ml.net to the build profile

c62fc1d

singlis changed the title ~~WIP: Updates to the docker files to support OpenMP~~ Adding ml.net centos6 and ubuntu 16.04 container for openmp support Mar 11, 2019

MichaelSimons approved these changes Mar 11, 2019

View reviewed changes

- Adding comments based on feedback.

f17a511

MichaelSimons merged commit 207e097 into dotnet:master Mar 12, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding ml.net centos6 and ubuntu 16.04 container for openmp support #94

Adding ml.net centos6 and ubuntu 16.04 container for openmp support #94

singlis commented Mar 7, 2019

MichaelSimons commented Mar 7, 2019

shauheen commented Mar 7, 2019

MichaelSimons commented Mar 7, 2019

eerhardt commented Mar 7, 2019

singlis commented Mar 7, 2019

singlis commented Mar 8, 2019

singlis commented Mar 8, 2019

MichaelSimons commented Mar 8, 2019

singlis Mar 8, 2019 •

edited

Loading

MichaelSimons Mar 8, 2019

MichaelSimons Mar 8, 2019

MichaelSimons Mar 8, 2019

singlis commented Mar 11, 2019

singlis commented Mar 11, 2019

MichaelSimons left a comment

MichaelSimons Mar 11, 2019

singlis Mar 11, 2019

singlis Mar 11, 2019

MichaelSimons Mar 11, 2019

MichaelSimons commented Mar 12, 2019 •

edited

Loading

MichaelSimons commented Mar 13, 2019

singlis commented Mar 13, 2019

MichaelSimons commented Mar 13, 2019

		@@ -0,0 +1,26 @@
		FROM mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7

Adding ml.net centos6 and ubuntu 16.04 container for openmp support #94

Adding ml.net centos6 and ubuntu 16.04 container for openmp support #94

Conversation

singlis commented Mar 7, 2019

MichaelSimons commented Mar 7, 2019

shauheen commented Mar 7, 2019

MichaelSimons commented Mar 7, 2019

eerhardt commented Mar 7, 2019

singlis commented Mar 7, 2019

singlis commented Mar 8, 2019

singlis commented Mar 8, 2019

MichaelSimons commented Mar 8, 2019

singlis Mar 8, 2019 • edited Loading

Choose a reason for hiding this comment

MichaelSimons Mar 8, 2019

Choose a reason for hiding this comment

MichaelSimons Mar 8, 2019

Choose a reason for hiding this comment

MichaelSimons Mar 8, 2019

Choose a reason for hiding this comment

singlis commented Mar 11, 2019

singlis commented Mar 11, 2019

MichaelSimons left a comment

Choose a reason for hiding this comment

MichaelSimons Mar 11, 2019

Choose a reason for hiding this comment

singlis Mar 11, 2019

Choose a reason for hiding this comment

singlis Mar 11, 2019

Choose a reason for hiding this comment

MichaelSimons Mar 11, 2019

Choose a reason for hiding this comment

MichaelSimons commented Mar 12, 2019 • edited Loading

MichaelSimons commented Mar 13, 2019

singlis commented Mar 13, 2019

MichaelSimons commented Mar 13, 2019

singlis Mar 8, 2019 •

edited

Loading

MichaelSimons commented Mar 12, 2019 •

edited

Loading