CI: build in Docker with MUSL libc for static library #736

breznak · 2019-10-30T13:52:32Z

avoid glibc with -static build, as it does not work.
https://stackoverflow.com/questions/26304531/compiling-with-static-libgcc-static-libstdc-still-results-in-dynamic-depende
adds deterministic checks for MUSL libc based systems Alpine build (musl) fails on determinism checks #707
Also bump gtest to v 1.10
for PyPI add build in Docker on Alpine linux that uses MUSL libc, which can be compiled into static transferable library

avoid setting "static" for glibc, as glibc does not really support that. Use Musl if we want static libstdc++

now possible that we use libyaml instead of yaml-cpp

if exists on host, as it likely contains incompatible build files

`

- "fixes" (by providing custom) deterministic results for MUSL libc runtime - pass NTA_LIBC_MUSL from command-line when needed (in Docker which runs on musl/Alpine) - properly build as static library (cannot be on GLIBC)

breznak

Please have a look, this is now working, but some points could be probably done better (see below).

TODO:

build PyPI for amd64 also on Docker (has working static lib build)

breznak · 2019-10-30T20:00:48Z

.github/workflows/build.yml

@@ -157,12 +157,17 @@ jobs:
  build-debug:
    name: Build and test in Debug mode
    #currently cannot run on Linux & Debug due to a bug in YAML parser: issue #218
-    runs-on: macOS-latest
+    runs-on: ubuntu-18.04


switching to ubuntu for we can use gcc-8 (c++17) there

CMakeLists.txt

breznak · 2019-10-30T20:02:48Z

CommonCompilerConfig.cmake

-	  set(stdlib_cxx ${stdlib_cxx} -stdlib=libc++)
-	endif()
-
-# TODO: investigate if we should use static or shared stdlib and gcc lib.


-static never worked properly with gcc, so we'll use musl for the binary releases.

breznak · 2019-10-30T20:03:43Z

external/gtest.cmake

@@ -34,7 +34,7 @@
 if(EXISTS "${REPOSITORY_DIR}/build/ThirdParty/share/googletest.tar.gz")
    set(URL "${REPOSITORY_DIR}/build/ThirdParty/share/googletest.tar.gz")
 else()
-    set(URL https://github.com/abseil/googletest/archive/release-1.8.1.tar.gz)
+    set(URL https://github.com/abseil/googletest/archive/release-1.10.0.tar.gz)


external/gtest.cmake

src/CMakeLists.txt

breznak · 2019-10-30T20:06:52Z

src/examples/hotgym/HelloSPTP.cpp

@@ -200,7 +195,11 @@ EPOCHS = 2; // make test faster in Debug

      SDR goldSP({COLS});
      const SDR_sparse_t deterministicSP{
+#ifdef NTA_LIBC_MUSL
+        62, 72, 73, 82, 85, 102, 263, 277, 287, 303, 306, 308, 309, 322, 337, 339, 340, 352, 370, 493, 1094, 1095, 1114, 1115, 1120, 1463, 1512, 1518, 1647, 1651, 1691, 1694, 1729, 1745, 1746, 1760, 1770, 1774, 1775, 1781, 1797, 1798, 1803, 1804, 1805, 1812, 1827, 1828, 1831, 1832, 1858, 1859, 1860, 1861, 1862, 1875, 1878, 1880, 1881, 1898, 1918, 1923, 1929, 1931, 1936, 1950, 1953, 1956, 1958, 1961, 1964, 1965, 1967, 1971, 1973, 1975, 1976, 1979, 1980, 1981, 1982, 1984, 1985, 1986, 1988, 1991, 1994, 1996, 1997, 1998, 1999, 2002, 2006, 2008, 2011, 2012, 2013, 2017, 2019, 2022, 2027, 2030


added custom deterministic results for MUSL. I don't know if we can achieve same on glibc & musl.

that is transferable and used by PyPI

dkeeney

This is ok as is but see my comments.
What I do is run a build and see where the lib ends up. then go change the link path accordingly.

CMakeLists.txt

external/gtest.cmake

as the CI does not have a TTY

use option() rather than set()

the CI does "docker run" without -it, so we need to run python tests directly, not using multi-line commands.

This reverts commit c9f2ff0.

dkeeney

COMMON_COMPILER_DEFS seems to be empty in src/CMakeLists,

It should not be empty. If it is then we are not getting any of our settings. COMMON_COMPILER_DEFS and its string version COMMON_COMPILER_DEFINITIONS_STR should be inherited from the top CMakeList.txt. The string version of it is displayed in a message statement there.

dkeeney · 2019-10-31T13:40:37Z

src/CMakeLists.txt

-  target_compile_definitions(${src_dyn_executable_hotgym} PRIVATE ${COMMON_COMPILER_DEFINITIONS})
+  if(${NTA_LIBC_MUSL})
+    target_compile_definitions(${src_dyn_executable_hotgym} PRIVATE ${COMMON_COMPILER_DEFINITIONS} -DNTA_LIBC_MUSL=${NTA_LIBC_MUSL})
+  else()


${COMMON_COMPILER_DEFINITIONS} -DNTA_LIBC_MUSL=${NTA_LIBC_MUSL})

Should not need to do that if it is already in COMMON_COMPILER_DEFINITIONS.

breznak · 2019-10-31T14:04:40Z

https://github.com/htm-community/htm.core/pull/736/checks?check_run_id=283078173#step:5:1176

I'm unable to reproduce locally (but using the same Docker), @dkeeney when you have time, would you please test for me on another system? You'll need docker, and then follow the Dockerfile instructions at the top of the file (or see the CI section), please?

dkeeney · 2019-10-31T17:40:46Z

I don't know anything about Docker so this is a good time to find out. I can put my other PR on hold for now.

Now, what you are trying to do is use the static C++ runtime library and you are doing it with this MUSL thing. I see three builds that fail...are they one for each platform?

dkeeney · 2019-10-31T20:04:20Z

I ran into a MAJOR snag. Docker Desktop for windows requires Hyper-V to work. VirtualBox does not work at all with Hyper-V turned on. I use VirtualBox as my Ubuntu installation. I cannot install Docker inside another virtualization either. There does not appear to be any way around this short of buying another computer.

dkeeney · 2019-10-31T20:48:52Z

There is an older Docker Toolbox that does run Docker under VirtualBox. I could use that but it might not be the environment that you want me to try.

breznak · 2019-11-01T07:51:26Z

I don't know anything about Docker so this is a good time to find out.

thanks a lot for willing to try that for me, David!

are trying to do is use the static C++ runtime library and you are doing it with this MUSL thing

MUSL is an alternative stdlibc++ for linux, most commonly used is glibc (only which we've been using so far), MSVC has its version, etc. GLIBC is known for its static libraries (when you link stdlibc in your code) are not truly independent, transferable. Unlike musl, where this should work ok. The intention is to use such statically linked lib as a binary for all linux distributions on PyPI

I see three builds that fail...are they one for each platform?

it's the UX here messed up, it's the same platform - x86_64 on Alpine linux (with musl C) ran inside docker. I've restarted the PR and the old results just keep showing up.

Docker Desktop for windows requires Hyper-V to work. VirtualBox does not work at all with Hyper-V turned on. I use VirtualBox as my Ubuntu installation

can you enable HyperV in BIOS, or is it a HW limitation? If you could, you'd just test the docker from Windows, and then disable HyperV again to be able to run VBox. I was not aware of such complication on Win, sorry. On linux (running directly on HW) I run both VBox and Docker containers.

https://docs.docker.com/docker-for-windows/install/#what-to-know-before-you-install

dkeeney · 2019-11-01T12:44:15Z

can you enable HyperV in BIOS

Ok, I think I understand what you are looking for. Yes it is possible to toggle HyperV in the BIOS.
I am leaving this morning for a few days in San Jose with our grandson. Will be back Monday afternoon. I will give this a try when I return.

breznak · 2019-11-01T12:50:45Z

Enjoy your weekend! 🍂
PS: I'm able to replicate the bug locally, crash in Classifier, leads to someting with NTA_CHECK macros and way how we handle exceptions. Preparing a PR that aims to streamline to process of our exceptions. (removing LogItem and the likes)

dkeeney · 2019-11-01T12:59:46Z

(removing LogItem and the likes)

Oh, very good. That is still some old original code. Could use a cleanup.

dkeeney · 2019-11-09T00:31:00Z

Do you still need me to try docker on my machine? I am reluctant to risk messing up my Ubuntu OS which is under virtualbox but if you still need me to do this I will give it a try.

breznak · 2019-11-09T22:13:31Z

Do you still need me to try docker on my machine?

so far I'm good, or I can replicate the err seen in CI. Thank you very much for offering the help!

breznak added 11 commits October 30, 2019 14:29

gtest: bump to version 1.10

4b43c49

Cmake: static glibc does not work

07438fc

avoid setting "static" for glibc, as glibc does not really support that. Use Musl if we want static libstdc++

CMake: let cmake set correct libc++ for Darwin

65675f2

CMake: default visibility: hidden

62d9331

CMake: hidden inline symbols

3ece7db

CI: run Debug build on Ubuntu

661c470

now possible that we use libyaml instead of yaml-cpp

gtest: change path to .a file

d84f40b

Docker: cleanup build/ dir

8796a30

if exists on host, as it likely contains incompatible build files

gtest: fix path also on Win

160e515

`

Docker: cleanup

452d2f9

fix static build on MUSL libc++

cf538b3

- "fixes" (by providing custom) deterministic results for MUSL libc runtime - pass NTA_LIBC_MUSL from command-line when needed (in Docker which runs on musl/Alpine) - properly build as static library (cannot be on GLIBC)

breznak commented Oct 30, 2019

View reviewed changes

breznak self-assigned this Oct 30, 2019

breznak requested a review from dkeeney October 30, 2019 20:08

breznak added 2 commits October 30, 2019 21:33

cmake: fix non-MUSL build

c0fcc64

CI: build Linux on Docker/Alpine/musl for static binary

76be0eb

that is transferable and used by PyPI

dkeeney previously approved these changes Oct 30, 2019

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

external/gtest.cmake Show resolved Hide resolved

external/gtest.cmake Show resolved Hide resolved

breznak added 2 commits October 30, 2019 23:57

CI: fix docker run without -it

fd6ae19

as the CI does not have a TTY

review feedback

c9f2ff0

use option() rather than set()

breznak dismissed dkeeney’s stale review via c9f2ff0 October 30, 2019 23:43

CI: fix running tests in Docker on CI

fee3e64

the CI does "docker run" without -it, so we need to run python tests directly, not using multi-line commands.

breznak added ready tooling labels Oct 31, 2019

breznak requested a review from dkeeney October 31, 2019 06:28

breznak changed the title ~~Avoid trying static glibc WIP~~ CI: build in Docker with MUSL libc for static library Oct 31, 2019

breznak mentioned this pull request Oct 31, 2019

Alpine build (musl) fails on determinism checks #707

Open

2 tasks

breznak added 2 commits October 31, 2019 09:31

Revert "review feedback"

29dbb04

This reverts commit c9f2ff0.

fix cmake option

e709f24

breznak closed this Oct 31, 2019

breznak reopened this Oct 31, 2019

breznak closed this Oct 31, 2019

breznak reopened this Oct 31, 2019

dkeeney approved these changes Oct 31, 2019

View reviewed changes

Log: simplify NTA_ASSERT macro

530cfce

breznak mentioned this pull request Nov 1, 2019

Cleanup exceptions #742

Merged

Merge branch 'master_community' into pypi_try_150

fc0f920

Merge branch 'master_community' into pypi_try_150

2761587

breznak added platform and removed ready labels Nov 12, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: build in Docker with MUSL libc for static library #736

CI: build in Docker with MUSL libc for static library #736

breznak commented Oct 30, 2019 •

edited

Loading

breznak left a comment •

edited

Loading

breznak Oct 30, 2019

breznak Oct 30, 2019

breznak Oct 30, 2019

breznak Oct 30, 2019

breznak Oct 31, 2019

dkeeney left a comment

dkeeney left a comment

dkeeney Oct 31, 2019

breznak commented Oct 31, 2019 •

edited

Loading

dkeeney commented Oct 31, 2019

dkeeney commented Oct 31, 2019

dkeeney commented Oct 31, 2019

breznak commented Nov 1, 2019

dkeeney commented Nov 1, 2019

breznak commented Nov 1, 2019

dkeeney commented Nov 1, 2019

dkeeney commented Nov 9, 2019

breznak commented Nov 9, 2019

CI: build in Docker with MUSL libc for static library #736

Are you sure you want to change the base?

CI: build in Docker with MUSL libc for static library #736

Conversation

breznak commented Oct 30, 2019 • edited Loading

breznak left a comment • edited Loading

Choose a reason for hiding this comment

breznak Oct 30, 2019

Choose a reason for hiding this comment

breznak Oct 30, 2019

Choose a reason for hiding this comment

breznak Oct 30, 2019

Choose a reason for hiding this comment

breznak Oct 30, 2019

Choose a reason for hiding this comment

breznak Oct 31, 2019

Choose a reason for hiding this comment

dkeeney left a comment

Choose a reason for hiding this comment

dkeeney left a comment

Choose a reason for hiding this comment

dkeeney Oct 31, 2019

Choose a reason for hiding this comment

breznak commented Oct 31, 2019 • edited Loading

dkeeney commented Oct 31, 2019

dkeeney commented Oct 31, 2019

dkeeney commented Oct 31, 2019

breznak commented Nov 1, 2019

dkeeney commented Nov 1, 2019

breznak commented Nov 1, 2019

dkeeney commented Nov 1, 2019

dkeeney commented Nov 9, 2019

breznak commented Nov 9, 2019

breznak commented Oct 30, 2019 •

edited

Loading

breznak left a comment •

edited

Loading

breznak commented Oct 31, 2019 •

edited

Loading