Make finding Python in build system more robust, update Docker container #2672

nilsvu · 2020-12-17T15:11:39Z

Proposed changes

Switch to CMake's FindPython
Install Pybind11 through pip. This should work on all systems that
support Python. It means we can use Pybind11's CMake tools,
don't have to bundle them and don't need special finding-code.

Upgrade instructions

To specify a Python executable when configuring CMake, use the variable Python_EXECUTABLE instead of the old PYTHON_EXECUTABLE, e.g.:
```
cmake -D Python_EXECUTABLE=/usr/bin/python3 $SPECTRE_HOME
```
More information on configuring CMake with Python: https://cmake.org/cmake/help/latest/module/FindPython.html
If you use the Python bindings at all, install Pybind11 via pip:
```
pip install pybind11
```
More information on installing Pybind11: https://pybind11.readthedocs.io/en/stable/installing.html

CMake should find the installation automatically. If you run into trouble, make sure you use the Python executable from the environment that you have installed pybind11 in, then file an issue. More information on finding Pybind11 in CMake: https://pybind11.readthedocs.io/en/stable/cmake/index.html

Code review checklist

The code is documented and the documentation renders correctly. Run
make doc to generate the documentation locally into BUILD_DIR/docs/html.
Then open index.html.
The code follows the stylistic and code quality guidelines listed in the
code review guide.
The PR lists upgrade instructions and is labeled bugfix or
major new feature if appropriate.

Further comments

nilsdeppe · 2020-12-17T17:42:02Z

cmake/SetupPypp.cmake

+
+# We can't rely on CMake 3.14 quite yet so we backport the `Python::NumPy`
+# imported target that FindPython in CMake 3.14+ could also provide.
+find_package(NumPy 1.10 REQUIRED)


Do we/can we check that SciPy was correctly found with finding NumPy? I believe nowadays it is, but I don't know when that changed.

Probably a good idea to check that, added a fixup

nilsdeppe

LGTM! Please go ahead and squash :)

nilsdeppe · 2020-12-17T23:32:50Z

@nilsleiffischer this needs a rebase to fix a (hopefully minor) conflict

kidder · 2020-12-17T23:36:24Z

@wthrowe do you want to test this out on your machine?
I need to test this on my machine as well

wthrowe · 2020-12-18T02:01:14Z

-- Python: /usr/bin/python3.7
-- Python Version: 3.7.9
-- BUILD_PYTHON_BINDINGS: YES
-- Found clang-tidy: /usr/lib/llvm/10/bin/clang-tidy
-- Doxygen: /usr/bin/doxygen
-- Configuring done
CMake Error at /usr/share/cmake/Modules/FindPython/Support.cmake:2571 (add_library):
  Target "PyDataStructures" links to target "pybind11::headers" but the
  target was not found.  Perhaps a find_package() call is missing for an
  IMPORTED target, or an ALIAS target is missing?
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPython.cmake:364 (__Python_add_library)
  cmake/SpectreSetupPythonPackage.cmake:91 (Python_add_library)
  src/DataStructures/Python/CMakeLists.txt:6 (spectre_python_add_module)

Followed by a bunch of similar errors.

nilsvu · 2020-12-18T02:02:37Z

@wthrowe could you post the full cmake trace please?

wthrowe · 2020-12-18T02:07:27Z

$ cmake -DCHARM_ROOT=/usr/include/charm -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_COMPILER=clang-10 -DCMAKE_CXX_COMPILER=clang++-10 -DCMAKE_Fortran_COMPILER=gfortran -DCMAKE_C_FLAGS=-fPIC\ -march=native\  -DCMAKE_CXX_FLAGS=-fPIC\ -fdiagnostics-color=always\ -ferror-limit=1\ -ftemplate-backtrace-limit=0\ -march=native\  -DCMAKE_Fortran_FLAGS=-march=native\  -DCMAKE_EXE_LINKER_FLAGS=-march=native\ -no-pie\  -DCMAKE_SHARED_LINKER_FLAGS=-march=native\ -no-pie\  -Dpybind11_INCLUDE_DIRS=/usr/include/python3.7m -DBUILD_PYTHON_BINDINGS=YES /home/wthrowe/spectre
-- SpECTRE release version: 2020.12.07
-- The CXX compiler identification is Clang 10.0.1
-- The C compiler identification is Clang 10.0.1
-- The Fortran compiler identification is GNU 7.5.0
-- Check for working CXX compiler: /usr/lib/llvm/10/bin/clang++-10
-- Check for working CXX compiler: /usr/lib/llvm/10/bin/clang++-10 - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /usr/lib/llvm/10/bin/clang-10
-- Check for working C compiler: /usr/lib/llvm/10/bin/clang-10 - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working Fortran compiler: /usr/bin/gfortran
-- Check for working Fortran compiler: /usr/bin/gfortran - works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /usr/bin/gfortran supports Fortran 90
-- Checking whether /usr/bin/gfortran supports Fortran 90 - yes
-- Found Git: /usr/bin/git (found version "2.26.2") 
-- Found Yapf: /usr/bin/yapf (found version "0.30.0") 
-- yapf found at: /usr/bin/yapf
-- yapf version: 0.30.0
-- Found Python: /usr/bin/python3.7 (found version "3.7.9") found components: Interpreter 
-- CMAKE_BUILD_TYPE: Debug
-- Found Boost: /usr/include (found suitable version "1.72.0", minimum required is "1.60.0") found components: program_options 
-- Boost include: /usr/include
-- Boost libraries: /usr/lib64/libboost_program_options-mt.so
-- Could not find ccache
-- Found Charm: /usr/bin/charmc (found suitable exact version "6.10.2") 
-- Charm++ built with shared memory parallelism
-- Checking for broken std::array<..., 0>
-- Checking for broken std::array<..., 0> -- works
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.2") 
-- Checking for module 'blas'
--   Found blas, version 3.8.0
-- BLAS libs: /usr/lib64/libblas.so
-- BLAS vendor is probably not OpenBLAS. Make sure it doesn't try to do multithreading that might conflict with Charm++ parallelism.
-- Checking for module 'lapack'
--   Found lapack, version 3.8.0
-- Found LAPACK: /usr/lib64/liblapack.so  
-- LAPACK libs: /usr/lib64/liblapack.so
-- Found LIBXSMM: /usr/include (found version "1.8.1") 
-- LIBXSMM libs: /usr/lib64/libxsmm.so
-- LIBXSMM incl: /usr/include
-- LIBXSMM vers: 1.8.1
-- Found Blaze: /usr/include (found suitable version "3.7", minimum required is "3.5") 
-- Blaze incl: /usr/include
-- Blaze vers: 3.7
-- Found Brigand: /usr/include  
-- Brigand include: /usr/include
-- Found Catch: /usr/include/catch2 (found suitable version "2.9.1", minimum required is "2.8.0") 
-- Catch include: /usr/include/catch2
-- Catch version: 2.9.1
-- Google Benchmark libs: /usr/lib64/libbenchmark.so
-- Google Benchmark incl: /usr/include/benchmark
-- Found GSL: /usr/include (found version "2.5") 
-- GSL libs: /usr/lib64/libgsl.so;/usr/lib64/libgslcblas.so
-- GSL incl: /usr/include
-- GSL vers: 2.5
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /usr/lib64/libhdf5.so;/usr/lib64/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so (found version "1.10.5") found components: C 
-- HDF5 libs: /usr/lib64/libhdf5.so/usr/lib64/libz.so/usr/lib64/libdl.so/usr/lib64/libm.so
-- HDF5 incl: /usr/include
-- HDF5 vers: 1.10.5
-- Found JEMALLOC: /usr/include (found version "..") 
-- jemalloc libs: /usr/lib64/libjemalloc.so
-- jemalloc incl: /usr/include
-- jemalloc vers: ..
-- Found Python: /usr/bin/python3.7 (found version "3.7.9") found components: Interpreter Development 
-- Found PythonInterp: /usr/bin/python3.7 (found version "3.7.9") 
-- Found PythonLibs: /usr/lib64/libpython3.7m.so
-- Pybind11 include: /usr/include
-- Found Libsharp: /usr/include  
-- libsharp libs: /usr/lib64/libsharp.a
-- libsharp libs: /usr/lib64/libfftpack.a
-- libsharp libs: /usr/lib64/libc_utils.a
-- libsharp incl: /usr/include
-- Found YAMLCPP: /usr/include  
-- yaml-cpp libs: /usr/lib64/libyaml-cpp.so
-- yaml-cpp incl: /usr/include
-- clang-tidy: /usr/lib/llvm/10/bin/clang-tidy
-- iwyu: Cannot use include-what-you-use with precompiled header. Pass USE_PCH=OFF to CMake to disable the precompiled header.
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.17") found components: doxygen missing components: dot
-- Found Python: /usr/bin/python3.7 (found version "3.7.9") found components: Interpreter 
-- Could NOT find PY_bs4 (missing: PY_BS4) 
-- Could NOT find PY_pybtex (missing: PY_PYBTEX) 
CMake Warning at cmake/SetupDoxygen.cmake:64 (message):
  Doxygen documentation postprocessing is disabled because Python
  dependencies were not found:
Call Stack (most recent call first):
  CMakeLists.txt:117 (include)


CMake Warning at cmake/SetupDoxygen.cmake:67 (message):
  BeautifulSoup4 missing.  Install with: pip install beautifulsoup4
Call Stack (most recent call first):
  CMakeLists.txt:117 (include)


CMake Warning at cmake/SetupDoxygen.cmake:71 (message):
  Pybtex missing.  Install with: pip install pybtex
Call Stack (most recent call first):
  CMakeLists.txt:117 (include)


-- Could NOT find PY_coverxygen (missing: PY_COVERXYGEN) 
-- Found Python: /usr/lib64/libpython3.7m.so (found version "3.7.9") found components: Development 
-- Found Python: /usr/bin/python3.7 (found version "3.7.9") found components: Interpreter 
-- Found NumPy: /usr/lib/python3.7/site-packages/numpy/core/include (found suitable version "1.19.1", minimum required is "1.10") 
-- NumPy ver. 1.19.1 found (include: /usr/lib/python3.7/site-packages/numpy/core/include)
-- NumPy incl: /usr/lib/python3.7/site-packages/numpy/core/include
-- NumPy vers: 1.19.1
-- Found PY_scipy: /usr/lib/python3.7/site-packages/scipy  
-- 
Useful Information:
-- Git Branch: HEAD
-- Git Hash: v2020.12.07
-- Build Directory: /home/wthrowe/spectre-work
-- Source Directory: /home/wthrowe/spectre
-- Bin Directory: /home/wthrowe/spectre-work/bin/
-- CMake Modules Path: /home/wthrowe/spectre/cmake/usr/share/cmake/pybind11
-- CMAKE_CXX_FLAGS: -fPIC -fdiagnostics-color=always -ferror-limit=1 -ftemplate-backtrace-limit=0 -march=native
-- CMAKE_CXX_LINK_FLAGS:  -L/usr/lib64 -fuse-ld=gold
-- CMAKE_CXX_FLAGS_DEBUG: -g -DSPECTRE_DEBUG
-- CMAKE_CXX_FLAGS_RELEASE: -O3 -DNDEBUG
-- CMAKE_EXE_LINKER_FLAGS: -march=native -no-pie
-- CMAKE_BUILD_TYPE: Debug
-- CMAKE_CXX_COMPILER: /usr/lib/llvm/10/bin/clang++-10
-- BUILD_SHARED_LIBS: 
-- USE_PCH: ON
-- ASAN: OFF
-- UBSAN_UNDEFINED: OFF
-- UBSAN_INTEGER: OFF
-- USE_SYSTEM_INCLUDE: ON
-- Python: /usr/bin/python3.7
-- Python Version: 3.7.9
-- BUILD_PYTHON_BINDINGS: YES
-- Found clang-tidy: /usr/lib/llvm/10/bin/clang-tidy
-- Doxygen: /usr/bin/doxygen
-- Configuring done
CMake Error at /usr/share/cmake/Modules/FindPython/Support.cmake:2571 (add_library):
  Target "PyDataStructures" links to target "pybind11::headers" but the
  target was not found.  Perhaps a find_package() call is missing for an
  IMPORTED target, or an ALIAS target is missing?
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPython.cmake:364 (__Python_add_library)
  cmake/SpectreSetupPythonPackage.cmake:91 (Python_add_library)
  src/DataStructures/Python/CMakeLists.txt:6 (spectre_python_add_module)


CMake Error at /usr/share/cmake/Modules/FindPython/Support.cmake:2571 (add_library):
  Target "PyDomainCreators" links to target "pybind11::headers" but the
  target was not found.  Perhaps a find_package() call is missing for an
  IMPORTED target, or an ALIAS target is missing?
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPython.cmake:364 (__Python_add_library)
  cmake/SpectreSetupPythonPackage.cmake:91 (Python_add_library)
  src/Domain/Creators/Python/CMakeLists.txt:6 (spectre_python_add_module)


CMake Error at /usr/share/cmake/Modules/FindPython/Support.cmake:2571 (add_library):
  Target "PyDomain" links to target "pybind11::headers" but the target was
  not found.  Perhaps a find_package() call is missing for an IMPORTED
  target, or an ALIAS target is missing?
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPython.cmake:364 (__Python_add_library)
  cmake/SpectreSetupPythonPackage.cmake:91 (Python_add_library)
  src/Domain/Python/CMakeLists.txt:6 (spectre_python_add_module)


CMake Error at /usr/share/cmake/Modules/FindPython/Support.cmake:2571 (add_library):
  Target "PyInformer" links to target "pybind11::headers" but the target was
  not found.  Perhaps a find_package() call is missing for an IMPORTED
  target, or an ALIAS target is missing?
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPython.cmake:364 (__Python_add_library)
  cmake/SpectreSetupPythonPackage.cmake:91 (Python_add_library)
  src/Informer/Python/CMakeLists.txt:6 (spectre_python_add_module)


CMake Error at /usr/share/cmake/Modules/FindPython/Support.cmake:2571 (add_library):
  Target "PyH5" links to target "pybind11::headers" but the target was not
  found.  Perhaps a find_package() call is missing for an IMPORTED target, or
  an ALIAS target is missing?
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPython.cmake:364 (__Python_add_library)
  cmake/SpectreSetupPythonPackage.cmake:91 (Python_add_library)
  src/IO/H5/Python/CMakeLists.txt:6 (spectre_python_add_module)


CMake Error at /usr/share/cmake/Modules/FindPython/Support.cmake:2571 (add_library):
  Target "PyGeneralRelativitySolutions" links to target "pybind11::headers"
  but the target was not found.  Perhaps a find_package() call is missing for
  an IMPORTED target, or an ALIAS target is missing?
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPython.cmake:364 (__Python_add_library)
  cmake/SpectreSetupPythonPackage.cmake:91 (Python_add_library)
  src/PointwiseFunctions/AnalyticSolutions/GeneralRelativity/Python/CMakeLists.txt:6 (spectre_python_add_module)


CMake Error at /usr/share/cmake/Modules/FindPython/Support.cmake:2571 (add_library):
  Target "PyEquationsOfState" links to target "pybind11::headers" but the
  target was not found.  Perhaps a find_package() call is missing for an
  IMPORTED target, or an ALIAS target is missing?
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPython.cmake:364 (__Python_add_library)
  cmake/SpectreSetupPythonPackage.cmake:91 (Python_add_library)
  src/PointwiseFunctions/Hydro/EquationsOfState/Python/CMakeLists.txt:6 (spectre_python_add_module)


-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

nilsvu · 2020-12-18T02:44:31Z

@wthrowe Have you installed pybind11 via pip? Why are you passing pybind11_INCLUDE_DIRS manually to cmake?

nilsdeppe · 2020-12-18T04:10:57Z

I'll add that something like pybind11_ROOT should be supported. I don't think we can reasonably force people to install something through pip.

wthrowe · 2020-12-18T05:24:29Z

@wthrowe Have you installed pybind11 via pip?

No, I installed it from my distro's package.

Why are you passing pybind11_INCLUDE_DIRS manually to cmake?

This is a good question. I assume I added it for some reason in the past, but it doesn't seem to be necessary on develop. Removing it did not fix the errors, though.

nilsvu · 2020-12-18T09:42:08Z

I'll add that something like pybind11_ROOT should be supported. I don't think we can reasonably force people to install something through pip.

Yes it definitely should be supported. I also installed via brew, should work fine.

@wthrowe which version of pybind11 did you install? Please try 2.6.0 or later (they added FindPython support on Aug 19 pybind/pybind11#2370).

nilsdeppe · 2020-12-18T19:46:40Z

Question: is there a way we can require a specific version and newer? It sounds like part of the trouble is that? I noticed you updated the docs, not sure if the find(pybind11) command supports requiring a version number

nilsvu · 2020-12-18T20:35:09Z

Sure, added the version requirement to the find_package(pybind11) command (squashed it in).

nilsdeppe

LGTM thanks!

nilsdeppe

@kidder have you tested this on your machine?

nilsdeppe · 2020-12-18T22:24:34Z

.github/workflows/Tests.yaml

@@ -114,7 +114,7 @@ jobs:
      || github.ref != 'refs/heads/develop'
    runs-on: ubuntu-latest
    container:
-      image: sxscollaboration/spectrebuildenv:latest
+      image: nilsleiffischer/spectrebuildenv:pybind11_with_pip


So we don't forget: this needs to get set back to the right container

nilsvu · 2020-12-20T17:26:26Z

Added another commit that cleans up the link dependencies for the Pybindings in the build system.

wthrowe · 2020-12-21T18:29:31Z

@wthrowe which version of pybind11 did you install? Please try 2.6.0 or later (they added FindPython support on Aug 19 pybind/pybind11#2370).

I was using 2.5.0. It works after upgrading to 2.6.0.

kidder · 2020-12-22T01:14:19Z

works for me; needs a rebase

- Switch to CMake's FindPython - Install Pybind11 through pip. This should work on all systems that support Python. It means we can use Pybind11's CMake tools, don't have to bundle them and don't need special finding-code.

Add missing link dependencies and link explicitly with pybind11 when including its headers.

This ensures that pre-built binaries are used if possible.

nilsvu · 2020-12-22T17:17:57Z

@nilsdeppe @kidder @wthrowe rebased, and added a few more container updates so we don't have to rebuild the container too many times. CI runs over the new container successfully here. I'll push the new container to sxscollaboration/spectrebuildenv:latest once you have approved this PR, so we can rerun CI and merge.

nilsdeppe · 2020-12-22T17:40:01Z

containers/Dockerfile.buildenv

-    && ./build LIBS multicore-linux-x86_64 gcc ${PARALLEL_MAKE_ARG} -g -O0 \
-    && ./build charm++ multicore-linux-x86_64 clang ${PARALLEL_MAKE_ARG} -g -O0 \
-    && ./build LIBS multicore-linux-x86_64 clang ${PARALLEL_MAKE_ARG} -g -O0 \
+    && ./build LIBS multicore-linux-x86_64 gcc \


why build the shared libs here? Linux allows you to mix shared and static libs as long as the static libs are built with -fPIC and our build system doesn't look for Charm++'s shared libs (because they stick them in the wrong directory)

The Charm docs say to compile shared libs: https://github.com/UIUC-PPL/charm#building-dynamic-libraries. Then we can pass -charm-shared to charmc --print-building-blocks and get the compiler configuration for shared libs. If we don't do both (have the Charm shared libs compiled and pass the -charm-shared flag) then I don't think they guarantee that things will work.

Sounds good!

nilsvu requested a review from nilsdeppe December 17, 2020 15:20

nilsdeppe previously approved these changes Dec 17, 2020

View reviewed changes

nilsvu dismissed nilsdeppe’s stale review via 706bacc December 17, 2020 17:52

nilsdeppe reviewed Dec 17, 2020

View reviewed changes

nilsvu force-pushed the cmake_find_python branch from 706bacc to 61b8c24 Compare December 17, 2020 17:59

nilsdeppe previously approved these changes Dec 17, 2020

View reviewed changes

nilsvu dismissed nilsdeppe’s stale review via 930be99 December 18, 2020 09:34

nilsvu force-pushed the cmake_find_python branch from 61b8c24 to 930be99 Compare December 18, 2020 09:34

nilsvu force-pushed the cmake_find_python branch from 930be99 to 59b61d6 Compare December 18, 2020 20:34

nilsdeppe reviewed Dec 18, 2020

View reviewed changes

nilsdeppe requested changes Dec 18, 2020

View reviewed changes

nilsvu added 5 commits December 22, 2020 18:07

Require CMake 3.12+

6e09a25

Make finding Python in build system more robust

25579e7

- Switch to CMake's FindPython - Install Pybind11 through pip. This should work on all systems that support Python. It means we can use Pybind11's CMake tools, don't have to bundle them and don't need special finding-code.

Add Pybind11 version to installation instructions

0e6f6cb

Fix linking of Pybinding libs

c2c90ac

Add missing link dependencies and link explicitly with pybind11 when including its headers.

Install pybind11 in container via pip

5647299

nilsvu added 5 commits December 22, 2020 18:07

Install unittest2 in container for Py2 compatibility

be26532

Install CMake 3.12 in container so we can test it on CI

66fd806

Upgrade pip in container

7bf731b

This ensures that pre-built binaries are used if possible.

Install Charm with shared libs in container

d28ad31

Remove clang-6 from container

6b3d1dd

nilsvu force-pushed the cmake_find_python branch from 7094afd to 6b3d1dd Compare December 22, 2020 17:07

nilsvu changed the title ~~Make finding Python in build system more robust~~ Make finding Python in build system more robust, update Docker container Dec 22, 2020

nilsdeppe reviewed Dec 22, 2020

View reviewed changes

nilsdeppe approved these changes Dec 22, 2020

View reviewed changes

kidder approved these changes Dec 22, 2020

View reviewed changes

wthrowe approved these changes Dec 22, 2020

View reviewed changes

nilsvu merged commit dfe1c6f into sxs-collaboration:develop Dec 22, 2020

nilsvu deleted the cmake_find_python branch December 22, 2020 23:03

nilsdeppe mentioned this pull request Dec 23, 2020

Can't find pybind11 #2705

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make finding Python in build system more robust, update Docker container #2672

Make finding Python in build system more robust, update Docker container #2672

nilsvu commented Dec 17, 2020 •

edited

Loading

nilsdeppe Dec 17, 2020

nilsvu Dec 17, 2020

nilsdeppe left a comment

nilsdeppe commented Dec 17, 2020

kidder commented Dec 17, 2020

wthrowe commented Dec 18, 2020

nilsvu commented Dec 18, 2020

wthrowe commented Dec 18, 2020

nilsvu commented Dec 18, 2020

nilsdeppe commented Dec 18, 2020

wthrowe commented Dec 18, 2020

nilsvu commented Dec 18, 2020

nilsdeppe commented Dec 18, 2020

nilsvu commented Dec 18, 2020

nilsdeppe left a comment

nilsdeppe left a comment

nilsdeppe Dec 18, 2020

nilsvu commented Dec 20, 2020

wthrowe commented Dec 21, 2020

kidder commented Dec 22, 2020

nilsvu commented Dec 22, 2020

nilsdeppe Dec 22, 2020

nilsvu Dec 22, 2020

nilsdeppe Dec 22, 2020

Make finding Python in build system more robust, update Docker container #2672

Make finding Python in build system more robust, update Docker container #2672

Conversation

nilsvu commented Dec 17, 2020 • edited Loading

Proposed changes

Upgrade instructions

Code review checklist

Further comments

nilsdeppe Dec 17, 2020

Choose a reason for hiding this comment

nilsvu Dec 17, 2020

Choose a reason for hiding this comment

nilsdeppe left a comment

Choose a reason for hiding this comment

nilsdeppe commented Dec 17, 2020

kidder commented Dec 17, 2020

wthrowe commented Dec 18, 2020

nilsvu commented Dec 18, 2020

wthrowe commented Dec 18, 2020

nilsvu commented Dec 18, 2020

nilsdeppe commented Dec 18, 2020

wthrowe commented Dec 18, 2020

nilsvu commented Dec 18, 2020

nilsdeppe commented Dec 18, 2020

nilsvu commented Dec 18, 2020

nilsdeppe left a comment

Choose a reason for hiding this comment

nilsdeppe left a comment

Choose a reason for hiding this comment

nilsdeppe Dec 18, 2020

Choose a reason for hiding this comment

nilsvu commented Dec 20, 2020

wthrowe commented Dec 21, 2020

kidder commented Dec 22, 2020

nilsvu commented Dec 22, 2020

nilsdeppe Dec 22, 2020

Choose a reason for hiding this comment

nilsvu Dec 22, 2020

Choose a reason for hiding this comment

nilsdeppe Dec 22, 2020

Choose a reason for hiding this comment

nilsvu commented Dec 17, 2020 •

edited

Loading