Conan and Python related fixes

- README.md Be more explicit about clang versions included in ci_common containers - packages/conan/recipes/pyside/... Use standard patching mechanism instead of hard coding (requires vendoring a couple of patches). Also pyside is build in vfx20XX context, not ci_commonX - packages/conan/recipes/clang/... Resolve circular dependency between clang built in ci_commonX context and python build in vfx20XX context: build our own Python to satisfy clang test suite whether the base container has a usable Python 3 or not. - python/aswfdocker/... Cleanup pylint warnings / errors that were failing SonarCloud - scripts/common/install_conanpackages.sh Clean up spurious debugging print Signed-off-by: Jean-Francois Panisset <panisset@gmail.com>
AcademySoftwareFoundation · Jun 21, 2023 · 43b5442 · 43b5442
1 parent fa63dfc
commit 43b5442
Show file tree

Hide file tree

Showing 10 changed files with 857 additions and 41 deletions.
diff --git a/README.md b/README.md
@@ -17,9 +17,9 @@ Each image (apart from `ci-common`) is available for multiple VFX Platform Years
 
 | Image  | Stats | Description |
 | ------ | ----- | ----------- |
-| [aswf/ci-common:1](https://hub.docker.com/r/aswf/ci-common/tags?name=1) ![Image Version](https://img.shields.io/docker/v/aswf/ci-common/latest) | ![Image Size](https://img.shields.io/docker/image-size/aswf/ci-common/latest) ![Pulls](https://img.shields.io/docker/pulls/aswf/ci-common) | A base CentOS-7 image with devtoolset-6 (GCC-6.3.1), clang-7 and cuda-10.2. |
-| [aswf/ci-common:2](https://hub.docker.com/r/aswf/ci-common/tags?name=2) ![Image Version](https://img.shields.io/docker/v/aswf/ci-common/preview) | ![Image Size](https://img.shields.io/docker/image-size/aswf/ci-common/preview) ![Pulls](https://img.shields.io/docker/pulls/aswf/ci-common) | A base CentOS-7 image with devtoolset-9.1 (GCC-9.3.1), clang-10 and cuda-11. |
-| [aswf/ci-common:3](https://hub.docker.com/r/aswf/ci-common/tags?name=3) ![Image Version](https://img.shields.io/docker/v/aswf/ci-common/preview) | ![Image Size](https://img.shields.io/docker/image-size/aswf/ci-common/preview) ![Pulls](https://img.shields.io/docker/pulls/aswf/ci-common) | A base Rocky-8 image with gcc-toolset-11 (GCC-11.2.x), clang-14 and cuda-11.8. |
+| [aswf/ci-common:1](https://hub.docker.com/r/aswf/ci-common/tags?name=1) ![Image Version](https://img.shields.io/docker/v/aswf/ci-common/latest) | ![Image Size](https://img.shields.io/docker/image-size/aswf/ci-common/latest) ![Pulls](https://img.shields.io/docker/pulls/aswf/ci-common) | A base CentOS-7 image with devtoolset-6 (GCC-6.3.1), clang-6-10 and cuda-10.2. |
+| [aswf/ci-common:2](https://hub.docker.com/r/aswf/ci-common/tags?name=2) ![Image Version](https://img.shields.io/docker/v/aswf/ci-common/preview) | ![Image Size](https://img.shields.io/docker/image-size/aswf/ci-common/preview) ![Pulls](https://img.shields.io/docker/pulls/aswf/ci-common) | A base CentOS-7 image with devtoolset-9.1 (GCC-9.3.1), clang-10-14 and cuda-11.4. |
+| [aswf/ci-common:3](https://hub.docker.com/r/aswf/ci-common/tags?name=3) ![Image Version](https://img.shields.io/docker/v/aswf/ci-common/preview) | ![Image Size](https://img.shields.io/docker/image-size/aswf/ci-common/preview) ![Pulls](https://img.shields.io/docker/pulls/aswf/ci-common) | A base RockyLinux-8 image with gcc-toolset-11 (GCC-11.2.x), clang-14-15 and cuda-11.8. |
 | [aswf/ci-base:2018](https://hub.docker.com/r/aswf/ci-base/tags?name=2018) ![Image Version](https://img.shields.io/docker/v/aswf/ci-base/2018) | ![Image Size](https://img.shields.io/docker/image-size/aswf/ci-base/2018) ![Pulls](https://img.shields.io/docker/pulls/aswf/ci-base) | Based on `aswf/ci-common:1` with most VFX Platform requirements pre-installed. |
 | [aswf/ci-base:2019](https://hub.docker.com/r/aswf/ci-base/tags?name=2019) ![Image Version](https://img.shields.io/docker/v/aswf/ci-base/2019) | ![Image Size](https://img.shields.io/docker/image-size/aswf/ci-base/2019) ![Pulls](https://img.shields.io/docker/pulls/aswf/ci-base) | Based on `aswf/ci-common:1` with most VFX Platform requirements pre-installed. |
 | [aswf/ci-base:2020](https://hub.docker.com/r/aswf/ci-base/tags?name=2020) ![Image Version](https://img.shields.io/docker/v/aswf/ci-base/2020) | ![Image Size](https://img.shields.io/docker/image-size/aswf/ci-base/2020) ![Pulls](https://img.shields.io/docker/pulls/aswf/ci-base) | Based on `aswf/ci-common:1` with most VFX Platform requirements pre-installed. |

diff --git a/packages/conan/recipes/clang/conanfile.py b/packages/conan/recipes/clang/conanfile.py
@@ -33,9 +33,23 @@ class ClangConan(ConanFile):
     def _source_subfolder(self):
         return "source"
 
-    def build_requirements(self):
-        if tools.Version(self.version) > "11":
-            self.build_requires(f"python/3.9.7@{self.user}/vfx2022")
+    # def build_requirements(self):
+    # if tools.Version(self.version) > "11":
+    # We can't use our python package for a few reasons:
+    # - it introduces a circular dependency between the clang package build, in ci_commonX context
+    #   and vfx20XX context
+    # - the Package ID won't match: the vfx20XX profile adds a python=3.x setting which the ci_commonX
+    #   doesn't have, and if you don't define ASWF_NUMPY_VERSION, the conanfile.py for the python package
+    #   sets the option with_numpy=False, which also invalidates the Package ID
+    #
+    # Also the process which installed Conan in our build container built a Python in /tmp/pyconan, but
+    # it cleans up up once Conan has been installed and turned into an executable with pyinstaller.
+    #
+    # We could of course make sure we have a Python 3 installed in the docker image.
+    #
+    # Instead we just go ahead and rebuild Python from scratch.
+    #
+    # self.build_requires(f"python/3.9.7@{self.user}/vfx2022")
 
     def configure(self):
         compiler = self.settings.compiler.value
@@ -107,10 +121,25 @@ def source(self):
         )
 
     def build(self):
-        with tools.environment_append(tools.RunEnvironment(self).vars):
-            cmake = self._configure_cmake()
-            cmake.configure(source_folder="source/llvm")
-            cmake.build()
+        # See commented out build_requirements(): we have to build our own Python to break circular dependency
+        pyclangenv = {
+            "PATH": f"/tmp/pyclang/bin:{os.environ['PATH']}",
+            "LD_LIBRARY_PATH": f"/tmp/pyclang/lib:{os.environ['LD_LIBRARY_PATH']}",
+        }
+        tools.get(
+            f"https://www.python.org/ftp/python/{os.environ['ASWF_CONAN_PYTHON_VERSION']}/Python-{os.environ['ASWF_CONAN_PYTHON_VERSION']}.tgz"
+        )
+        with tools.environment_append(pyclangenv):
+            with tools.chdir(f"Python-{os.environ['ASWF_CONAN_PYTHON_VERSION']}"):
+                self.run(
+                    "./configure --prefix=/tmp/pyclang --enable-unicode=ucs4 --enable-shared"
+                )
+                self.run(f"make -j{tools.cpu_count()}")
+                self.run("make install")
+            with tools.environment_append(tools.RunEnvironment(self).vars):
+                cmake = self._configure_cmake()
+                cmake.configure(source_folder="source/llvm")
+                cmake.build()
 
     def package(self):
         self.copy(

diff --git a/packages/conan/recipes/pyside/conandata.yml b/packages/conan/recipes/pyside/conandata.yml
@@ -9,6 +9,13 @@ sources:
     url: "https://download.qt.io/official_releases/QtForPython/pyside2/PySide2-5.15.9-src/pyside-setup-opensource-src-5.15.9-1.tar.xz"
     sha256: "3298660742037c71274a30770541abf6d1552ec65fc93678eedee5718852a62c"
 patches:
+  "5.12.6":
+    # typing-patch.zip from https://codereview.qt-project.org/changes/pyside%2Fpyside-setup~271412/revisions/4/patch?zip
+    - patch_file: "patches/28958df.diff"
+      base_path: "source_subfolder"
+    # clang10-patch.zip from https://codereview.qt-project.org/changes/pyside%2Fpyside-setup~296271/revisions/2/patch?zip
+    - patch_file: "patches/9ae6382.diff"
+      base_path: "source_subfolder"
   "5.15.9":
-    - patch_file: "shiboken_numpy_1_23.diff"
-      base_path: "patches_subfolder"
+    - patch_file: "patches/shiboken_numpy_1_23.diff"
+      base_path: "source_subfolder"
diff --git a/packages/conan/recipes/pyside/conanfile.py b/packages/conan/recipes/pyside/conanfile.py
@@ -11,6 +11,7 @@ class PySideConan(ConanFile):
     homepage = "https://www.qt.io/qt-for-python"
     license = "LGPL-3.0"
     url = "https://github.com/AcademySoftwareFoundation/aswf-docker"
+    exports = ["patches/*.diff"]
     settings = (
         "os",
         "arch",
@@ -28,33 +29,25 @@ def requirements(self):
 
     def build_requirements(self):
         self.build_requires(
-            f"clang/(latest)@{self.user}/ci_common{self.settings.ci_common}"
+            f"clang/(latest)@{self.user}/ci_common{os.environ['CI_COMMON_VERSION']}"
         )
 
     def source(self):
         tools.get(**self.conan_data["sources"][self.version])
-        os.rename(
+        extractdir = (
             self.conan_data["sources"][self.version]["url"]
             .split("/")[-1]
-            .replace(".tar.xz", ""),
+            .replace(".tar.xz", "")
+        )
+        if self.version == "5.15.9":
+            # tar file for 5.15.9 adds -1 suffix, but extracts to 5.15.9
+            extractdir = extractdir.rstrip("-1")
+        os.rename(
+            extractdir,
             self._source_subfolder,
         )
-        if self.version == "5.12.6":
-            # Apply typing patch
-            tools.get(
-                "https://codereview.qt-project.org/changes/pyside%2Fpyside-setup~271412/revisions/4/patch?zip",
-                filename="typing-patch.zip",
-            )
-            self.run("patch -p1 < ../28958df.diff", cwd=self._source_subfolder)
-            # Apply clang10 patch
-            tools.get(
-                "https://codereview.qt-project.org/changes/pyside%2Fpyside-setup~296271/revisions/2/patch?zip",
-                filename="clang10-patch.zip",
-            )
-            self.run("patch -p1 < ../9ae6382.diff", cwd=self._source_subfolder)
-        else:
-            for patch in self.conan_data.get("patches", {}).get(self.version, []):
-                tools.patch(**patch)
+        for patch in self.conan_data.get("patches", {}).get(self.version, []):
+            tools.patch(**patch)
 
     def build(self):
         vars = tools.RunEnvironment(self).vars