[Python] Add and document setup.py script for Python bindings. (#1913)

This allows users to `pip install lib/Bindings/Python`. Similarly,
this supports `pip wheel lib/Bindings/Python`. The script generally
follows llvm/torch-mlir#256, with some tweaks
that are specific to CIRCT's CMake choices (e.g. using an external
projects unified build).

.github/workflows/buildAndTest.yml

@@ -254,7 +254,7 @@ jobs:
         if: ${{ always() }}
         shell: bash
         run: |
-          files=$(git diff --name-only $DIFF_COMMIT | grep .py || echo -n)
+          files=$(git diff --name-only $DIFF_COMMIT | grep -e '\.py' || echo -n)
           if [[ ! -z $files ]]; then
             yapf --diff $files
           fi

.gitignore

@@ -1,4 +1,5 @@
 /build*
+/install*
 /docker_build
 tags
 .vscode
@@ -10,5 +11,10 @@ compile_commands.json
 __pycache__
 lit.site.cfg.py
 
+# Pip artifacts
+/lib/Bindings/Python/build
+*.egg-info
+*.whl
+
 # External software
 /ext

docs/PythonBindings.md

@@ -1,6 +1,50 @@
 # Using the Python Bindings
 
-If you are mainly interested in using CIRCT from Python scripts, you need to compile both LLVM/MLIR and CIRCT with Python bindings enabled. Furthermore, you must use a unified build, where LLVM/MLIR and CIRCT are compiled together in one step. To do this, you can use a single CMake invocation like this:
+If you are mainly interested in using CIRCT from Python scripts, you need to compile both LLVM/MLIR and CIRCT with Python bindings enabled. Furthermore, you must use a unified build, where LLVM/MLIR and CIRCT are compiled together in one step. 
+
+## Installing and Building with Wheels
+
+CIRCT provides a `setup.py` script that take care of configuring and building LLVM/MLIR, CIRCT, and CIRCT's Python bindings. You can install the CIRCT Python bindings with the `pip install` command:
+
+```
+$ cd circt
+$ pip install lib/Bindings/Python --use-feature=in-tree-build
+```
+
+If you just want to build the wheel, use the `pip wheel` command:
+
+```
+$ cd circt
+$ pip wheel lib/Bindings/Python --use-feature=in-tree-build
+```
+
+This will create a `circt_core-<version>-<python version>-<platform>.whl` file in the root of the repo.
+
+There are some environment variables you can set to control the script. These should be prefixed to the above command(s), or `export`ed in your shell.
+
+To specify an existing CMake build directory, you can set `CIRCT_CMAKE_BUILD_DIR`:
+
+```
+export CIRCT_CMAKE_BUILD_DIR=/path/to/your/build/dir
+```
+
+To specify an alternate LLVM directory, you can set `CIRCT_LLVM_DIR`:
+
+```
+export CIRCT_LLVM_DIR=/path/to/your/llvm
+```
+
+Finally, you can set other environment variables to control CMake. By default, the script uses the same settings as [Manual Compilation](#manual-compilation) below. It is recommended to use Ninja and CCache, which can be accomplished with:
+
+```
+export CMAKE_GENERATOR=Ninja CMAKE_C_COMPILER_LAUNCHER=ccache CMAKE_CXX_COMPILER_LAUNCHER=ccache
+```
+
+All other [CMake environment variables](https://cmake.org/cmake/help/latest/manual/cmake-env-variables.7.html) can also be used.
+
+## Manual Compilation
+
+To manually compile LLVM/MLIR, CIRCT, and CIRCT's Python bindings, you can use a single CMake invocation like this:
 
 ```
 $ cd circt
@@ -18,15 +62,15 @@ $ cmake -G Ninja ../llvm/llvm \
 
 Afterwards, use `ninja check-circt-integration` to ensure that the bindings work. (This will now additionally spin up a couple of Python scripts to test that they are accessible.)
 
-## Without Installation
+### Without Installation
 
 If you want to try the bindings fresh from the compiler without installation, you need to ensure Python can find the generated modules:
 
 ```
 export PYTHONPATH="$PWD/llvm/build/tools/circt/python_packages/circt_core"
 ```
 
-## With Installation
+### With Installation
 
 If you are installing CIRCT through `ninja install` anyway, the libraries and Python modules will be installed into the correct location automatically.
 

lib/Bindings/Python/pyproject.toml

@@ -0,0 +1,11 @@
+[build-system]
+requires = [
+    "setuptools>=42",
+    "wheel",
+    "cmake>=3.12",
+    # MLIR build depends.
+    "numpy",
+    "pybind11>=2.7.1",
+    "PyYAML",
+]
+build-backend = "setuptools.build_meta"

lib/Bindings/Python/setup.py

@@ -0,0 +1,141 @@
+#  Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+#  See https://llvm.org/LICENSE.txt for license information.
+#  SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# Build/install the circt-core python package.
+# Note that this includes a relatively large build of LLVM (~2400 C++ files)
+# and can take a considerable amount of time, especially with defaults.
+# To install:
+#   pip install . --use-feature=in-tree-build
+# To build a wheel:
+#   pip wheel . --use-feature=in-tree-build
+#
+# It is recommended to build with Ninja and ccache. To do so, set environment
+# variables by prefixing to above invocations:
+#   CMAKE_GENERATOR=Ninja \
+#   CMAKE_C_COMPILER_LAUNCHER=ccache \
+#   CMAKE_CXX_COMPILER_LAUNCHER=ccache
+#
+# On CIs, it is often advantageous to re-use/control the CMake build directory.
+# This can be set with the CIRCT_CMAKE_BUILD_DIR env var.
+#
+# By default, this will use the llvm-project submodule included with CIRCT.
+# This can be overridden with the CIRCT_LLVM_DIR env var.
+
+import os
+import platform
+import shutil
+import subprocess
+import sysconfig
+
+from distutils.command.build import build as _build
+from setuptools import find_namespace_packages, setup, Extension
+from setuptools.command.build_ext import build_ext
+from setuptools.command.build_py import build_py
+
+
+# Build phase discovery is unreliable. Just tell it what phases to run.
+class CustomBuild(_build):
+
+  def run(self):
+    self.run_command("build_py")
+    self.run_command("build_ext")
+    self.run_command("build_scripts")
+
+
+class CMakeExtension(Extension):
+
+  def __init__(self, name, sourcedir=""):
+    Extension.__init__(self, name, sources=[])
+    self.sourcedir = os.path.abspath(sourcedir)
+
+
+class CMakeBuild(build_py):
+
+  def run(self):
+    target_dir = self.build_lib
+    circt_dir = os.path.abspath(
+        os.path.join(os.path.dirname(__file__), "..", "..", ".."))
+    cmake_build_dir = os.getenv("CIRCT_CMAKE_BUILD_DIR")
+    if not cmake_build_dir:
+      cmake_build_dir = os.path.join(circt_dir, "build")
+    cmake_install_dir = os.path.join(cmake_build_dir, "..", "install")
+    llvm_dir = os.getenv("CIRCT_LLVM_DIR")
+    if not llvm_dir:
+      llvm_dir = os.path.join(circt_dir, "llvm")
+    cmake_args = [
+        "-DCMAKE_BUILD_TYPE=Release",  # not used on MSVC, but no harm
+        "-DCMAKE_INSTALL_PREFIX={}".format(os.path.abspath(cmake_install_dir)),
+        "-DLLVM_ENABLE_PROJECTS=mlir",
+        "-DLLVM_EXTERNAL_PROJECTS=circt",
+        "-DLLVM_EXTERNAL_CIRCT_SOURCE_DIR={}".format(circt_dir),
+        "-DLLVM_TARGETS_TO_BUILD=host",
+        "-DMLIR_ENABLE_BINDINGS_PYTHON=ON",
+        "-DCIRCT_BINDINGS_PYTHON_ENABLED=ON",
+    ]
+
+    # HACK: CMake fails to auto-detect static linked Python installations, which
+    # happens to be what exists on manylinux. We detect this and give it a dummy
+    # library file to reference (which is checks exists but never gets
+    # used).
+    if platform.system() == "Linux":
+      python_libdir = sysconfig.get_config_var('LIBDIR')
+      python_library = sysconfig.get_config_var('LIBRARY')
+      if python_libdir and not os.path.isabs(python_library):
+        python_library = os.path.join(python_libdir, python_library)
+      if python_library and not os.path.exists(python_library):
+        print("Detected static linked python. Faking a library for cmake.")
+        fake_libdir = os.path.join(cmake_build_dir, "fake_python", "lib")
+        os.makedirs(fake_libdir, exist_ok=True)
+        fake_library = os.path.join(fake_libdir,
+                                    sysconfig.get_config_var('LIBRARY'))
+        subprocess.check_call(["ar", "q", fake_library])
+        cmake_args.append("-DPython3_LIBRARY:PATH={}".format(fake_library))
+
+    build_args = []
+    os.makedirs(cmake_build_dir, exist_ok=True)
+    if os.path.exists(cmake_install_dir):
+      shutil.rmtree(cmake_install_dir)
+    cmake_cache_file = os.path.join(cmake_build_dir, "CMakeCache.txt")
+    if os.path.exists(cmake_cache_file):
+      os.remove(cmake_cache_file)
+    subprocess.check_call(["cmake", llvm_dir] + cmake_args, cwd=cmake_build_dir)
+    subprocess.check_call(["cmake", "--build", ".", "--target", "install"] +
+                          build_args,
+                          cwd=cmake_build_dir)
+    shutil.copytree(os.path.join(cmake_install_dir, "python_packages",
+                                 "circt_core"),
+                    target_dir,
+                    symlinks=False,
+                    dirs_exist_ok=True)
+
+
+class NoopBuildExtension(build_ext):
+
+  def build_extension(self, ext):
+    pass
+
+
+setup(
+    name="circt-core",
+    version="0.0.1",
+    author="Mike Urbach",
+    author_email="mike@alloystack.io",
+    description="CIRCT Core",
+    long_description="",
+    include_package_data=True,
+    ext_modules=[
+        CMakeExtension("mlir._mlir_libs._mlir"),
+        CMakeExtension("mlir._mlir_libs._circt"),
+    ],
+    cmdclass={
+        "build": CustomBuild,
+        "built_ext": NoopBuildExtension,
+        "build_py": CMakeBuild,
+    },
+    zip_safe=False,
+    packages=find_namespace_packages(include=[
+        "circt",
+        "circt.*",
+    ]),
+)