[ci] Use clang+Ninja to build Taichi on windows #3735

bobcao3 · 2021-12-07T08:58:29Z

Related issue = #

netlify · 2021-12-07T08:58:38Z

✔️ Deploy Preview for jovial-fermat-aa59dc ready!

🔨 Explore the source changes: 525186b

🔍 Inspect the deploy log: https://app.netlify.com/sites/jovial-fermat-aa59dc/deploys/61bc00aae9cfd70007f0cb8e

😎 Browse the preview: https://deploy-preview-3735--jovial-fermat-aa59dc.netlify.app

bobcao3 · 2021-12-07T23:39:33Z

/format

bobcao3 · 2021-12-09T05:26:03Z

Cache is all miss, not sure why. The build works

frostming · 2021-12-09T06:18:31Z

Cache is all miss, not sure why. The build works

Maybe need to add -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache

bobcao3 · 2021-12-09T07:47:07Z

Cache is all miss, not sure why. The build works

Maybe need to add -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache

No need. ccache is called, it's just that ccache thinks it's all miss. (in the cmakelist it detects the ccache program and automatically set the launchers)

ailzhang · 2021-12-09T08:30:46Z

setup.py

@@ -187,7 +187,7 @@ def prepare_package(self):
                    shutil.copy(moltenvk_path,
                                os.path.join(target, 'libMoltenVK.dylib'))
            else:
-                shutil.copy('runtimes/Release/taichi_core.dll',
+                shutil.copy('runtimes/taichi_core.dll',


OOC do we need Release here (seems like we explicitly opened a Release folder above for ninja as well) ?

The release folder is generated by msbuild. I can't think of a good way to detect whether msbuild or ninja is used. Maybe just enforce ninja?

IIUC this line already implicitly enforced ninja? ;) If so we'll need to be extra careful and make sure the dev install is updated correctly. ;)

Yes... kind of sad, but I can't think of a way to cleanly deal with this.

@bobcao3 I don't mind switching to ninja for now ;) but this PR better stays consistent ;) . For example, if (CMAKE_GENERATOR EQUAL "Ninja") above kinda hints that we still support msbuild :D
If we are sure ninja is easy to install on windows and we know how to update dev install doc, we could just error out on msbuild in cmake?

lin-hitonami · 2021-12-09T09:38:05Z

Do all the compilations happen in C:/Users/runneradmin/AppData/Local/Temp/pip-req-build-n_d7za0z/ ? This seems to be a randomly generated directory and it needs some configuration to enable sharing caches when compiling in different directories. See https://ccache.dev/manual/4.5.1.html#_compiling_in_different_directories . (btw sccache does not support that at all)

frostming · 2021-12-09T09:54:33Z

Do all the compilations happen in C:/Users/runneradmin/AppData/Local/Temp/pip-req-build-n_d7za0z/ ? This seems to be a randomly generated directory and it needs some configuration to enable sharing caches when compiling in different directories. See https://ccache.dev/manual/4.5.1.html#_compiling_in_different_directories . (btw sccache does not support that at all)

Yes, because it is built by pip, which sets up a temp dir for build. Change it to python setup.py install to check if cache hits, but I have to mention invoking setup.py directly is not a good practice.

Update win_build.ps1

ailzhang · 2021-12-13T06:00:06Z

@frostming

but I have to mention invoking setup.py directly is not a good practice.

Is it better to use pip install . instead ? If so feel free to send PRs to update our CI & doc sites :D!

ailzhang

Nice work, thanks!!

ailzhang · 2021-12-13T06:03:37Z

.github/workflows/scripts/win_build.ps1

 if ($install) {
    if ($develop) {
-        python -m pip install -v -e .
+        python setup.py develop


These commands are kinda equivalent (and IIUC pip install version is preferred :P so let's keep this unchanged?

In the pip version caching won't work because it uses a different path every time and the generators use absolute path everywhere

ailzhang · 2021-12-13T06:07:45Z

setup.py

@@ -187,7 +187,7 @@ def prepare_package(self):
                    shutil.copy(moltenvk_path,
                                os.path.join(target, 'libMoltenVK.dylib'))
            else:
-                shutil.copy('runtimes/Release/taichi_core.dll',
+                shutil.copy('runtimes/taichi_core.dll',


@bobcao3 I don't mind switching to ninja for now ;) but this PR better stays consistent ;) . For example, if (CMAKE_GENERATOR EQUAL "Ninja") above kinda hints that we still support msbuild :D
If we are sure ninja is easy to install on windows and we know how to update dev install doc, we could just error out on msbuild in cmake?

bobcao3 · 2021-12-13T06:39:21Z

I tried CMAKE_GENERATOR but it doesn't really work, it did not detect Ninja at all so... not sure

Update win_build.ps1

frostming · 2021-12-15T03:12:59Z

The changes made here LGTM and I hope this will be landed in the master soon as it dramatically accelerates the windows build 💯

We can figure out how to use the preferred build approach pip install later in another PR.

ailzhang · 2021-12-15T05:46:45Z

@bobcao3 Would you mind resolving the merge conflict so that we can get it in? Thanks a lot!

…into bobcao3-patch-2

…bcao3-patch-2

k-ye · 2021-12-16T11:44:18Z

docs/lang/articles/contribution/dev_install.md

-   ```shell
-   .\build_taichi.ps1
-   ```
+```shell


Let's add a three-space indentation to this code block

ailzhang · 2021-12-16T11:46:15Z

docs/lang/articles/contribution/dev_install.md

+
+4. Build taichi by using `python setup.py develop`
+
+:::note


Did we just add this part (or move from other places)? I somehow couldn't find where it was moved from hmmm

Previously we asked the user to run build_taichi.ps1. Now we are asking them to build it through command line, the build script is for CI, not really for end user

bobcao3 · 2021-12-17T03:14:06Z

/format

ailzhang

LGTM！

bobcao3 force-pushed the bobcao3-patch-2 branch 2 times, most recently from 579728b to e474e20 Compare December 9, 2021 00:48

bobcao3 marked this pull request as ready for review December 9, 2021 05:25

bobcao3 changed the title ~~[ci] Use clang to build windows (Test)~~ [ci] Use clang to build windows Dec 9, 2021

bobcao3 requested review from frostming, Leonz5288, k-ye and ailzhang and removed request for Leonz5288 December 9, 2021 05:26

bobcao3 changed the title ~~[ci] Use clang to build windows~~ [ci] Use clang+Ninja to build Taichi on windows Dec 9, 2021

ailzhang reviewed Dec 9, 2021

View reviewed changes

Try to use clang in windows CI

20dceb7

Update win_build.ps1

bobcao3 force-pushed the bobcao3-patch-2 branch from b368f5a to 20dceb7 Compare December 10, 2021 06:26

bobcao3 added 2 commits December 10, 2021 02:40

Generate PDB files on windows & enable LTO when using Clang

e7bfdc4

Fix lto flags (only enable when on windows or mac using Clang and LLD)

9a0b9cd

bobcao3 mentioned this pull request Dec 13, 2021

taichi failed to build on Windows with MSVC on latest version 5953e03 #3743

Closed

ailzhang reviewed Dec 13, 2021

View reviewed changes

bobcao3 added 2 commits December 13, 2021 04:51

Try to use clang in windows CI

0739ad7

Update win_build.ps1

Generate PDB files on windows & enable LTO when using Clang

881b81e

bobcao3 added 5 commits December 15, 2021 23:29

Merge branch 'bobcao3-patch-2' of https://github.com/taichi-dev/taichi …

b01170c

…into bobcao3-patch-2

Merge branch 'master' of https://github.com/taichi-dev/taichi into bo…

bad5553

…bcao3-patch-2

update dev install docs

b8538a2

update dev install docs

681b1ea

update dev install docs

7844d0c

bobcao3 requested a review from ailzhang December 16, 2021 08:59

k-ye reviewed Dec 16, 2021

View reviewed changes

ailzhang reviewed Dec 16, 2021

View reviewed changes

Update dev_install.md

8140948

bobcao3 requested review from k-ye and ailzhang December 16, 2021 20:08

Merge branch 'master' into bobcao3-patch-2

28aeec3

Auto Format

525186b

ailzhang approved these changes Dec 17, 2021

View reviewed changes

bobcao3 merged commit 2618db2 into master Dec 17, 2021

bobcao3 mentioned this pull request Dec 17, 2021

Explore using clang-cl instead of MSVC on windows #3720

Closed

strongoier mentioned this pull request Dec 20, 2021

[ci] Enforce using ninja on Windows in both release and normal testing #3837

Merged

bobcao3 deleted the bobcao3-patch-2 branch March 27, 2022 05:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ci] Use clang+Ninja to build Taichi on windows #3735

[ci] Use clang+Ninja to build Taichi on windows #3735

bobcao3 commented Dec 7, 2021

netlify bot commented Dec 7, 2021 •

edited

Loading

bobcao3 commented Dec 7, 2021

bobcao3 commented Dec 9, 2021

frostming commented Dec 9, 2021

bobcao3 commented Dec 9, 2021

ailzhang Dec 9, 2021

bobcao3 Dec 10, 2021

ailzhang Dec 13, 2021

bobcao3 Dec 13, 2021

ailzhang Dec 13, 2021

lin-hitonami commented Dec 9, 2021

frostming commented Dec 9, 2021 •

edited

Loading

ailzhang commented Dec 13, 2021

ailzhang left a comment

ailzhang Dec 13, 2021

bobcao3 Dec 13, 2021

ailzhang Dec 13, 2021

bobcao3 commented Dec 13, 2021

frostming commented Dec 15, 2021

ailzhang commented Dec 15, 2021

k-ye Dec 16, 2021

ailzhang Dec 16, 2021

bobcao3 Dec 16, 2021

bobcao3 commented Dec 17, 2021

ailzhang left a comment

[ci] Use clang+Ninja to build Taichi on windows #3735

[ci] Use clang+Ninja to build Taichi on windows #3735

Conversation

bobcao3 commented Dec 7, 2021

netlify bot commented Dec 7, 2021 • edited Loading

bobcao3 commented Dec 7, 2021

bobcao3 commented Dec 9, 2021

frostming commented Dec 9, 2021

bobcao3 commented Dec 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lin-hitonami commented Dec 9, 2021

frostming commented Dec 9, 2021 • edited Loading

ailzhang commented Dec 13, 2021

ailzhang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobcao3 commented Dec 13, 2021

frostming commented Dec 15, 2021

ailzhang commented Dec 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobcao3 commented Dec 17, 2021

ailzhang left a comment

Choose a reason for hiding this comment

netlify bot commented Dec 7, 2021 •

edited

Loading

frostming commented Dec 9, 2021 •

edited

Loading