Publish on asset library & bundling distributable files

[RechieKho]

So iree.gd will soon to be on asset library. Unfortunately I have no knowledge on:

• Publishing on asset library
• cross-compiling with CMake
• bundle the libraries together
• Do everything with github action

I do tried writing a simple python script that generate .gdextention file, hopefully it is helpful.

[fire]

I'm having trouble keeping up but here some examples of github action.

See https://github.com/tefusion/godot-subdiv and https://github.com/ashtonmeuser/godot-wasm

[fire]

I wrote up a proposal here #17 and I'll look into this.

[RechieKho]

Since we are going to cross-platform via Github actions, I should list down the important key points so that I don't forget:

• Instead of building only in Release mode, we should also build for Debug mode (or RelWithDebInfo) to generate both release library and debug library, both of them will be used by the consumer in which the debug library will be run in the editor while release library will be run in the exported projects.
• The library naming convention follow this format libiree-gd.${SYSTEM_NAME}.${BUILD_TYPE}.${ARCH}, check out sample/extensions/iree-gd/iree-gd.gdextension for all the possible library names.
• We can distribute it by either:
  • packing only the cross-compiled libraries, the license, and the sample/extensions/iree-gd/iree-gd.gdextension into one ZIP file,
  • packing the sample as well.

[RechieKho]

I've attempt on tackle this problem. My work is on build-workflow branch. I've define all the possible targets and their host platform. Now I just need to setup CMake Toolchain file for each platform, and hopefully it will be done.

[RechieKho]

Progress report:

• Reaching 50 commits mark, I have no idea this will take this much effort. (Exhaustion)
• Currently trying to extract all target OS into individual workflow as everything in one workflow is hard to test things out per platform.
• We found out that any cross-compilation requires iree host binary to be present first (either compile or download, though IREE only release for linux).
• x86_32 arch is not supported.
  • The IREE devs says it is because nobody asks, it won't be hopeless to port to x86_32.
  • But if nobody asked, should we target them though? It will only affect windows and linux with that arch.
• I tried compile via Microsoft VS for Windows as it is much easier to search for how to target arch with it, but it seems like IREE's build file doesn't consider for Microsoft VS. So we are back to ninja build system with vcvarsall.bat to set up all the env variable.
• For Linux, I found a very cool toolchain files from Google's chromium repo that targets arm32 and arm64. Currently trying it out, but I'm still setting up iree host binary as it is a cross-compilation.
• For macOS, I'm trying to compile the iree host tools with preinstalled LLVM (so it'll be much faster). But Github action give up and my macOS workflow forever be queueing.
• For mobile platforms, I'm yet to extract them out.

[Ughuuu]

It seems very complicated tbh. To me it feels like the main problem is setting up the build for everything.
So for example, if you want linux build, you have to setup linux compiler. But if you want android build, you have to setup compiler that is different, and also some other frameworks, libraries. Building everything and making sure then that it matches what godot knows how to load.

What I have ended up doing for other GDExtensions is just porting the CMakeLists.txt file to SConstruct. Basically saying in SConstruct all dependencies, all files, all headers, all defines, etc. The benefits of this is that you can use the normal godot-cpp build system, that already works for all environments it exports to.

Are you also interested in doing so?

[Ughuuu]

Take a look at https://github.com/V-Sekai/godot-whisper for example. The Whisper.CPP project does have a CMakelists where they define everything(haven't ended up porting everything yet). Just include all the files from there and kinda do what the CMake build system does in SConstruct. Then, run the build as you would normally.

Also, if you are interested, I am working on building a set of reusable actions for godot-cpp so that people can reuse and build for multiple platforms and with different options with little problem.

[Ughuuu]

There is also the popular Godot Jolt, which uses CMake as you do. But it doesn't have web or mobile builds I think, or it's hard to do as it has to write all that by hand from scratch and make sure compilers match, etc.

[RechieKho]

Though I've already speak about this in the super secret discord group chat, but I'll still repeat this for future references.

The hustle is mostly done. I don't know scons can fix these issue...

iOS

The headache part is that it needs some host binary if it goes cross platform, and if the binary depends on the llvm, then we got to include them as well through the submodule as llvm from most of the package manager is not nightly. The problem is the compile time if we compile LLVM....

x8632 and arm

It is IREE's code and not the build system, they did some oopsies here and there and no one test out by targeting x8632 and arm. ⁠OpenXLA IREE⁠

the code change too fast

As the title said, IREE is much active. Code changes a lot.

[RechieKho]

Temporary disabling MacOS and LinuxARM build workflow

Now, universal MacOS build won't work because of cpuinfo doesn't support that.

cpuinfo strikes again! we've been wanting to remove it for awhile but it's been low priority - we already stopped using it on windows (which incidentally had the same ARM issue: iree-org/iree#14130), and mac is next on the list to remove (as it's pretty much useless there)
iree-org/iree#8469 is the issue tracking its removal
By Ben Vanik

Linux ARM build also doesn't work.

/home/runner/work/iree.gd/iree.gd/thirdparty/iree/runtime/src/iree/task/executor.c: In function ‘iree_task_executor_create’:
/home/runner/work/iree.gd/iree.gd/thirdparty/iree/runtime/src/iree/task/executor.c:127:44: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
  127 |   iree_prng_splitmix64_initialize(/*seed=*/(uint64_t)(out_executor),
      |                                            ^
  

Someone would need to hack into IREE's code and fix this error by casting it to uintptr_t prior.

(uint64_t)((uintptr)out_executor)

Cool that we are helping IREE by testing it to compile in different environment. Maybe google should hire me.

Thus, the workflow will be disabled until IREE's code is fixed. Do notify me if it is fixed.

---

Update: Possible Bandage Found for Mac OS 🩹

For Mac OS, There is a IREE_ENABLE_CPUINFO CMake definition that can disable cpuinfo, it might fix the issue.

Something worth reading.
DmitriySalnikov/godot_debug_draw_3d#34 (comment)

[RechieKho]

Reaching alpha

After this issue regarding failing build is resolved, I think I'll publish alpha version as my building and publishing workflows are nearly complete. This version will only include:

• windows build
• linux build
• android build

The rest (MacOS, Linux arm and IOS) is, unfortunately, still failing so hard that I guess I'll delay the fix to the next alpha version.

Before publishing the alpha version, I'll still need to:

• Disable bundling failed build into the published version.
• Modify the .gdextension's library path.
• Update README.md for mentioning the progress of the builds.

[RechieKho]

Hello world

First alpha 🎆

[RechieKho]

Linux ARM is fixed, but not for Mac OS and IOS. I'm not planning to include them into releases, unless this technical issue is fixed.

Edit: It is fixed.

[fire]

Published at https://godotengine.org/asset-library/asset/2639