Optionally use the temp directory on windows instead of Ninja

[Lokathor]

Could it be set up so that instead of needing ninja the build script optionally uses the windows temp directory to build the shaderc lib, instead of the OUT_DIR path?

[mooman219]

+1.

It feels pretty bad to require msys in the readme if you're using msvc on windows even with ninja and cmake on your path already.

[antiagainst]

@mooman219 : Yes, we do not need to require msys2 for windows-msvc. I updated README.md with 53408c3.

[antiagainst]

@Lokathor: So shaderc-sys is already quite complicated regarding build options right now. I'm a little bit reluctant to make it even more so. Is there a compelling reason that downloading ninja and putting it in path is not acceptable for you? I think using ninja is generally better as for compilation speed? But I am definitely not aware of your settings. I guess I'd like to know some context to understand it better. :)

Besides, the temp directory prefix is not necessary shorter. It's C:\Users\xxx\Local\Temp already...

[icefoxen]

Needing to install ninja is not hard but is one step further away from being able to do cargo build to build a program. Given that shaderc-rs is part of gfx-hal, which is part of Amethyst and will eventually be part of ggez, making the build process as simple as possible will impact lots of users. ggez at least is targeted at hobbyist game devs, which means if life is difficult/complicated the ggez issue tracker will fill up with people asking for help and I'll have to deal with it.

If there's alternatives that make it simpler to be able to make a window that says "hello world", I'll happily help implement and maintain them.

[Lokathor]

(I say all of the following as a person who is myself the maintainer of a bindings crate, so I know somewhat the troubles you face)

So, the temp directory seems to default to %USERPROFILE%\AppData\Local\Temp, which for me for example is C:\Users\Daniel\AppData\Local\Temp, 34 code units. I'm going to measure this all in code units not code points because that's what the path length limit is actually measured in, but since all my paths here are ASCII those two numbers will be equal for WTF16 file paths.

So say we're developing learn-gfx-hal. It's in a development folder that I keep at the root of my data drive, D:\dev\, and so the base path is D:\dev\learn-gfx-hal (20 code units). Many people don't necessarily do that, so their base path might be much longer. Perhaps C:\Users\Daniel\Documents\learn-gfx-hal (39 units) or maybe something longer if there's a "code" folder that all the projects go in (which would cost you 5 units).

So naturally the files go into OUT_DIR which is something like D:\dev\learn-gfx-hal\target\debug\build\shaderc-0b3a14f59f26d605\out\ (69 units, hold the jokes), but then that's where we just start the building, and in here we have several layers of project folder before we finally get to the folder that holds the output DLL: D:\dev\learn-gfx-hal\target\debug\build\shaderc-0b3a14f59f26d605\out\build\shaderc\libshaderc\Release (101 units). Of course this isn't even the longest path'd thing I can find in that mess of files (and one of which could exceed the max path and blow up the build), a quick glance found one item with a 193 unit path when you include the filename. The .dll / .lib is just the thing we care about the most, since that's the goal.

So since the limit is 240, then going from OUT_DIR of ~70 units to TEMP of ~35 units is a notable chunk of improvement. The big win is that we don't have to require developers to download and install any special tool.

However, here's a better idea, I don't know why it didn't come to me before: Just take all the library files that we need to link against: shaderc.lib, shaderc_combined.lib, shaderc_shared.dll, shaderc_shared.exp, and shaderc_shared.lib, then decide what ones are actually necessary for building the crate on windows (probably just that .lib if I had to guess) and then ship those in the repository files. And just immediately link to them unless there's some cargo flag that people flip on that says "please, i do want to kill my compile times, please build the whole project from absolute scratch".

And everyone else can just link to a pre-built lib file. and the whole crate will build in like 1 second instead of like 500 seconds.

[antiagainst]

I still feel it's amusing that in the year of 2019, we are still fighting some historical limitation from DOS time... But anyway, seems things will be better with MSBuild 16 (or at least a good step forward I guess).

@icefoxen: Yeah, it makes sense for me. Agreed that requiring an additional download step before building is not a pleasant thing for developers/users. People generally does not really look at README.md that much. Given shaderc-sys is likely a dependency of many other projects (which is nice ;-P), it helps to avoid the fraction.

@Lokathor: thanks a lot for the detailed analysis! I see your point that moving to use TEMP is mitigating the problem. However, I'm worried that it's not a once-for-all solution and we can be bitten again because of this. Note that the username part can be arbitrarily long; and we control neither the OUT_DIR path hierarchy of cargo nor the target names in SPIRV-Tools... It is not a very scalable solution of moving directories every time something goes out of limit...

Instead, what about this:

I know that Visual Studio has already shipped CMake and Ninja: https://devblogs.microsoft.com/cppblog/cmake-support-in-visual-studio-whats-new-in-2017-15-3-update/.

So I first check the path for Ninja. If not found, I then check the Visual Studio directory for it. The fallback can be using MSBuild and hope for the best. (SPIRV-Tools recently landed a commit to reduce the length, which hopefully will make MSBuild work again. I need to refresh the SPIRV-Tools hash in this repo to adopt it. But that means Python3 is forced then...)

I'm also considering download Ninja. (Well downloading is a bit unconventional so we can renegotiate this, but we are already accessing the Internet using Git anyway.)

WDYT?

@Lokathor: Your idea of shipping pre-built binaries sounds interesting; I think we can explore that.

[Lokathor]

I think that just having pre-built binary files on Win32 is the best option. It makes the build time for new builds drop down to 1-2 seconds.

Unfortunately, as i recently learned when trying to add pre-built binary files to my own bindings crate, Linux and Mac aren't nearly as friendly to the idea, and those platforms might need to keep building shaderc from source. They don't have the path limit to worry about at least, though it still takes ages.