ESP-IDF-4 - notes regarding building on windows from earlier versions

[wilberforce]

Build environment: Windows
Target device: ESP32

Notes on updating from IDF 3.2

errors around Python module click not found

Check python27 is on the system path and getting found before python 3:
remove:

C:\Python27\
C:\Python27\Scripts

Launch script

C:\Windows\system32\cmd.exe /k ""C:\Users\<username>\.espressif\idf_cmd_init.bat" "C:\Users\<username>\AppData\Local\Programs\Python\Python38-32\" "C:\Program Files\Git\cmd\""

can be simplified to:

C:\Windows\system32\cmd.exe /k ""%USERPROFILE%.espressif\idf_cmd_init.bat" "%USERPROFILE%\AppData\Local\Programs\Python\Python38-32\" "C:\Program Files\Git\cmd\""

with the setup as documented, when launching the cmd prompt I see:

This script must be invoked from ESP-IDF directory.

set env var: MODDABLE_ESP32_LAUNCH
""C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars32.bat" && pushd %IDF_PATH% && "%USERPROFILE%\.espressif\idf_cmd_init.bat" "%USERPROFILE%\AppData\Local\Programs\Python\Python38-32\" "C:\Program Files\Git\cmd\" && popd"

and in the target set the directory to default repos folder.

When the popd is execute the cmd prompt is then in your repository folders.

[andycarle]

Hi @wilberforce — thank you for the suggestions and for trying out the ESP-IDF 4.2 preview!

Good point on de-listing old Python versions. I'll make a note about that in the "Windows-Updating" section of the docs.

You're right that using %USERPROFILE% can simplify that shortcut; I'll recommend that as well.

On the This script must be invoked from ESP-IDF directory error, that probably means that you did not start with a copy of the ESP-IDF Command Prompt shortcut. They set the "Start in" property of the shortcut to be the IDF Path to avoid that problem. But your suggestion is more general, so I think I prefer it. (As long as people are comfortable doing the environment variable solution. I worried while writing that instruction, because it is a complicated fix to the problem!)

[wilberforce]

Thanks @andycarle

1. On my desktop machine - I had python 3.5 installed and intially used that. The was missing the click module so perhaps recommend installing the suggest later version of python?

2. I have managed to shorten the launch target by using more env vars, so the MODDABLE_ESP32_LAUNCH is not needed:

%comspec% /k ""%ProgramFiles(x86)%\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars32.bat" && pushd %IDF_PATH% && %IDF_TOOLS_PATH%\idf_cmd_init.bat %LOCALAPPDATA%\Programs\Python\Python38-32 %ProgramFiles(x86)%\Git\cmd && popd"

3. in the docs here:

https://github.com/Moddable-OpenSource/moddable/blob/esp-idf-4/documentation/Moddable%20SDK%20-%20Getting%20Started.md#win-instructions

There are references to C:\Users\<user> that could be simplified to use %USERPROFILE%

4. For the intial build of tools, I have added: rebuild.bat

rmdir /S /Q ..\..\tmp
rmdir /S /Q ..\..\bin
build.bat

Thinking about this - do we just need to add the rmdir to the start of the normal build script?

[andycarle]

Thank you for the documentation suggestions, @wilberforce! I love all this shortcut property cleverness... it feels very retro.

I've incorporated your suggestions into our documentation locally. That will make it to GitHub in our next open source push.

For the intial build of tools, I have added: rebuild.bat
rmdir /S /Q ....\tmp
rmdir /S /Q ....\bin
build.bat
Thinking about this - do we just need to add the rmdir to the start of the normal build script?

This one I'm a bit hesitant to do. You're right that making this change would solve a common problem for new users. But it is also often sufficient to do an incremental build of the tools--especially for those of us who actively work on the tools themselves.

One option here would be to have build.bat take an argument to choose which behavior you want. Or, as you note, to have two different batch files for the two types of builds... I'll think on that one a bit.

[wilberforce]

@andycarle

build clean would be good way of supporting both

[andycarle]

Yeah, I like the idea of having build clean as an option. I've added that locally; it will show up on GitHub in our next push.

[andycarle]

Speaking of Windows build improvements... @wilberforce, would you be willing to test something for me?

On my own Windows machine, I've been running for months with a set of changes that improve the parallelism of our (non-device) builds. They work well on my system, but I would love to hear if they are stable and helpful on other build hosts.

The changes speed up builds for the simulator a bit, but the difference is most dramatic on tools builds.

There's a set of changes to use better makefile rules/recipes to get some multi-processor building out of cl. On my system, that takes a clean build of tools from about 2 minutes to about 1:30.

Then, for the truly adventurous, there's a new (optional) mp_build.bat that uses multiple command prompt instances to really get some parallelism going in the tools build. That takes my tools build all the way down to 30 seconds. It's a little wonky, but perhaps useful to some of us.

I've attached those changes as a diff off of the current GitHub top-of-tree. If you feel like giving them a try, let me know how they turn out!

 - Andy

(reposted to fix an issue with the attachment)

windows_mp.zip

[wilberforce]

Hi,

On my crusty laptop - the tools normal build takes 6:30 ( it might have been slightly quicker - as I missed the stop watch near the end)

I had trouble applying you patch:

F:\Projects\moddable>code windows_mp.diff

F:\Projects\moddable>git apply windows_mp.diff
error: build/makefiles/win/mp_build.bat: already exists in working directory
error: patch failed: build/makefiles/win/serial2xsbug.mak:1
error: build/makefiles/win/serial2xsbug.mak: patch does not apply
error: patch failed: build/makefiles/win/simulator.mak:1
error: build/makefiles/win/simulator.mak: patch does not apply
error: patch failed: build/makefiles/win/tools.mak:1
error: build/makefiles/win/tools.mak: patch does not apply
error: patch failed: tools/mcconfig/nmake.win.mk:1
error: tools/mcconfig/nmake.win.mk: patch does not apply
error: patch failed: tools/mcconfig/nmake.x-cli-win.mk:119
error: tools/mcconfig/nmake.x-cli-win.mk: patch does not apply
error: patch failed: tools/mcconfig/nmake.x-win.mk:1
error: tools/mcconfig/nmake.x-win.mk: patch does not apply
error: patch failed: xs/makefiles/win/xsc.mak:1
error: xs/makefiles/win/xsc.mak: patch does not apply
error: patch failed: xs/makefiles/win/xsid.mak:1
error: xs/makefiles/win/xsid.mak: patch does not apply
error: patch failed: xs/makefiles/win/xsl.mak:1
error: xs/makefiles/win/xsl.mak: patch does not apply
error: patch failed: xs/makefiles/win/xst.mak:1
error: xs/makefiles/win/xst.mak: patch does not apply

Perhaps if you do this as another branch I could then pull that?

The top of tree branch I have does not seem to have the build clean option.

[wilberforce]

I have pulled from the main public branch and updated the build tools. Everything built as expected.