Add support for global build defines and options (#8504)

* Add support to specify global build defines and options

A script manages the use of a file with a unique name, like
`SketchName.ino.globals.h`, in the Sketch source directory to provide compiler
command-line options (build options) and sketch global defines. The build
option data is encapsulated in a unique "C" comment block and extracted into
the build tree during prebuild.

* Applied os.path.normpath() liberally to input arguments. Fixes windows file path issue.
Improved helpful message for adding embedded build options.

* doubleup '\'

* Added context help for build option support

* expunged sketchbook global
added workaround for aggressive caching

* inital pass at searching for and reading preferences.txt

* Correct Windows path for preferences.txt
Added portable path for preferences.txt
Expanded file timestamp granularity
Improved error message printing for Arduino IDE 2.0 RC4

* Improved portable path and various Windows paths to preferences.txt

* Add cleanup logic and identify 1st run after IDE restart

* text corrections

* Create mkbuildoptglobals.py

When global header file does not exist, this print makes it easier for user to create the header file by providing its name and documentation pointer.

* build.opt heads up to user

Compiler command line changes from build.opt are shown to user

* Updated text

* oops

* Expanded comment and made print help consistent

* Improve handling stderr/stdout with "no verbose output"
Grouped helpful info to print at the end.
Added missing return value.

* Correct timestamp on CommonHFile.h
More improvements to printing
Updated docs.

* Added command-line parser

Support hints for compiler.cache_core. For use when Arduino IDE uses
command-line options that override compiler.cache_core.

Removed overuse of ()

Improve FAQ entry

* Fix script failure under windows

Rely on argpaser for checking that all arguments are present.
Removed redundant argument check in main().

Added '--debug' option and print_dbg method.

Rethink failures on overrides. Remove well know path fallbacks,
error exit when override file is missing.

In well-known path search for preferences.txt, do not assume true.
Make failure to find an error exit event.

When Windows has two preferences.txt files and they have different
values for caching and globals.h is used, error exit. It is not
possible to know from the script which is being used.

* Use quotes on build.opt

Update comment
Include the @ within the expantion string use quotes around file name.
Update doc example to remind and use quotes.

* Update CI for build option and global support

Added "mkbuildoptglobals.extra_flags=--cache_core" to platform.loca.txt
Update "-ide-version=10802" this version number indicates aggressive caching support
Added example to test global .h support

* Add debug prints
Added --debug to CI - this needs to be removed later
Tweaks to touch...

* Give each build VM a unique build.tmp space

* Corrected style on example
temp CI changes
debug crud
Added --ci switch

* Removed CI debug crud

run_CI_locall.sh works fine locally. Hosted Multi-VM CI fails
to work with 'aggressive caching' workaround method.

Add #if defined(CORE_MOCK) to failing example.

* Try HOST_MOCK

* CI adjustments

mkbuildoptglobals.py is optimized around the Arduino IDE 1.x
behaviour. One way the CI differs from the Arduino IDE is in the
handling of core and caching core. With the Arduino IDE, each sketch
has a private copy of core and contributes to a core cache. With the
CI, there is one shared copy of core for all sketches. When global
options are used, the shared copy of core and cache are removed before
and after the build.

* Doc update

doc/faq/a06-global-build-options.rst

@@ -0,0 +1,252 @@
+How to specify global build defines and options
+===============================================
+
+To create globally usable macro definitions for a Sketch, create a file
+with a name based on your Sketch’s file name followed by ``.globals.h``
+in the Sketch folder. For example, if the main Sketch file is named
+``LowWatermark.ino``, its global ``.h`` file would be
+``LowWatermark.ino.globals.h``. This file will be implicitly included
+with every module built for your Sketch. Do not directly include it in
+any of your sketch files or in any other source files. There is no need
+to create empty/dummy files, when not used.
+
+This global ``.h`` also supports embedding compiler command-line options
+in a unique “C” block comment. Compiler options are placed in a “C”
+block comment starting with ``/*@create-file:build.opt@``. This
+signature line must be alone on a single line. The block comment ending
+``*/`` should also be alone on a single line. In between, place your
+compiler command-line options just as you would have for the GCC @file
+command option.
+
+Actions taken in processing comment block to create ``build.opt`` \* for
+each line, white space is trimmed \* blank lines are skipped \* lines
+starting with ``*``, ``//``, or ``#`` are skipped \* the remaining
+results are written to build tree\ ``/core/build.opt`` \* multiple
+``/*@create-file:build.opt@`` ``*/`` comment blocks are not allowed \*
+``build.opt`` is finished with a ``-include ...`` command, which
+references the global .h its contents were extracted from.
+
+Example Sketch: ``LowWatermark.ino``
+
+.. code:: cpp
+
+   #include <umm_malloc/umm_malloc.h>  // has prototype for umm_free_heap_size_min()
+
+   void setup() {
+     Serial.begin(115200);
+     delay(200);
+   #ifdef MYTITLE1
+     Serial.printf("\r\n" MYTITLE1 MYTITLE2 "\r\n");
+   #else
+     Serial.println("ERROR:  MYTITLE1 not present");
+   #endif
+     Serial.printf("Heap Low Watermark %u\r\n", umm_free_heap_size_min());
+   }
+
+   void loop() {}
+
+Global ``.h`` file: ``LowWatermark.ino.globals.h``
+
+.. code:: cpp
+
+   /*@create-file:build.opt@
+   // An embedded build.opt file using a "C" block comment. The starting signature
+   // must be on a line by itself. The closing block comment pattern should be on a
+   // line by itself. Each line within the block comment will be space trimmed and
+   // written to build.opt, skipping blank lines and lines starting with '//', '*'
+   // or '#'.
+
+    * this line is ignored
+    # this line is ignored
+   -DMYTITLE1="\"Running on \""
+     -O3
+   //-fanalyzer
+   -DUMM_STATS_FULL=1
+   */
+
+   #ifndef LOWWATERMARK_INO_GLOBALS_H
+   #define LOWWATERMARK_INO_GLOBALS_H
+
+   #if !defined(__ASSEMBLER__)
+   // Defines kept away from assembler modules
+   // i.e. Defines for .cpp, .ino, .c ... modules
+   #endif
+
+   #if defined(__cplusplus)
+   // Defines kept private to .cpp and .ino modules
+   //#pragma message("__cplusplus has been seen")
+   #define MYTITLE2 "Empty"
+   #endif
+
+   #if !defined(__cplusplus) && !defined(__ASSEMBLER__)
+   // Defines kept private to .c modules
+   #define MYTITLE2 "Full"
+   #endif
+
+   #if defined(__ASSEMBLER__)
+   // Defines kept private to assembler modules
+   #endif
+
+   #endif
+
+Aggressively cache compiled core
+================================
+
+This feature appeared with the release of Arduino IDE 1.8.2. The feature
+“Aggressively Cache Compiled core” refers to sharing a single copy of
+``core.a`` across all Arduino IDE Sketch windows. This feature is on by
+default. ``core.a`` is an archive file containing the compiled objects
+of ``./core/esp8266/*``. Created after your 1ST successful compilation.
+All other open sketch builds use this shared file. When you close all
+Arduino IDE windows, the core archive file is deleted.
+
+This feature is not compatible with using global defines or compiler
+command-line options. Without mediation, bad builds could result, when
+left enabled. When ``#define`` changes require rebuilding ``core.a`` and
+multiple Sketches are open, they can no longer reliably share one cached
+``core.a``. In a simple case: The 1st Sketch to be built has its version
+of ``core.a`` cached. Other sketches will use this cached version for
+their builds.
+
+There are two solutions to this issue: 1. Turn off the “Aggressively
+Cache Compiled core” feature, by setting ``compiler.cache_core=false``.
+2. Rely on the not ideal fail-safe, aggressive cache workaround built
+into the script.
+
+Using “compiler.cache_core=false”
+---------------------------------
+
+There are two ways to turn off the “Aggressively Cache Compiled core”
+feature: This can be done with the Arduino IDE command-line or a text
+editor.
+
+Using the Arduino IDE command-line from a system command line, enter the
+following:
+
+::
+
+   arduino --pref compiler.cache_core=false --save-prefs
+
+For the text editor, you need to find the location of
+``preferences.txt``. From the Arduino IDE, go to *File->Preferences*.
+Make note of the path to ``prefereces.txt``. You *cannot* edit the file
+while the Arduino IDE is running. Close all Arduino IDE windows and edit
+the file ``preferences.txt``. Change ``compiler.cache_core=true`` to
+``compiler.cache_core=false`` and save. Then each sketch will maintain
+its *own* copy of ``core.a`` built with the customization expressed by
+their respective ``build.opt`` file.
+
+The “workaround”
+----------------
+
+When the “Aggressively Cache Compiled core” feature is enabled and the
+global define file is detected, a workaround will turn on and stay on.
+When you switch between Sketch windows, core will be recompiled and the
+cache updated. The workaround logic is reset when Arduino IDE is
+completely shutdown and restarted.
+
+The workaround is not perfect. These issues may be of concern: 1. Dirty
+temp space. Arduino build cache files left over from a previous run or
+boot. 2. Arduino command-line options: \* override default
+preferences.txt file. \* override a preference, specifically
+``compiler.cache_core``. 3. Multiple versions of the Arduino IDE running
+
+**Dirty temp space**
+
+A minor concern, the workaround is always on. Not an issue for build
+accuracy, but ``core.a`` maybe rebuild more often than necessary.
+
+Some operating systems are better at cleaning up their temp space than
+others at reboot after a crash. At least for Windows®, you may need to
+manually delete the Arduino temp files and directories after a crash.
+Otherwise, the workaround logic may be left on. There is no harm in the
+workaround being stuck on, the build will be correct; however, the core
+files will occasionally be recompiled when not needed.
+
+For some Windows® systems the temp directory can be found near
+``C:\Users\<user id>\AppData\Local\Temp\arduino*``. Note ``AppData`` is
+a hidden directory. For help with this do an Internet search on
+``windows disk cleanup``. Or, type ``disk cleanup`` in the Windows®
+taskbar search box.
+
+With Linux, this problem could occur after an Arduino IDE crash. The
+problem would be cleared after a reboot. Or you can manually cleanup the
+``/tmp/`` directory before restarting the Arduino IDE.
+
+**Arduino command-line option overrides**
+
+The script needs to know the working value of ``compiler.cache_core``
+that the Arduino IDE uses when building. This script can learn the state
+through documented locations; however, the Arduino IDE has two
+command-line options that can alter the results the Arduino IDE uses
+internally. And, the Arduino IDE does not provide a means for a script
+to learn the override value.
+
+These two command-line options are the problem:
+
+::
+
+   ./arduino --preferences-file other-preferences.txt
+   ./arduino --pref compiler.cache_core=false
+
+Hints for discovering the value of ``compiler.cache_core``, can be
+provided by specifying ``mkbuildoptglobals.extra_flags=...`` in
+``platform.local.txt``.
+
+Examples of hints:
+
+::
+
+   mkbuildoptglobals.extra_flags=--preferences_sketch            # assume file preferences.txt in the sketch folder
+   mkbuildoptglobals.extra_flags=--preferences_sketch "pref.txt" # is relative to the sketch folder
+   mkbuildoptglobals.extra_flags=--no_cache_core
+   mkbuildoptglobals.extra_flags=--cache_core
+   mkbuildoptglobals.extra_flags=--preferences_file "other-preferences.txt" # relative to IDE or full path
+
+If required, remember to quote file or file paths.
+
+**Multiple versions of the Arduino IDE running**
+
+You can run multiple Arduino IDE windows as long as you run one version
+of the Arduino IDE at a time. When testing different versions,
+completely exit one before starting the next version. For example,
+Arduino IDE 1.8.19 and Arduino IDE 2.0 work with different temp and
+build paths. With this combination, the workaround logic sometimes fails
+to enable.
+
+At the time of this writing, when Arduino IDE 2.0 rc5 exits, it leaves
+the temp space dirty. This keeps the workaround active the next time the
+IDE is started. If this is an issue, manually delete the temp files.
+
+Custom build environments
+=========================
+
+Some custom build environments may have already addressed this issue by
+other means. If you have a custom build environment that does not
+require this feature and would like to turn it off, you can add the
+following lines to the ``platform.local.txt`` used in your build
+environment:
+
+::
+
+   recipe.hooks.prebuild.2.pattern=
+   build.opt.flags=
+
+Other build confusion
+=====================
+
+1. Renaming a file does not change the last modified timestamp, possibly
+   causing issues when adding a file by renaming and rebuilding. A good
+   example of this problem would be to have then fixed a typo in file
+   name ``LowWatermark.ino.globals.h``. You need to touch (update
+   timestamp) the file so a “rebuild all” is performed.
+
+2. When a ``.h`` file is renamed in the sketch folder, a copy of the old
+   file remains in the build sketch folder. This can create confusion if
+   you missed an edit in updating an ``#include`` in one or more of your
+   modules. That module will continue to use the stale version of the
+   ``.h`` until you restart the IDE or other major changes that would
+   cause the IDE to delete and recopy the contents from the source
+   Sketch directory. Changes on the IDE Tools board settings may cause a
+   complete rebuild, clearing the problem. This may be the culprit for
+   “What! It built fine last night!”

doc/faq/readme.rst

@@ -191,3 +191,12 @@ How to resolve "undefined reference to ``flashinit`'" error ?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Please read `flash layout <../filesystem.rst>`__ documentation entry.
+
+How to specify global build defines and options?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By using a uniquely named `.h` file, macro definitions can be created and
+globally used. Additionally, compiler command-line options can be embedded in
+this file as a unique block comment.
+
+`Read more <a06-global-build-options.rst>`__.

libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino

@@ -0,0 +1,31 @@
+/*
+ * Showcase the use of embedded build options and global defines through a specially named .h file.
+ * Sketch file name followed by ".globals.h", "GlobalBuildOptions.ino.globals.h"
+ *
+ * Example from https://arduino-esp8266.readthedocs.io/en/latest/faq/a06-global-build-options.html
+ *
+ * Note, we do not "#include" the special file "GlobalBuildOptions.ino.globals.h".
+ * The prebuild script will make it available to all modules.
+ *
+ * To track the new sketch name when saving this sketch to a new location and
+ * name, remember to update the global .h file name.
+ */
+
+#include <umm_malloc/umm_malloc.h>  // has prototype for umm_free_heap_size_min()
+
+void setup() {
+  Serial.begin(115200);
+  delay(200);
+
+#ifdef MYTITLE1
+  Serial.printf("\r\n" MYTITLE1 MYTITLE2 "\r\n");
+#else
+  Serial.println("ERROR:  MYTITLE1 not present");
+#endif
+
+#if defined(UMM_STATS_FULL)
+  Serial.printf("Heap Low Watermark %u\r\n", umm_free_heap_size_min());
+#endif
+}
+
+void loop() {}

libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino.globals.h

@@ -0,0 +1,39 @@
+/*@create-file:build.opt@
+  // An embedded build.opt file using a "C" block comment. The starting signature
+  // must be on a line by itself. The closing block comment pattern should be on a
+  // line by itself. Each line within the block comment will be space trimmed and
+  // written to build.opt, skipping blank lines and lines starting with '//', '*'
+  // or '#'.
+  -DMYTITLE1="\"Running on \""
+ * this line is ignored
+  *@create-file:build.opt@
+  # this line is ignored
+  -O3
+  // -fanalyzer
+  -DUMM_STATS_FULL=1
+*/
+
+#ifndef GLOBALBUILDOPTIONS_INO_GLOBALS_H
+#define GLOBALBUILDOPTIONS_INO_GLOBALS_H
+
+#if !defined(__ASSEMBLER__)
+// Defines kept away from assembler modules
+// i.e. Defines for .cpp, .ino, .c ... modules
+#endif
+
+#if defined(__cplusplus)
+// Defines kept private to .cpp and .ino modules
+//#pragma message("__cplusplus has been seen")
+#define MYTITLE2 "Empty"
+#endif
+
+#if !defined(__cplusplus) && !defined(__ASSEMBLER__)
+// Defines kept private to .c modules
+#define MYTITLE2 "~Full"
+#endif
+
+#if defined(__ASSEMBLER__)
+// Defines kept private to assembler modules
+#endif
+
+#endif

platform.txt

@@ -17,6 +17,7 @@ runtime.tools.signing={runtime.platform.path}/tools/signing.py
 runtime.tools.elf2bin={runtime.platform.path}/tools/elf2bin.py
 runtime.tools.sizes={runtime.platform.path}/tools/sizes.py
 runtime.tools.makecorever={runtime.platform.path}/tools/makecorever.py
+runtime.tools.mkbuildoptglobals={runtime.platform.path}/tools/mkbuildoptglobals.py
 runtime.tools.mkdir={runtime.platform.path}/tools/mkdir.py
 runtime.tools.cp={runtime.platform.path}/tools/cp.py
 runtime.tools.eboot={runtime.platform.path}/bootloaders/eboot/eboot.elf
@@ -58,11 +59,18 @@ build.spiffs_start=
 build.spiffs_end=
 build.spiffs_blocksize=
 
+# Fully qualified file names for processing sketch global options
+globals.h.source.fqfn={build.source.path}/{build.project_name}.globals.h
+commonhfile.fqfn={build.core.path}/CommonHFile.h
+build.opt.fqfn={build.path}/core/build.opt
+build.opt.flags="@{build.opt.fqfn}"
+mkbuildoptglobals.extra_flags=
+
 compiler.path={runtime.tools.xtensa-lx106-elf-gcc.path}/bin/
 compiler.sdk.path={runtime.platform.path}/tools/sdk
 
 compiler.libc.path={runtime.platform.path}/tools/sdk/libc/xtensa-lx106-elf
-compiler.cpreprocessor.flags=-D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -D_GNU_SOURCE -DESP8266 "-I{compiler.sdk.path}/include" "-I{compiler.sdk.path}/{build.lwip_include}" "-I{compiler.libc.path}/include" "-I{build.path}/core"
+compiler.cpreprocessor.flags=-D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -D_GNU_SOURCE -DESP8266 {build.opt.flags} "-I{compiler.sdk.path}/include" "-I{compiler.sdk.path}/{build.lwip_include}" "-I{compiler.libc.path}/include" "-I{build.path}/core"
 
 # support precompiled libraries in IDE v1.8.6+
 compiler.libraries.ldflags=
@@ -107,7 +115,11 @@ compiler.elf2hex.extra_flags=
 ## needs git
 recipe.hooks.sketch.prebuild.pattern="{runtime.tools.python3.path}/python3" -I "{runtime.tools.signing}" --mode header --publickey "{build.source.path}/public.key" --out "{build.path}/core/Updater_Signing.h"
 # This is quite a working hack. This form of prebuild hook, while intuitive, is not explicitly documented.
-recipe.hooks.prebuild.pattern="{runtime.tools.python3.path}/python3" -I "{runtime.tools.makecorever}" --build_path "{build.path}" --platform_path "{runtime.platform.path}" --version "{version}"
+recipe.hooks.prebuild.1.pattern="{runtime.tools.python3.path}/python3" -I "{runtime.tools.makecorever}" --build_path "{build.path}" --platform_path "{runtime.platform.path}" --version "{version}"
+
+# Handle processing sketch global options
+recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{runtime.ide.path}" {runtime.ide.version} "{build.path}" "{build.opt.fqfn}" "{globals.h.source.fqfn}" "{commonhfile.fqfn}" {mkbuildoptglobals.extra_flags}
+
 
 ## Build the app.ld linker file
 recipe.hooks.linking.prelink.1.pattern="{runtime.tools.python3.path}/python3" -I "{runtime.tools.mkdir}" -p "{build.path}/ld_h/"

tests/build.sh

@@ -19,4 +19,3 @@ install_arduino nodebug
 build_sketches_with_arduino "$mod" "$rem" lm2f
 
 rm -rf "$cache_dir"
-