Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add prebuild script to use additional build options #8095

Conversation

brainelectronics
Copy link

Additional compile arguments can be given by a file named "build_opt.h" and shall be placed next to the Sketch.ino file.
This enables users to easily extend their arguments on a central place.
The script has been taken from stm32duino's Arduino_Core_STM32 package

Additional compile arguments can be given by a file named "build_opt.h".
This enables users to easily extend their arguments on a central place.
The script has been taken from stm32duino's Arduino_Core_STM32 package
Copy link
Collaborator

@earlephilhower earlephilhower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea, but there are a few changes needed to get it going. Thanks for the PR!

tools/extras/prebuild.sh Outdated Show resolved Hide resolved
platform.txt Show resolved Hide resolved
platform.txt Show resolved Hide resolved
platform.txt Outdated Show resolved Hide resolved
Copy link
Collaborator

@earlephilhower earlephilhower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM! I was thinking we should add something to the docs about this new option, but I can't find anywhere that it fits in well presently in the doc/*.rst directory. If you think it might make sense to add a couple lines in the faq.rst about this, that would be icing on the cake.

Copy link
Collaborator

@d-a-v d-a-v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personnally like very much this addition.
It is a long waited feature in Arduino IDE that is still not addressed, I will cite arduino/Arduino#421 and arduino/arduino-cli#846. I also made a proposal that was not dismissed, nor closed nor commented yet.
It's a pity that a global solution is not available. This forces platforms to adopt local solution like this one.
Approving, and thanks for bringing this.

@mhightower83
Copy link
Contributor

@brainelectronics - This is great!

I think, when you create an empty build_opt.h, you need to backdate the file to avoid triggering a rebuild of libraries.

For a while now, I have been doing something similar to this PR. I have noticed that core '.S' files are not rebuilt when, for your case, the build_opt.h is changed. You may want to confirm your results and if necessary add a comment to the docs.
Look for something like this:

Using previously compiled file: /tmp/arduino_build_327471/core/cont.S.o

while everything else in core is rebuilt.

@d-a-v
Copy link
Collaborator

d-a-v commented Jun 3, 2021

I think, when you create an empty build_opt.h, you need to backdate the file to avoid triggering a rebuild of libraries.

True!
It should take users's build_opt.h date (it may have been updated and in this case, recompilation is necessary) or, if it does not exist, be created to the same date as any other existing source file (sketch/sketch.ino? core/core_version.h? )

@d-a-v d-a-v added alpha included in alpha release and removed alpha included in alpha release labels Jun 4, 2021
@brainelectronics
Copy link
Author

PR is ready for review again and could be merged if alpha testing has been passed

@d-a-v d-a-v added the alpha included in alpha release label Jun 12, 2021
@d-a-v
Copy link
Collaborator

d-a-v commented Jun 15, 2021

I'm using this sketch

#ifndef BLAH
#error
#endif

and this build_opt.h

#define BLAH toto

I made a build with arduino-builder with linux and got this:

Using board 'd1_mini' from platform in folder: /local/users/gauchard/arduino/arduino-210516/hardware/esp8266com/esp8266
Using core 'esp8266' from platform in folder: /local/users/gauchard/arduino/arduino-210516/hardware/esp8266com/esp8266
/local/users/gauchard/arduino/arduino-210516/hardware/esp8266com/esp8266/tools/python3/python3 /local/users/gauchard/arduino/arduino-210516/hardware/esp8266com/esp8266/tools/prebuild.py /tmp/build_opt_test-d1 /home/gauchard/dev/proj/arduino/build_opt_test
Traceback (most recent call last):
  File "/local/users/gauchard/arduino/arduino-210516/hardware/esp8266com/esp8266/tools/prebuild.py", line 121, in <module>
    sys.exit(main())
  File "/local/users/gauchard/arduino/arduino-210516/hardware/esp8266com/esp8266/tools/prebuild.py", line 117, in main
    create_build_options_file(source_path, build_path)
  File "/local/users/gauchard/arduino/arduino-210516/hardware/esp8266com/esp8266/tools/prebuild.py", line 62, in create_build_options_file
    build_opt_ctime = get_creation_time(build_opt_build_path)
  File "/local/users/gauchard/arduino/arduino-210516/hardware/esp8266com/esp8266/tools/prebuild.py", line 108, in get_creation_time
    return os.path.getctime(file_path)
  File "/usr/lib/python3.8/genericpath.py", line 65, in getctime
    return os.stat(filename).st_ctime
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/build_opt_test-d1/sketch/build_opt.h'
exit status 1

Windows alpha version 0.0.2 contains your PR (thanks to the tag alpha) and I get the same error (sketch/build_opt.h not found) using this time Arduino IDE.

edit: I removed the alpha tag so I could test other unmerged PRs with windows

@brainelectronics
Copy link
Author

@d-a-v That's strange. As you placed the build_opt.h next to the your-sketch.ino file, Arduino should take care of copying that build options file to the tmp build directory. This seems not to be the case. Could you check the content of the build directory? It should be similar to this content

./build.options.json
./your-sketch.ino.bin
./your-sketch.ino.elf
./your-sketch.ino.hex
./your-sketch.ino.map
./sketch/build_opt.h
./sketch/your-sketch.ino.cpp
./sketch/your-sketch.ino.cpp.d
./sketch/your-sketch.ino.cpp.o

@d-a-v d-a-v removed the alpha included in alpha release label Jun 15, 2021
@earlephilhower earlephilhower added this to the 3.1 milestone Jun 17, 2021
earlephilhower
earlephilhower previously approved these changes Jun 17, 2021
Copy link
Collaborator

@earlephilhower earlephilhower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, still. Thanks! Slated for 3.1.0, after the 3.0.x bugfixes are all settled.

@earlephilhower earlephilhower self-requested a review June 17, 2021 22:57
@earlephilhower
Copy link
Collaborator

The contents of the arduino_buid/sketch directory are unpopulated at the time of failure. Maybe the python is running too soon in the process and needs to be run elsewhere?

earle@server:~/Arduino/hardware/esp8266com/esp8266$ ls -lR /tmp/arduino_build_231337/
/tmp/arduino_build_231337/:
total 8
-rw-r--r-- 1 earle earle  910 Jun 17 17:57 build.options.json
drwxrwxr-x 2 earle earle 4096 Jun 17 17:57 sketch

/tmp/arduino_build_231337/sketch:
total 0

@brainelectronics
Copy link
Author

I'll try to check & fix the issue on the weekend

@d-a-v d-a-v added the alpha included in alpha release label Jun 23, 2021
@d-a-v
Copy link
Collaborator

d-a-v commented Jun 28, 2021

I had to do this change to make it work under linux:

(@file => -include file)

I did not try with windows, but without it, I get the same error as with linux.

Then it seems to work well and be used for every single file from core and external libs :)

diff --git a/platform.txt b/platform.txt
index 555171d4..551e1a14 100644
--- a/platform.txt
+++ b/platform.txt
@@ -115,13 +115,13 @@ recipe.hooks.linking.prelink.3.pattern="{compiler.path}{compiler.c.cmd}" -CC -E
 recipe.hooks.linking.prelink.4.pattern="{compiler.path}{compiler.c.cmd}" -CC -E -P {build.vtable_flags} {build.mmuflags} "{runtime.platform.path}/tools/sdk/ld/eagle.app.v6.common.ld.h" -o "{build.path}/local.eagle.app.v6.common.ld"
 
 ## Compile c files
-recipe.c.o.pattern="{compiler.path}{compiler.c.cmd}" {compiler.cpreprocessor.flags} {compiler.c.flags} -D{build.sdk}=1 -DF_CPU={build.f_cpu} {build.lwip_flags} {build.debug_port} {build.debug_level} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} -DARDUINO_BOARD="{build.board}" {build.led} {build.flash_flags} {compiler.c.extra_flags} {build.extra_flags} "@{build.opt.path}" {includes} "{source_file}" -o "{object_file}"
+recipe.c.o.pattern="{compiler.path}{compiler.c.cmd}" {compiler.cpreprocessor.flags} {compiler.c.flags} -D{build.sdk}=1 -DF_CPU={build.f_cpu} {build.lwip_flags} {build.debug_port} {build.debug_level} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} -DARDUINO_BOARD="{build.board}" {build.led} {build.flash_flags} {compiler.c.extra_flags} {build.extra_flags} -include "{build.opt.path}" {includes} "{source_file}" -o "{object_file}"
 
 ## Compile c++ files
-recipe.cpp.o.pattern="{compiler.path}{compiler.cpp.cmd}" {compiler.cpreprocessor.flags} {compiler.cpp.flags} -D{build.sdk}=1 -DF_CPU={build.f_cpu} {build.lwip_flags} {build.debug_port} {build.debug_level} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} -DARDUINO_BOARD="{build.board}" {build.led} {build.flash_flags} {compiler.cpp.extra_flags} {build.extra_flags} "@{build.opt.path}" {includes} "{source_file}" -o "{object_file}"
+recipe.cpp.o.pattern="{compiler.path}{compiler.cpp.cmd}" {compiler.cpreprocessor.flags} {compiler.cpp.flags} -D{build.sdk}=1 -DF_CPU={build.f_cpu} {build.lwip_flags} {build.debug_port} {build.debug_level} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} -DARDUINO_BOARD="{build.board}" {build.led} {build.flash_flags} {compiler.cpp.extra_flags} {build.extra_flags} -include "{build.opt.path}" {includes} "{source_file}" -o "{object_file}"
 
 ## Compile S files
-recipe.S.o.pattern="{compiler.path}{compiler.c.cmd}" {compiler.cpreprocessor.flags} {compiler.S.flags} -D{build.sdk}=1 -DF_CPU={build.f_cpu} {build.lwip_flags} {build.debug_port} {build.debug_level} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} -DARDUINO_BOARD="{build.board}" {build.led} {build.flash_flags} {compiler.S.extra_flags} {build.extra_flags} "@{build.opt.path}" {includes} "{source_file}" -o "{object_file}"
+recipe.S.o.pattern="{compiler.path}{compiler.c.cmd}" {compiler.cpreprocessor.flags} {compiler.S.flags} -D{build.sdk}=1 -DF_CPU={build.f_cpu} {build.lwip_flags} {build.debug_port} {build.debug_level} -DARDUINO={runtime.ide.version} -DARDUINO_{build.board} -DARDUINO_ARCH_{build.arch} -DARDUINO_BOARD="{build.board}" {build.led} {build.flash_flags} {compiler.S.extra_flags} {build.extra_flags} -include "{build.opt.path}" {includes} "{source_file}" -o "{object_file}"
 
 ## Create archives
 recipe.ar.pattern="{compiler.path}{compiler.ar.cmd}" {compiler.ar.flags} {compiler.ar.extra_flags} "{archive_file_path}" "{object_file}"

@d-a-v d-a-v added waiting for feedback Waiting on additional info. If it's not received, the issue may be closed. and removed alpha included in alpha release labels Jun 28, 2021
@d-a-v
Copy link
Collaborator

d-a-v commented Jun 28, 2021

I have to explain my last comment.
I realized later that this file had to contain gcc cmdline options -Dblah=123 or -fdo-some-optimization.
So @ is good, not -include.
But here's what I got when trying with this file:

-DBLAH
-DBLOH=1

It was copied to the tmp dir with this content:

#line 1 "/home/gauchard/dev/proj/arduino/8095/build_opt.h"
-DBLAH
-DBLOH=1

And that gave this error message when compiling:

xtensa-lx106-elf-g++: error: #line: No such file or directory
xtensa-lx106-elf-g++: error: 1: No such file or directory

@d-a-v
Copy link
Collaborator

d-a-v commented Jun 29, 2021

I had a try with your PR.
.h files are preprocessed by arduino-builder so I updated your script to:

  • use the filename without the .h extension,
  • copy the file if it exists,
  • create/truncate the destination file if source does not exist.

It seems to work locally and this is a great feature.

There is a caveat: the project must be cleared and rebuilt when the option file is changed. That was anyway already the case (everything had to be compiled and not only the sketch). Maybe the python script can do something about that.

Another caveat is that it becomes incompatible with the STM32 version.
Why is the .h file not updated by arduino-cli within the STM32 platform ?

Here's my proposal against your current patch.
I removed your date functions but they should be kept if implementing something about the above caveat.

diff --git a/platform.txt b/platform.txt
index 223c2cf5..48d06a7a 100644
--- a/platform.txt
+++ b/platform.txt
@@ -87,7 +87,7 @@ compiler.size.cmd=xtensa-lx106-elf-size
 build.extra_flags=-DESP8266
 
 # Pre and post build hooks
-build.opt.name=build_opt.h
+build.opt.name=build_opt
 build.opt.path={build.path}/sketch/{build.opt.name}
 
 # Create empty {build.opt} if it does not exist in the output sketch dir
diff --git a/tools/prebuild.py b/tools/prebuild.py
index 3b074ca6..1db9dc20 100644
--- a/tools/prebuild.py
+++ b/tools/prebuild.py
@@ -12,7 +12,7 @@
 
 import os
 import sys
-
+from shutil import copyfile
 
 def create_sketch_dir(build_path):
     """
@@ -26,96 +26,21 @@ def create_sketch_dir(build_path):
         os.makedirs(sketch_build_path)
 
 
-def create_build_options_file(source_path, build_path, build_opt_name="build_opt.h"):
+def create_build_options_file(source_path, build_path, build_opt_name="build_opt"):
     """
-    Create the build options file in the build directory.
-    The modification time of the build options file is set to the sketch
-    modification time in case the file did not exist or the users build options
-    modification time to reflect changes which require a recompilation
-    :param      source_path:     The path to the source directory
-    :type       source_path:     str
-    :param      build_path:      The path to the build directory
-    :type       build_path:      str
-    :param      build_opt_name:  The build option file name
-    :type       build_opt_name:  str, optional
+    Copy the build options file to the build directory,
+    or create an empty file there if source does not exist.
+    Creation or modification time does not matter.
     """
     build_opt_source_path = os.path.join(source_path, build_opt_name)
     build_opt_build_path = os.path.join(build_path, "sketch", build_opt_name)
 
     # check for an existing build options file in the source directory
     if os.path.exists(build_opt_source_path):
-        # user does have/use an build options file in it's sketch directory
-
-        if os.path.exists(build_opt_build_path):
-            # set build options file modification time to the same time as the
-            # sketch modification time to avoid rebuilding libraries
-            set_file_time_to_sketch_time(source_path, build_opt_build_path)
-        else:
-            # the build options file does not yet exist in the build directory
-            # just continue as it will be copied by the IDE automatically on
-            # time. This section is only entered on the very first run
-            pass
+        copyfile(build_opt_source_path, build_opt_build_path)
     else:
-        # user does not have a build options file, create empty one to avoid
-        # compilation error due to missing file for compilation
-        open(build_opt_build_path, 'a').close()
-
-        # set build options file modification time to the same time as the
-        # sketch modification time to avoid rebuilding libraries
-        set_file_time_to_sketch_time(source_path, build_opt_build_path)
-
-
-def set_file_time_to_sketch_time(source_path, destination_path):
-    """
-    Set the file ctime and mtime to the ctime and mtime of the sketch.
-    :param      source_path:       The source path to the sketch
-    :type       source_path:       str
-    :param      destination_path:  The destination path to the file to modify
-    :type       destination_path:  str
-    """
-    sketch_path = get_full_sketch_path(source_path)
-    sketch_mtime = get_modification_time(sketch_path)
-    destination_ctime = get_creation_time(destination_path)
-
-    os.utime(destination_path, (destination_ctime, sketch_mtime))
-
-
-def get_full_sketch_path(source_path):
-    """
-    Get the full sketch path.
-    :param      source_path:  The source path
-    :type       source_path:  str
-    :returns:   The full sketch path including the '.ino' extension.
-    :rtype:     str
-    """
-    base_name = os.path.basename(source_path)
-
-    sketch_path = os.path.join(source_path, base_name + '.ino')
-
-    return sketch_path
-
-
-def get_modification_time(file_path):
-    """
-    Get the modification time of a file.
-    :param      file_path:  The file path
-    :type       file_path:  str
-    :returns:   The modification time.
-    :rtype:     float
-    """
-    return os.path.getmtime(file_path)
-
-
-def get_creation_time(file_path):
-    """
-    Gets the creation time if a file.
-    :param      file_path:  The file path
-    :type       file_path:  str
-    :returns:   The creation time.
-    :rtype:     float
-    """
-    return os.path.getctime(file_path)
-
+        # create empty file
+        open(build_opt_build_path, 'w').close()
 
 def main():
     if len(sys.argv) == 3:

@mcspr
Copy link
Collaborator

mcspr commented Jun 29, 2021

About the

#line 1 "/home/gauchard/dev/proj/arduino/8095/build_opt.h"
-DBLAH
-DBLOH=1

And that gave this error message when compiling:

xtensa-lx106-elf-g++: error: #line: No such file or directory
xtensa-lx106-elf-g++: error: 1: No such file or directory

see arduino/arduino-cli#1338

@earlephilhower
Copy link
Collaborator

Copying the file directly like @d-a-v suggests may be the best way to get around this.

Alternatively, would it be possible to just post-process the pre-processed header file to do an equivalent sed 's/#.*$// and remove anything on any line after a #?

@d-a-v
Copy link
Collaborator

d-a-v commented Jun 29, 2021

Both would work for us.
If some sort of unification with STM32 is seeked, then python is needed on their side, or arduino-cli should be fixed regarding @mcspr's comment.
@brainelectronics what do you think ?

edit
that one is also very nice. It seems stm32 platform relies on bash, so on WSL2 ?

@fpistm
Copy link

fpistm commented Jul 6, 2021

that one is also very nice. It seems stm32 platform relies on bash, so on WSL2 ?

No not on WSL2, we simply have a busybox.exe for windows provided with the tools 😉

Just FYI, I'm waiting Arduino feedback about this. I would really like they allows some other extension like .opt which will be less confusing than .h.

@d-a-v
Copy link
Collaborator

d-a-v commented Jul 8, 2021

@fpistm
Did you see the above comment ?
Would it fit with your platform, or is the .h extension old enough to be a breaking change for you ?

@fpistm
Copy link

fpistm commented Jul 9, 2021

@fpistm
Did you see the above comment ?
Would it fit with your platform, or is the .h extension old enough to be a breaking change for you ?

I did not check your proposal. About breaking change well, I always try to avoid it but sometimes we have no choice.
anyway simply change file name could easily be managed without breaking anything.

@earlephilhower
Copy link
Collaborator

One more vote for not using the .h extension. It’s not a real header file, so don’t make users think it is. .opt or something would be very obviously different from the end user perspective.

I’d say there is no backward compatibility issue at all because the compiler flags between different architectures will almost always be different…

@d-a-v
Copy link
Collaborator

d-a-v commented Jul 9, 2021

Then @brainelectronics @fpistm @earlephilhower, would build.opt be a good compromise ?
@brainelectronics I understand this PR is a way to uniformize this feature accross stm32 and esp8266 arduino platforms, so we do need to reach to an agreement until Arduino provides a more "official" way for doing so when using the IDE, don't we ?

@brainelectronics
Copy link
Author

@d-a-v fully agree. Naming the file build.opt would be a good idea. Copying that file maybe as build.opt.h to the sketch compilation folder would be done by the script and thereby not confusing the user with a header file name.
Of course I would like to reach a unification with STM32duino package, so users can use the build options in both worlds.
I'll also update the docs about @mcspr's issue of using multiple arg lines in the file

@brainelectronics
Copy link
Author

I had a try with your PR.
.h files are preprocessed by arduino-builder so I updated your script to:

Basically that described issues should have been solved with the timestamp manipulation. At least that was the intension. I'll try to check again next week as I'm currently quite busy with another project

@d-a-v d-a-v added the merge-conflict PR has a merge conflict that needs manual correction label Jul 11, 2021
@fpistm
Copy link

fpistm commented Jul 12, 2021

@d-a-v
I've made a PR to fix the issue brought by arduino-cli. See stm32duino/Arduino_Core_STM32#1442
Very simple and backward compatible. 😉

@d-a-v
Copy link
Collaborator

d-a-v commented Jul 17, 2021

@brainelectronics
Regarding your current PR, do you agree with @fpistm 's changes in STM32 repo and adopt the new naming build.opt instead of build_opt.h for the file containing gcc cmdline options ?
It would be more easy to understand (.h files usually contains C source code) and would be a nice workaround for the issue introduced by arduino-builder (and the future ones regarding source files .h management).

@d-a-v d-a-v removed the merge-conflict PR has a merge conflict that needs manual correction label Jul 17, 2021
@d-a-v
Copy link
Collaborator

d-a-v commented Sep 13, 2021

@fpistm 's stm32duino/Arduino_Core_STM32#1442 is merged,
so what's the status of @brainelectronics 's PR - this PR ?

@mhightower83
Copy link
Contributor

Never mind, I seem to be out of sync with all the other threads out there, enough so that I am confused as to what they have decided and where it is implemented. In any event, the example I did works; however, it may not be the direction everyone is going. Sorry for the noise.

@mhightower83
Copy link
Contributor

Focusing on PR#1442

@brainelectronics If you are stuck or have just been too busy, here is my interpretation of the immediate contents of stm32duino/Arduino_Core_STM32#1442 (comment). And, how to resolve some of the shortfalls.

I say "immediate contents" because I get lost trying to see beyond that PR. There are a lot of additional thoughts and discussions out there concerning expanding support for build options on the Arduino IDE; however, those discussions have been ongoing for many years. My guess is they will continue for many more.

Observations of build_opt.h in PR#1442

  • To hold the command-line options, the sketch folder optionally contains the build_opt.h file.
  • It must not contain any "C" code and must not be referenced by any #include statements.
  • At build time, build_opt.h in the source sketch folder is copied to the build sketch folder as build.opt.
  • There is no global dependency to respond to build_opt.h changes.
  • If you change build_opt.h, you may need to restart the Arduino IDE to get a good build.
  • After global dependencies are addressed, need to address the issue of the default, "Aggressively cache compiled core".
  • While having non "C" code in a .h is a bit odd, the file having a .h extension allows you to edit it with the Arduino-IDE.

An easy solution for a global dependency is to create a proxy .h file and use @d-a-v 's -include ... suggestion. Updated the proxy file each time a new version of build.opt is written.

To confirm, that the ideas expressed above work, I updated your script, expanded on the thoughts in PR#1442, and added dependency support.

Revised script:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# prebuild.py — This script manages updates to build.opt in the build directory
# with changes from build_opt.h in the Sketch directory.
#
# Copy build_opt.h from the source sketch folder to build.opt in the build
# sketch folder. If source sketch folder does not have a build_opt.h file,
# create am empty build_opt.h files in the build sketch folder.
#
# For dependencies to work, maintain timestamp on a proxy .h file for build.opt.
#
# Written by brainelectronics, 2021.
#
# Update by M Hightower, 2022.
#
"""
"Aggressively cache compiled core" must be turned off for a reliable build process.
In ~/.arduino15/preferences.txt, to disable the feature:
  compiler.cache_core=false

Reference:
https://forum.arduino.cc/t/no-aggressively-cache-compiled-core-in-ide-1-8-15/878954/2
"""

"""
# Updates or Additions for platform.txt or platform.local.txt

runtime.tools.prebuild={runtime.platform.path}/tools/prebuild.py

build.opt.source.path={build.source.path}/build_opt.h
build.opt.path={build.path}/sketch/build.opt
build_opt.proxy.path={build.path}/sketch/PROXY.build_opt.h

recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.prebuild}" "{build.opt.source.path}" "{build.opt.path}" "{build_opt.proxy.path}"

compiler.cpreprocessor.flags=-D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -D_GNU_SOURCE @{build.opt.path} -include "{build_opt.proxy.path}" "-I{compiler.sdk.path}/include" "-I{compiler.sdk.path}/{build.lwip_include}" "-I{compiler.libc.path}/include" "-I{build.path}/core"
"""
import os
import sys
from shutil import copyfile

def main():
    if len(sys.argv) == 4:
        src_build_opt_path = sys.argv[1]
        dst_build_opt_path = sys.argv[2]
        dst_build_opt_proxy_path = sys.argv[3]

        build_path_sketch, not_used = os.path.split(dst_build_opt_path)

        if not os.path.exists(build_path_sketch):
            os.makedirs(build_path_sketch)

        # Conditionally copy a newer build_opt.h from source directory to the
        # build/sketch directory as build.opt. When build_opt.h does not exist
        # in the source  directory, create an empty build.opt file in the
        # build/sketch directory.
        if os.path.exists(src_build_opt_path):
            if (os.path.exists(dst_build_opt_path)) and \
            (os.path.getmtime(dst_build_opt_path) >= os.path.getmtime(src_build_opt_path)):
                # only copy newer file - do nothing, all is good
                print("Using sketch compiler command-line options in " + dst_build_opt_path)
            else:
                # The new copy gets stamped with the current time, just as other
                # files copied by `arduino-builder`.
                copyfile(src_build_opt_path, dst_build_opt_path)
                open(dst_build_opt_proxy_path, 'w').close()
                print("Updated sketch compiler command-line options in " + dst_build_opt_path)
        else: # The Sketch currently does not have a build_opt.h.
            if os.path.exists(dst_build_opt_path) and \
            os.path.getsize(dst_build_opt_path) == 0:
                # All done, the Sketch has not changed.
                pass
            else:
                # Place holder - Create an empty `build.opt` to satisfy the
                # `@file` parameter. Create proxy .h to track changes to
                # `build.opt`. This will help trigger dependency builds.
                open(dst_build_opt_path, 'w').close()
                open(dst_build_opt_proxy_path, 'w').close()

    else:
        print("Wrong argument count")
        print("  All arguments FQFN. Source path build_opt.h, Build path build.opt, Build path PROXY.build_opt.h")
        sys.exit(1)

if __name__ == '__main__':
    sys.exit(main())

Aggressive Caching of core.a

A compiler command-line option file and the Aggressively cache compiled core feature do not work well together.
For the case of build.opt in the build sketch folder, build.opt can contain defines that require core.a to be recompiled. When multiple Sketches are open, they can no longer reliably share one cached core.a. (Looks like) The 1st Sketch to be built has its version of core.a cached. Other sketches will get this cached version for their builds.

To see the issue:

  1. Close all Arduino IDE windows and do a fresh start of Arduino IDE.
  2. Do File->New, then click on Verify. Wait for the build to complete.
  3. Create another new sketch add this example code:
#include <umm_malloc/umm_malloc.h>
void setup() {
  Serial.begin(115200);
  delay(200);
  Serial.printf("\r\nRunning on empty\r\n\r\n");
  Serial.printf("Heap Low Watermark %u\r\n", umm_free_heap_size_min());
}
void loop() {}
  1. Add New Tab build_opt.h with the following:
-DUMM_STATS_FULL=1
  1. Click on Verify. The build will fail.

To use a compiler command-line option file reliably, you need to set compiler.cache_core=false. Then each sketch will maintain its own copy of core.a. The alternative when using compiler.cache_core=true, is to close all Arduino IDE sketch windows. Start and run only one instance of the IDE, while building a Sketch that uses build.opt.

@d-a-v
Copy link
Collaborator

d-a-v commented Jun 15, 2022

closed per merged #8504

@d-a-v d-a-v closed this Jun 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
waiting for feedback Waiting on additional info. If it's not received, the issue may be closed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants