libdefs: a way for user to configure libraries

[d-a-v]

Please check if the PR fulfills these requirements

• The PR has no duplicates (please search among the Pull Requests
  before creating one)
• The PR follows
  our contributing guidelines
• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)
• Not a breaking change

• What kind of change does this PR introduce?

This proposal gives the ability for any sketch to use additional configuration files for libraries that are aware of this feature.

• What is the current behavior?

This is yet another proposal for #846 and arduino/Arduino#421: Compiler flags are not addressed, global defines for sketch and libraries is the target.
It would additionally replace and close #1117.

• What is the new behavior?

A library source file (header or code) could optionally use this kind of snippet:

CoolLibrary/CoolLibrary.h

#if defined __has_include
#  if __has_include (<CoolLibrary_user_config.h>)
#    pragma message "CoolLibrary: Including local project config file <CoolLibrary_user_config.h>"
#    include <CoolLibrary_user_config.h>
#  endif
#endif

#ifndef COOLLIBRARY_BUFFER_SIZE
#define COOLLIBRARY_BUFFER_SIZE 42  // default value if sketch did not redefined it in its local config file
#endif 

CoolLibrary/CoolLibrary.cpp

#include "CoolLibrary.h"

// COOLLIBRARY_BUFFER_SIZE is 42 by default per header file
// but it can also be overridden by configuration file from sketch directory and taken into account in cpp file
static const unsigned char libbuf [COOLLIBRARY_BUFFER_SIZE];
...

A sketch using this library can create a local subdirectory called libdefs/ and store, in this specific case, a file named CoolLibrary_user_config.h. Name and content of this file need to match the specific library requirements.

The goal for the library maintainer is to willingly allow users to change default values, or provide some specific configuration (like #define COOLLIBRARY_DEBUG_MODE) without having to force users to modify the library files (header or code). #1117 may give more details on the same subject with additional examples.

For security, files added in libdefs/ are accessible during the compilation phases of libraries and sketch, but not when core source files are involved.

[ubidefeo]

hi @d-a-v
unfortunately while your solution would be clever, you'd have to convince thousands of libraries maintainers to adopt it.
We're working on a system to allow such defines to be tackled without asking anything from existing developers.
Once it's in place it would just allow anyone to do (which already is in place for libraries supporting override through other means of building Arduino Sketches)

#ifndef MACRO
#define MACRO XXX
#endif

It will come as an expansion to this RFC
https://github.com/arduino/tooling-rfcs/blob/main/RFCs/0003-build-profiles.md

and configuration will happen in the sketch.yaml file as an additional field

[ubidefeo]

@d-a-v
also take vision of this issue
arduino/tooling-rfcs#9

[d-a-v]

@ubidefeo Thanks for reviewing !
I read your answer and you are right, it will be difficult to tell and convince library devs that they are now able to use these preprocessing options.

The new system you're working on looks promising and shall allow to automate builds in a more fluent way than today, for users who are used to arduino-cli.

Through my attempts, my objective is similar to part of what describes the RFC in your link above, but they don't have the same goal. The single issue I'm trying to address is about that having to modify a library with different parameters for different sketches is counter-productive.
While it will surely be very useful for releasers, creating a yaml file next to the project imho is not compliant with the "Arduino simplicity contract" which is why people (average users, and beginners) are coming and using it.

I must say that my proposals #1117 and #1517 are directed to the arduino-GUI users, so it can be easier for a anyone (beginners) to configure large libraries, such as those which do not provide user functions for setting up internals, thus heavily relying on cpp-defines to "configure" the code.
I know you clearly understand what I'm talking about but I'll give two examples that convinced me to try and tackle this issue one year ago: I'm particularly thinking of arduino-lmic's project config file or long-term issues like setting a bigger buffer size.

Now thanks to your review, I'm coming back again with a third attempt which is even simpler because it requires no changes from libraries which are already supporting overrides. It is an optional, generic, universal and (=> your favorite superlative <=) per-sketch configuration file (link is below).

[ubidefeo]

@d-a-v

While our RFC is focused on the CLI, our final goal is to expose GUI elements which will allow a user to set custom #define fields, which are themselves not exactly aligning to the "Arduino simplicity contract".
Everything we do on the CLI can and will eventually consumed by the IDE 2.0.

For a beginner, having to create a folder on the Sketch level, means finding the Sketch folder and creating folder libdefs containing LIBRARY_NAME_user_config.h which in our opinion is way more cumbersome than offering a GU in which to add a number of custom fields, which will be propagated as #define directives during the pre-compile steps.

Anyway if you feel your solution is the one we should go for, it need to be peer reviewed, in which case you can submit a full RFC to the RFC repo I posted above, you can find the rules of this RFC Repo here
https://github.com/arduino/tooling-rfcs

This is the process, but as I said we're getting there and we will support the option to properly define macros without having to ask developers of Libraries to do anything they aren't yet doing

[d-a-v]

For a beginner, having to create a folder on the Sketch level, means finding the Sketch folder and creating folder libdefs containing LIBRARY_NAME_user_config.h which in our opinion is way more cumbersome than offering a GU in which to add a number of custom fields, which will be propagated as #define directives during the pre-compile steps.

This is true and I have much simplified this in #1524 which requires no other change in the current tools. No need to ask or change anything from libraries, a regular per-project GUI-editable header file backward compatible with a number of libraries. Also compatible with Arduino 1.8.

I will close this PR and #1117 and see if #1524 finds your interest.