Optimize build. Only used libraries downloaded & compiled.

[rhapsodyv]

Description

With more time, and digging deep in the PlatformIO code, I could create a script that enable/install the dependencies according with the features enabled in Configuration.h.

To do it, the script change the current configuration (platformio.ini contents) loaded by the PlatformIO. It isn't a "standard" way, but works.

The script:

• Uses the current compiler to check the enabled features in Configuration.h
• Must keep a list of the features and your dependencies
• Must run in every env, first, with a "pre:". (well, at least in env that work with features that have dependencies)
• Works with private libs in Marlin/lib dir too.
• Export a method that all other scripts can use to check if a Marlin feature is enabled: env.MarlinFeatureIsEnabled(feature)
• It may work with any feature that have any dependency or extra_script
• It may work to add/remove src_filter too. Important: its doesn't support parenting. If a parent folder was removed from compile, a child folder cannot be added.
• It respects env lib_ignore, so it dont add a lib if its ignored
• If the env already have a lib with the same name, the script dont append it. So, the env can have their own version of the lib.
• Its even better: it may be able to change almost any platformio.ini config according with features enabled in Configuration.h, at runtime.

Output while compiling:

Feature enabled: TFT_LVGL_UI
Adding lib_deps for TFT_LVGL_UI...

Benefits

1. We won't need anymore create an env for each feature in a board
2. Dependencies only will be installed when needed
3. Dependencies only will be compiled when needed
4. Save times and downloads

Related Issues

#18500

Tasks

• Move the features configs to platformio.ini
• Load the build flags for each platform automatically (created buildroot/share/PlatformIO/scripts/common-features-dependencies.h to get features without any HAL)
• Make a dependency overwritable by the env lib_deps (check if the env already have a dep with the same name, so
  ignore it)
• Respect lib_ignore configs
• Do we need that all lib_ignore, now that deps are add only when used?
• src_filter change support
• comma support
• remove env for specific features

[rhapsodyv]

@thinkyhead Look the last commit. Now it can check even for more advanced features, like HAS_TRINAMIC, to install tmc lib only when need.

But I don’t know if we need go too deep.

Anyway, it’s a good prof of concept.

[rhapsodyv]

CI is falling, because Marlin/src/inc/MarlinConfigPre.h includes Marlin/src/HAL/platforms.h, that needs to know in advance which platform we are compiling.
To discover that, I did the script process all build flags and pass to g++ all the defines I found in the current build process.

But, in some boards, the definition of the platform isn't in a build flag (like SAMD51_grandcentral_m4). To solve the failing build, we need just to add the platform definition for each "parent" env. It's easy.

Another approach, is to have all features definition in more "simple" file, that dont depends on HAL to generate the marlin features definitions.

[rhapsodyv]

Well, it was just one board missing the platform definition in the build flags ☺️ 🤣

[thinkyhead]

None of the "board" defines are present when the compiler is invoked from the python script, and this is why the platforms.h file needs extra help. I wonder if there's some way to get the flags from buildroot/share/PlatformIO/boards/MyBoard.json and/or the MyBoard.json files included in the platform and add them to the command-line.

[thinkyhead]

Here's a typical command-line for an AVR build, with our missing flags supplied by PlatformIO…

avr-g++ -o .pio/build/mega2560/src/src/lcd/extui/lib/mks_ui/draw_printing.cpp.o -c -Wno-register -fno-exceptions -fno-threadsafe-statics -fpermissive -std=gnu++11 -fmax-errors=5 -g -fmerge-all-constants -Os -Wall -ffunction-sections -fdata-sections -flto -mmcu=atmega2560 -DPLATFORMIO=40304 -DARDUINO_AVR_MEGA2560 -D__MARLIN_FIRMWARE__ -D__AVR__ -DF_CPU=16000000L -DARDUINO_ARCH_AVR -DARDUINO=10808 -IMarlin -I$HOME/.platformio/packages/framework-arduino-avr/libraries/EEPROM/src -I.pio/libdeps/mega2560/U8glib-HAL_ID1932/src -I$HOME/.platformio/packages/framework-arduino-avr/libraries/Wire/src -I.pio/libdeps/mega2560/TMCStepper/src -I$HOME/.platformio/packages/framework-arduino-avr/libraries/SoftwareSerial/src -I$HOME/.platformio/packages/framework-arduino-avr/libraries/SPI/src -I$HOME/.platformio/packages/framework-arduino-avr/cores/arduino -I$HOME/.platformio/packages/framework-arduino-avr/variants/mega Marlin/src/lcd/extui/lib/mks_ui/draw_printing.cpp

[rhapsodyv]

We can move the features deps to platformio.ini in a features section. What do you think?

#
# Features and Dependecies
#
[features]
TFT_LVGL_UI=MKS-LittlevGL=https://github.com/makerbase-mks/MKS-LittlevGL/archive/master.zip
HAS_TRINAMIC=TMCStepper@~0.7.1
SR_LCD_2W_NL=SailfishLCD=https://github.com/mikeshub/SailfishLCD/archive/master.zip
SR_LCD_3W_NL=SailfishLCD=https://github.com/mikeshub/SailfishLCD/archive/master.zip
DIGIPOT_MCP4018=SlowSoftI2CMaster
DIGIPOT_MCP4451=SlowSoftI2CMaster
HAS_TMC26X=TMC26XStepper=https://github.com/trinamic/TMC26XStepper/archive/master.zip
HAS_L64XX=Arduino-L6470@0.8.0
NEOPIXEL_LED=Adafruit NeoPixel@1.5.0
MAX6675_IS_MAX31865=Adafruit MAX31865 library@~1.1.0
HAS_GRAPHICAL_LCD=U8glib-HAL@0.4.1
HAS_CHARACTER_LCD=LiquidCrystal@1.5.0
  LiquidTWI2@1.2.7

[thinkyhead]

That could work. Note that under some environments, older/different versions of libraries are used. So we need to be able to specify those differences by overriding the values in the [features] section within individual environments.

[rhapsodyv]

That could work.

I will move to [features].

Note that under some environments, older/different versions of libraries are used. So we need to be able to specify those differences by overriding the values in the [features] section within individual environments.

Right now I just prepend the lib_deps to the current env lib_deps. But we can check: if the env already have a lib_dep with the same name, we just ignore.

[rhapsodyv]

None of the "board" defines are present when the compiler is invoked from the python script, and this is why the platforms.h file needs extra help. I wonder if there's some way to get the flags from buildroot/share/PlatformIO/boards/MyBoard.json and/or the MyBoard.json files included in the platform and add them to the command-line.

I loaded the boards.json flags. But, some platform, the arch definition is in a py script, like:
~/.platformio/packages/framework-arduinoespressif32/tools/platformio-build.py
It's that py that insert the ARDUINO_ARCH_ESP32 definition.

I will check if we can run our script after the platform platformio-build.py

[rhapsodyv]

pre: script runs before the current platform script. post: runs after, but after the library dependency finder too. So, if we use 'pre', we dont have all platform flags. If you run 'post', is late to add dependencies to the build.

PlatformIO code:

# HERE our script run
for item in env.GetExtraScripts("pre"):
    env.SConscript(item, exports="env")

# HERE the build platform script run, but it run all the lib deps too!
env.SConscript("$BUILD_SCRIPT")

...

# here is the post script
for item in env.GetExtraScripts("post"):
    env.SConscript(item, exports="env")

Do we really need process everything, including HAL, to know the enabled features?

If we create a simple ifdef to ignore HAL in the MarlinConfigPre, it may work... Or, we can create a header just with the propose of discover the enabled features, that dont include any HAL.
Or we will need to have the arch definition in the build_flags of each env. It’s not a big deal, considering that it will never change.

[ellensp]

That is wrong version of Adafruit NeoPixel library for STM32F103RET6
change
NEOPIXEL_LED = Adafruit NeoPixel@1.5.0
to
NEOPIXEL_LED = Adafruit NeoPixel@1.2.4

but I dont know how to do this automatically in the new system.
@rhapsodyv can this be done?

[rhapsodyv]

That is wrong version of Adafruit NeoPixel library for STM32F103RET6
change
NEOPIXEL_LED = Adafruit NeoPixel@1.5.0
to
NEOPIXEL_LED = Adafruit NeoPixel@1.2.4

but I dont know how to do this automatically in the new system.
@rhapsodyv can this be done?

The env can overwrite any lib it needs to a specific version.

[env:my_env]
lib_deps = FOO@1.2.3

But his problem is other. There's no NEOPIXEL_LED enabled in that config. I'm almost sure the script ins't running in his env... If the script isn't running, and they have forgotten unused libs in his .pio/lib_deps, the compiler will try use it...
If our script run, it will lib_ignore everything not used, so it will not break the compilation when there're unused libs in .pio/lib_deps.

@mindspawn could you confirm you env isn't (or it is...) running the commons scripts?

[rhapsodyv]

@ivankravets Another important thing: it would be very useful if we can have a script that run always, regardless of what is in the current env. Right now, to run our script, we had to edit all envs to call it.

[mindspawn]

@rhapsodyv your right deleting my workspace, cloning the repo and building again seemed to have fixed the build issue. Thanks a lot :)

[pfeerick]

it would be very useful if we can have a script that run always, regardless of what is in the current env. Right now, to run our script, we had to edit all envs to call it.

You should be able to use the common env for that? i.e. https://docs.platformio.org/en/latest/projectconf/section_env.html#common-env

Or are you referring to the common env extra_scripts being overridden by the local env extra_script setting (as IIRC that is what happens, rather than it being appended to, meaning you need to add something like${common.extra_scripts} to the extra_scripts of each env)?

[ivankravets]

Oh, very long thread :) Thanks to everyone who joined this thread.

1. If there is a bug linked with PlatformIO Core, please create a simple project to reproduce it and file an issue here https://github.com/platformio/platformio-core/issues

2. We have deployed the new registry (currently, only backend). The new registry has strict rules and requires a library to be compliant with Semantic Versioning.

3. Developers will be able in PlatformIO Core 4.4 to declare dependencies based on their owners (avoid IDs, conflicts, etc).

We highly recommend using SemVer for dependencies and avoid using external archives, tarballs, VCS repositories. If you need a library and it has not been published in the registry, you can publish it into the own scope https://docs.platformio.org/en/latest/librarymanager/creating.html

The only PIO Core 4.4 is released, we show instruction on how to use libraries by some owner. The syntax will be the next:

lib_deps = 
   ownername/packagename @ SemVer-requirements

For example,

lib_deps = 
   armmbed/mbedtls@^2

Means, use the latest version of mbedtls in range 2.0.0-2.Y.Z.

[rhapsodyv]

Or are you referring to the common env extra_scripts being overridden by the local env extra_script setting (as IIRC that is what happens, rather than it being appended to, meaning you need to add something like${common.extra_scripts} to the extra_scripts of each env)?

Yes, the common env extra_scripts are being overridden by the local env extra_script. We add ${common.extra_scripts} to all envs... But, some users had their platformio.ini edited and getting it messed up while merging.

It would be better if we can have a way to define a global pre script, the run always.

[rhapsodyv]

Oh, very long thread :) Thanks to everyone who joined this thread.

1. If there is a bug linked with PlatformIO Core, please create a simple project to reproduce it and file an issue here https://github.com/platformio/platformio-core/issues
2. We have deployed the new registry (currently, only backend). The new registry has strict rules and requires a library to be compliant with Semantic Versioning.
3. Developers will be able in PlatformIO Core 4.4 to declare dependencies based on their owners (avoid IDs, conflicts, etc).

It would be great if we have a 4 item:

4. Developers being able to declare dependencies based on a header file with all defined marcros/configurations the user enabled :-)

Thanks @ivankravets . I will create an issue to discuss about it.

[thinkyhead]

Thanks for moving discussion to the PIO forums. There's no bug to speak of; we're just trying to do fancy things with "Advanced Scripting" to optimize our build times. We'll be using the power of src_filter plus the added [features] block to get even more granular.

I will see about adding Adafruit Neopixel (and SPI) back to the lib_ignore of platforms that the library is incompatible with, but which include another library that declares it as a "dependency."

[rhapsodyv]

I will see about adding Adafruit Neopixel (and SPI) back to the lib_ignore of platforms that the library is incompatible with, but which include another library that declares it as a "dependency."

We only got this kind of compiling error when user had a merge issue in platformio.ini... Adding a lib_ignore, may not solve it too.... But anyway... adding it won't do any bad either.

Right now, every known lib that isn't used by the env is automatically and explicity ignored, adding it to lib_ignore.

[thinkyhead]

Right now, every known lib that isn't used by the env is automatically and explicity ignored, adding it to lib_ignore.

Interesting. When building for ESP32 with WIFISUPPORT you'll notice that ESPAsyncTCP is not named as a direct lib_deps entry. However, one of the other items in the lib_deps list does list it as a dependency, so it gets downloaded and compiled. For this reason I did need to add it to the lib_ignore list for WIFISUPPORT.

If the script is supposed to be stripping out all the lib_deps that we have not explicitly named, it is somehow missing ESPAsyncTCP?

[rhapsodyv]

If the script is supposed to be stripping out all the lib_deps that we have not explicitly named, it is somehow missing ESPAsyncTCP?

Take a look at the output... it must have outputting something like this: Ignoring libs: ...

Adding src_filter for TFT_LVGL_UI... 
Ignoring libs:  ['TMCStepper', 'Adafruit MAX31865 library', 'U8glib-HAL', 'LiquidCrystal', 'TMC26XStepper', 'SlowSoftI2CMaster', 'SailfishLCD', 'arduinoWebSockets', 'ESP32SSDP', 'ESP Async WebServer', 'AsyncTCP', 'Adafruit NeoPixel', 'Arduino-L6470', 'LiquidTWI2', 'ESP3DLib']

[rhapsodyv]

ESPAsyncTCP

Now I saw what is happening. The script dont know how to handle it: lib_ignore=ESPAsyncTCP...

HAS_LCD_MENU        = src_filter=+<src/lcd/menu>
(ESP32_)?WIFISUPPORT = AsyncTCP, ESP Async WebServer
  ESP3DLib=https://github.com/luc-github/ESP3DLib.git
  arduinoWebSockets=https://github.com/Links2004/arduinoWebSockets.git
  ESP32SSDP=https://github.com/luc-github/ESP32SSDP.git
  lib_ignore=ESPAsyncTCP

You added the parsing in the load_config function:

			elif name == 'lib_ignore':
				FEATURE_DEPENDENCIES[ukey]['lib_ignore'] = rest

But, it's missing the magic code in the install_features_dependencies() function :-)
so, right now, this lib_ignore=ESPAsyncTCP in features have no effect :-)

[rhapsodyv]

But, it's missing the magic code in the install_features_dependencies() function :-)
so, right now, this lib_ignore=ESPAsyncTCP in features have no effect :-)

Here is the missing code in install_features_dependencies():

		if 'lib_ignore' in FEATURE_DEPENDENCIES[feature]:
			print("Ignoring libs for %s... " % feature)
			lib_ignore = env.GetProjectOption("lib_ignore") + [FEATURE_DEPENDENCIES[feature]['lib_ignore']]
			proj = env.GetProjectConfig()
			proj.set("env:" + env["PIOENV"], "lib_ignore", lib_ignore)

[thinkyhead]

Added some extra logging, and it does seem to process the line…

Log Output

Feature Dep Line:  MKS-LittlevGL=https://github.com/makerbase-mks/MKS-LittlevGL/archive/master.zip
Feature Dep Line:  src_filter=+<src/lcd/extui/lib/mks_ui>
Feature Filter:  +<src/lcd/extui/lib/mks_ui>
Feature Dep Line:  TMCStepper@~0.7.1
Feature Dep Line:  SailfishLCD=https://github.com/mikeshub/SailfishLCD/archive/master.zip
Feature Dep Line:  SlowSoftI2CMaster
Feature Dep Line:  TMC26XStepper=https://github.com/trinamic/TMC26XStepper/archive/master.zip
Feature Dep Line:  Arduino-L6470@0.8.0
Feature Dep Line:  Adafruit NeoPixel@1.5.0
Feature Dep Line:  Adafruit MAX31865 library@~1.1.0
Feature Dep Line:  U8glib-HAL@0.4.1
Feature Dep Line:  src_filter=+<src/lcd/dogm>
Feature Filter:  +<src/lcd/dogm>
Feature Dep Line:  LiquidCrystal@1.5.0
Feature Dep Line:  LiquidTWI2@1.2.7
Feature Dep Line:  src_filter=+<src/lcd/extui/lib/ftdi_eve_touch_ui>
Feature Filter:  +<src/lcd/extui/lib/ftdi_eve_touch_ui>
Feature Dep Line:  src_filter=+<src/lcd/extui/lib/dgus>
Feature Filter:  +<src/lcd/extui/lib/dgus>
Feature Dep Line:  src_filter=+<src/lcd/dwin>
Feature Filter:  +<src/lcd/dwin>
Feature Dep Line:  src_filter=+<src/lcd/menu>
Feature Filter:  +<src/lcd/menu>
Feature Dep Line:  AsyncTCP
Feature Dep Line:  ESP Async WebServer
Feature Dep Line:  ESP3DLib=https://github.com/luc-github/ESP3DLib.git
Feature Dep Line:  arduinoWebSockets=https://github.com/Links2004/arduinoWebSockets.git
Feature Dep Line:  ESP32SSDP=https://github.com/luc-github/ESP32SSDP.git
Feature Dep Line:  lib_ignore=ESPAsyncTCP
Feature Ignore:  ESPAsyncTCP
CC -D__MARLIN_FIRMWARE__ -DCORE_DEBUG_LEVEL=0 -w -dM -E -x c++ buildroot/share/PlatformIO/scripts/common-features-dependencies.h
Adding lib_deps for (ESP32_)?WIFISUPPORT... 
Ignoring libs:  ['Adafruit MAX31865 library', 'LiquidTWI2', 'LiquidCrystal', 'SlowSoftI2CMaster', 'TMCStepper', 'U8glib-HAL', 'Arduino-L6470', 'TMC26XStepper', 'MKS-LittlevGL', 'SailfishLCD', 'Adafruit NeoPixel']

Dependency Graph

|-- <AsyncTCP> 1.1.1
|-- <ESP Async WebServer> 1.2.3
|   |-- <AsyncTCP> 1.1.1
|   |-- <FS> 1.0
|   |-- <WiFi> 1.0
|-- <ESP3DLib> 1.0.0 #fd0f7b6
|   |-- <DNSServer> 1.1.0
|   |   |-- <WiFi> 1.0
|   |-- <ESP32SSPD> 1.1.0 #36420ca
|   |   |-- <WiFi> 1.0
|   |-- <ESPmDNS> 1.0
|   |   |-- <WiFi> 1.0
|   |-- <FS> 1.0
|   |-- <Preferences> 1.0
|   |-- <SPIFFS> 1.0
|   |   |-- <FS> 1.0
|   |-- <Update> 1.0
|   |-- <WebServer> 1.0
|   |   |-- <WiFi> 1.0
|   |   |-- <FS> 1.0
|   |-- <WebSockets> 2.2.1 #19c39d5
|   |   |-- <WiFi> 1.0
|   |   |-- <SPI> 1.0
|   |   |-- <WiFiClientSecure> 1.0
|   |   |   |-- <WiFi> 1.0
|   |-- <WiFi> 1.0
|   |-- <ArduinoOTA> 1.0
|   |   |-- <Update> 1.0
|   |   |-- <WiFi> 1.0
|   |   |-- <ESPmDNS> 1.0
|   |   |   |-- <WiFi> 1.0
|-- <WebSockets> 2.2.1 #19c39d5
|   |-- <WiFi> 1.0
|   |-- <SPI> 1.0
|   |-- <WiFiClientSecure> 1.0
|   |   |-- <WiFi> 1.0
|-- <ESP32SSPD> 1.1.0 #36420ca
|   |-- <WiFi> 1.0
|-- <SPI> 1.0
|-- <EEPROM> 1.0.3
|-- <ArduinoOTA> 1.0
|   |-- <Update> 1.0
|   |-- <WiFi> 1.0
|   |-- <ESPmDNS> 1.0
|   |   |-- <WiFi> 1.0
|-- <ESPmDNS> 1.0
|   |-- <WiFi> 1.0
|-- <WiFi> 1.0
|-- <FS> 1.0
|-- <SPIFFS> 1.0
|   |-- <FS> 1.0
|-- <Wire> 1.0.1
|-- <ESP32 BLE Arduino> 1.0.1

[rhapsodyv]

Added some extra logging, and it does seem to process the line…

It process, put that in FEATURE_DEPENDENCIES[feature]['lib_ignore'], but it isn't processed after by install_features_dependencies(). I just sent the missing code.

If there's a folder named ESPAsyncTCP in your .pio/lib_deps, it could enter in your dependency graph and get compiled, if the LDF "thinks" you referenced it.... So its the reason for the explicit ignore for all known libs.

[thinkyhead]

What are the different "lib_ldf_mode" values in PIO, and is one of them more suitable for our usage?

[thinkyhead]

Looks like off may be better for our purposes. It considers the manifests but doesn't try to be clever and follow #includes.

[thinkyhead]

Yeah, so lib_ldf_mode=off is probably great for tiny projects.