Espressif Managed Component wolfSSL 5.7.4 post-release update

[gojimmypi]

Description

This PR updates the Espressif examples and builds on the updates in #8076. Enclosed are changes essential for the wolfSSL Managed Component examples to work properly with the v5.7.4-stable release of wolfssl.

Primary changes are to CMakeLists.txt and sdkconfig.defaults. Only minor changes made to example code.

Note this PR is based on the bdd62314 release commit, and DOES NOT INCLUDE other wolfSSL post-release updates.

The result will be an as-published wolfSSL 5.7.4 with only the changes needed for Espressif example updates. See my pr-post-release-bdd62314-espressif branch.

In particular, this PR addresses the new esp-tls and Certificate Bundle Support features introduced in #7936 and the respective interaction with Managed Components.

As not all versions of the ESP-IDF support the new esp-tls integration with wolfSSL, all of the examples now have the settings disabled by default in the respective dkconfig.defaults files:

# CONFIG_ESP_TLS_USING_WOLFSSL=y
# CONFIG_TLS_STACK_WOLFSSL=y

See my ESP-IDF v5.3.1 work and the respective release for PlatformIO.

To use it with PlatformIO, adjust this line in the platformio.ini file:

platform_packages = framework-espidf @ https://github.com/gojimmypi/esp-idf/releases/download/v5.3.1.20240926c/esp-idf-v5.3.1.20240926c.zip

I have an example with my esp_http_client_example.

Other Changes

• There are some corrections to cmake add_compile_definitions syntax.

• Improved checks for Managed Component names (e.g. staging site user names & mywolfssl)

• Adds new IS_GIT_REPO check to avoid attempts to glean attributes from non-git directories, such as those in Managed Component examples.

• client-tls.c changed the NoPeerCheck != 0 to be information, not a warning.

• TLS_SMP_CLIENT_TASK_BYTES bumped up to 10K default, rather than 8K.

• sdk_init_meminfo(); is only disabled when available and DEBUG_WOLFSSL enabled.

• The server example now gates out this_heap and stack_start variables not used as appropriate.

• While testing and editing the cmake files, I also discovered instances of message("text") that were being sent to stderr.
  I replaced those with message(STATUS "text")

• Various spelling corrections

• Addresses zd# 18228

Testing

Much of the testing occurred with the staging instance under my user name.

At the moment, the staging site is giving me a "Bad Gateway" error.

See also the internal wolfSSL scripts. I've added additional diagnostics and automated testing.

Release

As of the date of this PR creation, the release wolfSSL Managed Component has NOT yet been published. I plan to publish preview releases soon.

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[dgarske]

Looks good otherwise.

[dgarske]

Seems like you may have introduced CRLF into this file. Please resolve.

[gojimmypi]

During PR creation, there was no preview warning that there would be a merge conflict. There were petty changes needed so I tried to resolve them with the offered "edit in web" feature. That seems to have pulled in ALL of the changes from master in a new commit, which was specifically what I did not want.

I'm not sure how to proceed without adding a new commit to fix the CRLF from Visual Studio. I could close this one and start over? If I squash, all the specific changes will be lost and new changes included. Only my branch would then reflect the actual published code.

[gojimmypi]

Seems like you may have introduced CRLF into this file. Please resolve.

ok, @dgarske I'm getting better at recovering from git tangle.

I've fixed up this PR to exclude the problematic .gitignore, and two other petty files with only spell checks that were causing a merge conflict.

See additional PR comments, below.

[gojimmypi]

The staging site is still giving me a Bad Gateway error:

https://components-staging.espressif.com/components/gojimmypi/mywolfssl/versions/5.7.2

I've published a 5.7.4-preview1a production release to:

https://components.espressif.com/components/wolfssl/wolfssl/versions/5.7.4-preview1a?language=en

I'll be doing some final checks and testing of those examples.

Awaiting this PR to be approved before publishing a final 5.7.4 Espressif Managed Component release.

[gojimmypi]

ugh. At least one of the examples is failing to compile despite numerous successful tests on the staging site.

Unable to test again on staging site today, perhaps I missed a change. This is what I see for the production preview:

WRK_IDF_PATH=/mnt/c/SysGCC/esp32/esp-idf/v5.2
cd $WRK_IDF_PATH
. ./export.sh
cd /mnt/c/test
idf.py create-project-from-example "wolfssl/wolfssl^5.7.4-preview1a:wolfssl_client"
cd wolfssl_client
idf.py menuconfig
idf.py -p /dev/ttyS19 -b 115200 build flash monitor

results in

wolfssl_client/main/include/client-tls.h:27:10: fatal error: wolfssl/wolfcrypt/settings.h: No such file or directory
   27 | #include <wolfssl/wolfcrypt/settings.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

@dgarske I've reverted to draft, investigating....

[gojimmypi]

ok, this does not seem to be a problem with with publish. It's not just a settings.h that cannot be found, the entire example managed_components directory is empty from the above example, with no wolfssl component.

The entire wolfSSL component source is however present in the dist\wolfssl_5.7.4-preview1a.tgz file that was assembled and uploaded:

This command should have populated the entire directory with both the example and the wolfssl component:

idf.py create-project-from-example "wolfssl/wolfssl^5.7.4-preview1a:wolfssl_client"

I suspect there's a component web site problem, particularly with the "gateway error" on the staging site. I'll reach out to Espressif for help. It may possibly be some other issue.

For reference, here's the idf_component.yml, the same as staging site, other than version ID string:

description: wolfSSL Embedded SSL/TLS Library
discussion: https://www.wolfssl.com/forums
documentation: https://www.wolfssl.com/documentation/manuals/wolfssl/index.html
files:
  exclude:
  - '**/*'
  - INSTALL.md
  - '*.py'
  - '**/*.list'
  - '**/.vs/**/*'
  - '**/build/**/*'
  - '**/dist/**/*'
  - '**/dist.bak/**/*'
  - '**/out/**/*'
  - '*.sh'
  - '*.bak'
  - sdkconfig
  - README_DIST.md
  - README_REGISTRY_PREPEND.md
  - README_STAGING_PREPEND.md
  - README_STAGING.md
  - ./src/sp_arm64.c
  - ./src/sp_armthumb.c
  - ./src/sp_x86_64.c
  - ./src/sp_arm32.c
  - ./src/sp_c32.c
  - ./src/sp_cortexm.c
  include:
  - ./README.md
  - ./LICENSE.txt
  - ./idf_component.yml
  - ./CMakeLists.txt
  - ./Kconfig
  - ./install_log.txt
  - ./src/**/*
  - ./include/**/*
  - ./wolfssl/**/*
  - ./wolfcrypt/**/*
  - ./examples/**/*
issues: https://github.com/wolfSSL/wolfssl/issues
repository: https://github.com/wolfSSL/wolfssl.git
url: https://www.wolfssl.com
version: 5.7.4-Preview1a

found here:

I will leave this PR in draft mode until fully resolved.

[gojimmypi]

Additional information:

idf.py create-project-from-example "wolfssl/wolfssl^5.7.4-preview1a:wolfssl_client"
cd wolfssl_client

# NO managed_component directory exists

idf.py menuconfig

# managed_component directory exists

idf.py build

# managed_component directory is removed early in the build process

And sure enough, from the build log - note last lines:

Executing action: all (aliases: build)
Running ninja in directory /mnt/c/test/libtest/wolfssl_client/build
Executing "ninja all"...
[0/1] Re-running CMake...
-- Begin project wolfssl_client
-- Detected UNIX
-- Detected WSL
-- Detected Linux
-- Found PROTOCOL_EXAMPLES_DIR=/mnt/c/SysGCC/esp32/esp-idf/v5.2/examples/common_components/protocol_examples_common
-- USERNAME =
-- THIS_USER = gojimmypi
-- Checking for wolfSSL as Managed Component or not... /mnt/c/test/libtest/wolfssl_client
-- No conflicting wolfSSL components found as a Managed Component.
-- EXTRA_COMPONENT_DIRS WOLFSSL_PATH: /mnt/c/test/libtest/wolfssl_client/managed_components/wolfssl__wolfssl
-- git rev-parse returned 'fatal: not a git repository (or any parent up to mount point /mnt)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).'
-- Could not use 'git describe' to determine PROJECT_VER.
-- Building ESP-IDF components for target esp32
Manifest files have changed, solving dependencies.
Using component placed at /mnt/c/test/libtest/wolfssl_client/managed_components/wolfssl__wolfssl for dependency wolfssl/wolfssl(^5.7.4-Preview1a), specified in /mnt/c/test/libtest/wolfssl_client/main/idf_component.yml
..Updating lock file at /mnt/c/test/libtest/wolfssl_client/dependencies.lock
Deleting 1 unused components
 wolfssl__wolfssl

None of the published wolfSSL cmake ever files remove the wolfSSL component.

This is what is in /mnt/c/test/libtest/wolfssl_client/main/idf_component.yml:

dependencies:
  idf:
    version: '>=4.1.0'
  wolfssl/wolfssl: ^5.7.4-Preview1a

This did not happen with the staging-components. I believe the prior production work fine. However, this does not seem to be limited to the new preview. The latest 5.7.2 Managed Component is exhibiting this behavior as well. I suspect it is in the ESP-IDF, and not the published component. Investigation continues....

[gojimmypi]

The latest force push commit replaces the list(APPEND EXTRA_COMPONENT_DIRS ${WOLFSSL_PATH}) with the PRIV_REQUIRES in the CMakeLists.txt for main with an explicit reference to wolfSSL like this:

idf_component_register(SRCS main.c
                       INCLUDE_DIRS "."
                            "./include"
                       PRIV_REQUIRES driver "${MAIN_WOLFSSL_COMPONENT_NAME}")

In doing so, additional explicit references were also needed for some examples (e.g. protocol_examples_common, driver, and nvs_flash).

Note that a variable MAIN_WOLFSSL_COMPONENT_NAME is needed as the library name varies: it is mywolfssl on the staging site to clearly distinguish from the production release name of wolfssl.

This appears to resolve the undesired Deleting 1 unused components wolfssl__wolfssl only after first running idf.py menuconfig and then idf.py build.

I did try actually using other wolfSSL code directly in main.c (printing the macro value from version.h) so that wolfssl unused problem would be resolved... but no luck. Perhaps it would need to be actual binary code such as the sdk_init_meminfo() that was also gated out in this update to only be used with DEBUG_WOLFSSL. Just as well, as that would have been an unintuitive, hacky solution anyhow.

The observed behavior with the staging Managed Components was different from the release site. Code that worked perfectly on the staging site still suffered from the Deleting unused components problem on the production release site. Reason unknown. The PRIV_REQUIRES route seems to be considerably more robust that the EXTRA_COMPONENT_DIRS .

ESP-IDF v5.2 was used for all development testing.

Reminder that the exact same example code is used for all types of components: local install, ESP-IDF, and local Managed Components.

Note that additional changes have been made to the publishing scripts. One problematic issue encountered with my first release site preview is that I used mixed case in the 5.7.4-preview1a. (5.7.4-Preview1a). Using mixed case is now prohibited in the publishing script, with a hard failure error.

The latest preview release is currently 5.7.4-preview1e.

Additional testing is still underway, but things are finally looking promising.

[kumekay]

Hi @gojimmypi I tried to reproduce the issue with ESP-IDF v5.2.3 and the component manager v2.1.0, but didn't manage to reproduce. Could you please help with the steps?

$ idf.py create-project-from-example "wolfssl/wolfssl^5.7.4-preview1a:wolfssl_client"                                                                                       
$ idf.py menuconfig
$ ls managed_components/wolfssl__wolfssl                                                                                                                                        
CMakeLists.txt    Kconfig           LICENSE.txt       README.md         examples          idf_component.yml include           src               wolfcrypt         wolfssl
$ idf.py build
...
Successfully created esp32 image.
Generated /Users/ku/t/wolfssl_client/build/wolfssl_client.bin
[1046/1046] cd /Users/ku/t/wolfssl_client/build/esp-idf/esptool_py && /Users/ku/.espressif/p...uild/partition_table/partition-table.bin /Users/ku/t/wolfssl_client/build/wolfssl_client.bin
wolfssl_client.bin binary size 0x10b6e0 bytes. Smallest app partition is 0x177000 bytes. 0x6b920 bytes (29%) free.

Project build complete. To flash, run:
...
$ ls managed_components/wolfssl__wolfssl                                                                                                                                        [11:56:26]
CMakeLists.txt    Kconfig           LICENSE.txt       README.md         examples          idf_component.yml include           src               wolfcrypt         wolfssl

Using mixed case is now prohibited in the publishing script, with a hard failure error.

What kind of issue is caused by mixed casing? Can we do something on our side to avoid it?

[gojimmypi]

Hello @kumekay and thanks for taking a look at this.

I'm using ESP-IDF v5.2 release here, not the latest v5.2.3.

$ idf.py --version
ESP-IDF v5.2-dev-3903-g66992aca7a-dirty

The only changes are some backup files:

I also tried ESP-IDF v5.3 just now, and observed the same results.

With your create-project-from-example syntax, I think you may be grabbing the latest preview version of the component, which I finally (think) I have working.

To test my initial problematic preview version that deletes unused code, please use this syntax (note the = not ^ for version specification):

idf.py create-project-from-example "wolfssl/wolfssl=5.7.4-preview1a:wolfssl_client"
cd wolfssl_client
idf.py menuconfig
idf.py build

No changes made in idf.py menuconfig. Just run it and then exit.

I see this output:

$ idf.py build
Executing action: all (aliases: build)
Running ninja in directory /mnt/c/workspace/wolfssl_scripts-gojimmypi/espressif/test/wolfssl_client/build
Executing "ninja all"...
[0/1] Re-running CMake...
-- Begin project wolfssl_client
-- Detected UNIX
-- Detected WSL
-- Detected Linux
-- Found PROTOCOL_EXAMPLES_DIR=/mnt/c/SysGCC/esp32/esp-idf/v5.2/examples/common_components/protocol_examples_common
-- USERNAME =
-- THIS_USER = gojimmypi
-- Checking for wolfSSL as Managed Component or not... /mnt/c/workspace/wolfssl_scripts-gojimmypi/espressif/test/wolfssl_client
-- No conflicting wolfSSL components found as a Managed Component.
-- EXTRA_COMPONENT_DIRS WOLFSSL_PATH: /mnt/c/workspace/wolfssl_scripts-gojimmypi/espressif/test/wolfssl_client/managed_components/wolfssl__wolfssl
-- Building ESP-IDF components for target esp32
Manifest files have changed, solving dependencies.
Using component placed at /mnt/c/workspace/wolfssl_scripts-gojimmypi/espressif/test/wolfssl_client/managed_components/wolfssl__wolfssl for dependency wolfssl/wolfssl(^5.7.4-Preview1a), specified in /mnt/c/workspace/wolfssl_scripts-gojimmypi/espressif/test/wolfssl_client/main/idf_component.yml
..Updating lock file at /mnt/c/workspace/wolfssl_scripts-gojimmypi/espressif/test/wolfssl_client/dependencies.lock
Deleting 1 unused components
 wolfssl__wolfssl
Processing 1 dependencies:

Note that with automated testing without the interactive menuconfig, the examples compile, build, flash and run successfully.

I searched through my ESP32 directory (used in WSL, but initially installed in Windows via ViasualGDB), but could find no reference to the code that is printing the "Deleting 1 unused component" message. Is is possibly that the idf.py calls code outside of the IDF_PATH ?

 echo $IDF_PATH
/mnt/c/SysGCC/esp32/esp-idf/v5.2

Regarding the case, note the contents of ./wolfssl_client/main/idf_component.yml:

dependencies:
  idf:
    version: '>=4.1.0'
  wolfssl/wolfssl: ^5.7.4-Preview1a

The published Preview1a has a lower case p, but some enclosed references still have upper case.

What kind of issue is caused by mixed casing?

I did observe some inconsistencies, but I was primarily looking at the cause of component deletion.

Can we do something on our side to avoid it?

Perhaps either enforce a case, or don't change them? My Preview1a ended up as preview1a in some places.

Cheers & thank you again for your help.