We have problems

[UncleRus]

The number of components is growing and will continue to grow. This causes a bunch of problems, for example:

• Long assembly within the CI
• Latest tests crash with "no disk space" message (see, for example, https://github.com/UncleRus/esp-idf-lib/actions/runs/5642795414/job/15283368570?pr=542)
• We can't even really add new targets (c5, h6, etc.) because the full matrix build within the CI will become even more complicated
• We (thank you, Tomoyuki-san! @trombik) had to introduce label-based assembly to make it faster and easier.
• We can't (at least, I don't know how) to upload our components to the registry, since the registries most often rely on simple logic: one repository = one component.

What I suggest:

• Create an organization on github
• Separate components into repositories within this organization
• Create a head repository like the current one, in which to include these individual repositories as submodules for convenience and ease of use by new users.
• Consider the possibility of CI of this kind: each component repository conducts its own CI build cycle at the moment when it is necessary for it, and the main one collects these results and generalizes.

The disadvantages of this approach are also obvious: if we change the CI procedure, we'll have to make these changes to all component repositories. (I may be wrong on this point...)

In addition, I want to suggest that we stop using Ruby in developer utilities and switch to Python (to be honest, I already wrote a metadata parser and documentation/README generation on it).

@trombik Looking forward to your opinion on this matter!

[trombik]

contacted espressif and asked the status of esp-idf component registry for third-parties. no reply. for CI, espressif's GitHub Action might be reusable. but needs more research on it.

[igrr]

Just a comment regarding this part:

We can't (at least, I don't know how) to upload our components to the registry, since the registries most often rely on simple logic: one repository = one component.

This shouldn't be a problem when uploading to components.espressif.com, we do support "monorepos" with multiple components. For example, https://github.com/espressif/idf-extra-components contains several components. We pass the list of all the components to the Github action here. One note, with such a repository layout we can't use git tags to version components, so the version has to be specified in idf_component.yml files

On the topic of CI, we have recently released idf-build-apps tool which can recursively search for ESP-IDF applications (examples, test apps, etc) and build them in a variety of configurations (different chip targets, different sdkconfigs). It also has some logic to only build the apps which depend on components that have changed in a branch/PR. This can allow generating a build matrix for each PR on the fly, creating only the necessary number of jobs, depending on the changes. Simple usage of idf-build-apps can be seen in this repository. The more advanced usage with component dependency tracking is so far only in ESP-IDF itself, but we can create a demo if you are interested in such a feature.

[UncleRus]

idf-build-apps tool

Cool tool :)! But there are a number of questions: is v4.x fully supported? What about esp8266? One of the important factors behind the popularity of our repository is the fact that we still support esp8266. Will there be any problems whith ccache?

The more advanced usage with component dependency tracking is so far only in ESP-IDF itself, but we can create a demo if you are interested in such a feature.

This would be extremely helpful!

[igrr]

IDF v4.x CMake projects are supported by this tool.

We haven't tried using it with the ESP8266_RTOS_SDK. I suppose it could work for CMake based projects, with a bit of extra work.

We have already removed GNU make support from this tool, but it might be possible to bring it back. I will need to check this.

Note that dependency tracking (building only the apps which depend on the modified components) is only possible with CMake based build system...

As for ccache, I don't expect any issues with it. We do all the builds in espressif/idf Docker images which have ccache enabled by default.

One of the important factors behind the popularity of our repository is the fact that we still support esp8266.

Just curious about this, do you have any usage stats (esp8266 vs. esp32*)?

[UncleRus]

I suppose it could work for CMake based projects, with a bit of extra work.

AFAIK, ESP8266_RTOS_SDK has (or had some time ago) issues with CMake based projects. So in our CI for esp8266 Makefiles are used.

Just curious about this, do you have any usage stats (esp8266 vs. esp32*)?

No, there are no specific statistics, but there is an observation: a noticeable increase in the number of repo stars began after the addition of support for esp8266. In addition, judging by sales on marketplaces, this chip is still actively used by DIY people, because it's cheap and great for a number of tasks.

[trombik]

idf.py in ESP8266 RTOS SDK needs love. the last noticeable commit was in 2019. i tried to make it work back then with no avail. A few commits per year in the repository. The code works as our code depends on a very few functions in the esp-idf. but when esp-idf changes APIs, especially I2C, it would stop working. i do understand espressif no longer actively support ESP8266, but we need a recent version of idf.py if we keep supporting ESP8266.

[igrr]

Just as a follow-up, we will track Make support in espressif/idf-build-apps#62.

[trombik]

my initial work to split components at: https://github.com/trombik/eil_helpers

please see SPLIT.md for how.

[trombik]

as for the organization name, no idea. esp-idf-lib? hope espressif will not sue us. by the way, UncleRus/esp-idf-lib will remain so that all issue and PR links on GitHub still works. git-filter-repo will rewrite #${N} with a correct URL.

[trombik]

Even if we try to retain backward-compatibility, some code will break.

Say, someone writes on a book, "clone the repository by git clone https:...". It will not work as all the components will have moved, and the components directory will be empty.

We will break backward-compatibility. Personally, I don't care because, if a user is so naive to believe that old information would work indefinitely, the user deserves a lesson. You should have checked how old the information is, read the latest documentation, and known how fragile your code becomes when it depends on external resources.

If a user is paranoid enough, his code will not break because the submodule must have been tagged with a specific commit, or hard-vendored into the code (no git submodule at all).

My proposal only requires "git submodule update --init --recursive", or switching to esp repository. Old code should work after that (with some warnings). We will include all the submodules in release archive, but, even with such an effort, it will break someone's code.

This is a good timing to make big changes. I'll wait for what you would like to do. Your repository, your rules.

[UncleRus]

I understand your concern about overly generic names, but still no, we won't be renaming components and header files. The current component names work great.
It is still not clear to me how contributors without access rights to the organization will create PRs with new components. They will not be able to create a new repository within our organization... This problem seems to me to be a blocking one.

[trombik]

It is still not clear to me how contributors without access rights to the organization will create PRs with new components.

tl;dr: technically, they don't have to. they can publish their works to ESP Component Registry on their own (when the registry is open to public). still, it's possible to accept contributions.

there are two problems in esp-idf development. one problem is, there is no single place to publish and reuse code. search engines help a bit, but human-filter is required. UncleRus/esp-idf-lib is a great place for that. the ESP Component Registry is trying to solve this problem. another is duplicated efforts; there are too many libraries by different people. when I teach students Arduino, it is very difficult to explain how to find a library for a device. typing bmp280, or ws2812, into Library Manager produces too many results. PlatformIO is worse; it includes multiple versions of same library by different people, and a library that the publisher does not actually maintain (as some developers do not want to support platformio, and refuse to add whatever file required by platformio, and users fork the repo, and register under their name just because they want to use it with platformio).

with ESP Component Registry, there is less reason to contribute code to UncleRus/esp-idf-lib. still, the latter problem remains. if users support the idea to consolidate efforts, they can contribute their works.

the workflow will be determined when the CI supports multi-repositories, and when I get used to the ESP Component Registry. Here is what I have in my mind at the moment:

1. developing new code (with idf component manager, classic git submodule, or whatever) in a repository of the author
2. requesting inclusion by making an Issue at UncleRus/esp-idf-lib.
3. the organization forks the repository
4. the organization modifies UncleRus/esp-idf-lib, adding new dependency on the fork
5. archiving the original repository by the author (optional)

[UncleRus]

1. I decided NOT to split the repository. I don't like the PR process for new components in a split repo at all, and besides, the espressif registry supports mono-repos.
2. Labeller-based workflow CI can be replaced with a simple utility that analyzes changes in the PR and goal support in the component metadata (see feature: New devtool, simpler metadata, simpler CI #546)