compiling unit tests uses arduino platform specific code

[daniel-mohr]

Issue Summary

During compiling the unit tests, arduino_ci compiles all *.c* files of the library.

But it uses not the platform specific dependencies. Since the unit tests should run on system running arduino_ci this sounds clear. But its not; the source code was written for the specified arduino system and at most time can not be build on a system running arduino_ci.

Arduino or Unit Test Code, Illustrating the Problem

As example I used Udp.h for the platform zero in the source code of the library. So the unit tests cannot be compiled.

Since I'm writing the code I'm free to split it. Some parts could be tested -- e. g. macros running on linux systems,
which could be stored in separated header files.

So, I see a few solutions:

1. Add to arduino_ci the possibility to skip some files during building unit tests.
2. Add to arduino_ci the possibility to add custom code to mock some libraries or at least to mock some header files of used libraries.
3. Add documentation how to add custom code to mock something.

I'm not sure if adding mocked header files to /var/lib/gems/*/gems/arduino_ci-*/cpp/arduino/ or /var/lib/gems/*/bundler/gems/arduino_ci-*/cpp/arduino/ is the right way. But it works. ?

Arduino Architecture(s) Affected

all

[ianfixes]

Hi Daniel, thanks for reporting this.

There are a few possible ways to describe the issue that you're experiencing. One explanation is that (as you noticed) the UDP library isn't mocked out by Arduino-CI -- it's not part of the builtin libraries, so I wasn't even planning on mocking it for this milestone.

I think the best way to fix your issue is related to #159, where you would use an ifdef to determine whether a unit test or a "real" build was being compiled.

As for where to put the headers, at some point the Arduino community would need to have a discussion as to if/how it wants to organize compilation mocks for various libraries. It would be my suggestion that each library includes its own set of mocks for testing purposes, but I'm not sure how to start that discussion with the core team.

Your comment hints at some platform-specific issues regarding the inclusion of files. Do you think this could be addressed by the ifdef method above, or are you asking for something to be added to the config YAML file to allow for this?

[daniel-mohr]

Let be explain my ideas about the 3 possible solutions a little bit (as example maybe my project: https://github.com/ug-cp/precise_sntp)

1. Add to arduino_ci the possibility to skip some files during building unit tests.

On default Arduino-CI uses all files in src/ for compilation. If there is no platform specific dependency this is fine.

But unfortunately not in my case. But during writing the code, I can split my code in some (header) files in src/. Some of these files have platform specific dependencies and other do not have platform specific dependencies. The files without platform specific dependencies can be used in unit tests and can be tested with Arduino-CI.

But since Arduino-CI tries to compile all files I get some failures.

If there would be a way to explain Arduino-CI to ignore src/precise_sntp.cpp (or maybe more files) during unit tests everything would be fine.

One way could be a command line parameter like --skip-file src/precise_sntp.cpp .

Or maybe a list skip_during_unit_test inside the config YAML file to list all files to be ignored during unit tests.

Using ifdef -- as you advised -- could be a solution. But in this way I have to handle this in the code and (maybe) this reduces the readability.

2. Add to arduino_ci the possibility to add custom code to mock some libraries or at least to mock some header files of used libraries.

Here my idea was to explain Arduino-CI to add some custom code to mock some libraries during unit testing. For example a command line parameter of Arduino-CI like -Iextras/ could lead to adding -Iextras/ to the call of gcc during unit testing.

3. Add documentation how to add custom code to mock something.

The idea here was to add a section to REFERENCE.md to explain how to mock a library.

I'm using at the moment cp -p extras/udp_mock.h /var/lib/gems/*/gems/arduino_ci-*/cpp/arduino/Udp.h during running the pipeline to mock the library Udp.h. But I'm not sure if this is the right way. I only know that it works. ;-)

[jgfoster]

@daniel-mohr, Your need to mock external libraries is the problem I raised in #150. As you can see from the list of repositories, I've created a dozen or so forks of the external libraries we use in our project, including Ethernet. In your instruction you say to include the Ethernet library. I suggest that you try including our Ethernet library and see if that gives any compile errors. If you would like to use it and find problems you could submit issues and we could collaborate on solutions.

[daniel-mohr]

@jgfoster Sounds interesting! But your link 'our Ethernet library' is unfortunately the same as your link 'Ethernet'. ?

I will have a look the next days.

[jgfoster]

Yes, they were intended to be the same. I've updated the comment to include only one link to avoid that confusion.

[ianfixes]

Sounds like I need to revisit #164 but I don't know when I'll have time to digest all the threads that feed into that discussion (e.g. #150)