Enhance CMake to be able to support static link for Linux and macOS #1159
Labels
enhancement
New feature or request
Comments
|
We can see that Arduino is using the following patch on top of avrdude 7.0 release to link to static libraries. But it does not seem to work with git main as of now. |
|
I can try to do some digging into this tonight for MacOS. I don't know much about cmake, but I did have a lot of trouble building on MacOS (using homebrew) recently when trying to solve another issue. |
@danapplegate Please also give the current git main and the following PR a try as well. Thanks. |
|
Just an example under Linux. We can see that the dependancies are not pulled in, like libudev, pthread, etc. Please also take note that pkg-config is also not very reliable for static build. Once we add them, it will work. |
|
@mcuee Ok, calling it quits for tonight, but I was able to get through compilation using USE_STATIC_LIBS=ON. Unfortunately, I'm now getting some gnarly failures in the linker step. First, I noticed that it appeared that CMake was not properly finding the header files installed by Homebrew in Example output from cmake configure step
I fixed this by setting Configure step output with CMAKE_C_FLAGS=-I/usr/local/include
Now, I am getting "Undefined symbols for architecture x86_64" from trying to link the If anyone has any suggestions, I'm all ears. |
|
@danapplegate You need to specify the CMake option as per here. Default dynamic link option: For static link option, and for debugging, I will suggest you use the following option (it will fail in the last step, similar to my build log here: https://github.com/mcuee/avrdude/actions/runs/3312632411/jobs/5469546749). |
|
@mcuee Yeah, even with |
|
For Arduino, they have two patches for static linking for avrdude 7.0 release.
|
|
Following the approach from Arduino, the following patch seems to be good now for Linux. |
|
@mariusgreuel and @umbynos |
|
@danapplegate we use this to build macos: https://github.com/arduino/avrdude-packing/blob/7295e15feca9d40122a61ce7ca2842b9ab3da00e/.github/workflows/release.yml#L91 |
|
Info for macOS using pkg-config and Homebrew. Therefore the patch from Arduino is still good (we need ncurses). And then we need to add the few frameworks (IOkit, CoreFoundation, Security). We can see that the Arduino build instruction by @umbynos takes care of the above frameworks. |
|
The same approach can be done for MSYS2 mingw32/mingw64. Of course I will still need to rebuild termcap as per the workaround mentioned here. |
|
I tend to think some parts of the Arduino avrdude workflow file here seem to be good to be adopted by the official avrdude project as well. Good work! https://github.com/arduino/avrdude-packing/blob/main/.github/workflows/release.yml |
|
My trial and errors to use static linking for Linux and macOS are here. It is still using dynamic linking for hidapi, strange. |
A minor improvement to supress many warnings is here for the Arduino script. The frameworks are supposed to be for the linker option and not compiler option. I tested it in the github action and it works. |
|
For those who want to try out the Arduino crossbuild container, it is here, thanks to @umbynos. |
|
Just tried the container and it works well -- only tested v7.0 release and Linux. It may need minor tuning for git main -- I will try that tomorrow. |
And it works well for git main as well. I only need one patch as I do not need to use Arduino branding. Tested under Ubuntu 20.04. |
|
For macOS, it is also okay with git main. Tested under my Mac Mini M1. |
|
But the above generates one warning in configure stage (C++ compiler is not necessary) and too many warnings building the compiling stage (frameworks should be linker script and not compiler script). I also think AppKit is necessary. But I am not so sure if older version of macOS supports it or not. The following may be better. avrdude_macos_intel64bit_alt.zip Tested under my Mac Mini M1. |
|
@umbynos Then the build will still fail. Not so sure what is wrong. Your build here (https://github.com/arduino/avrdude-packing/files/9889835/avrdude-main-test-libusb-compat.zip) works very well. So the build process should work. The linking failed but the command seems to be correct to me. Strange. Edit: I figured out the issues.
|
|
May latest efforts mentioned here, still got problem with Windows. |
|
The thing is whether this patch is acceptable or not to avrdude project. |
I think this is a fair statement based on my experiences with other projects. In this case, I have closed PR #1275 as it is not an avrdude problem. |
|
Actually I can build git main under macOS using static link without any patches. |
|
However, I have not figured a way to do that for Linux. The linking command is basically correct but the sequence is wrong. |
|
It seems to me git main is not too bad to address this issue. Please take a look at the Linux build issue when you got some time. Thanks. |
|
Working macOS build scripts. Take note I have manually fixed the hidapi forumla to have the static library installed. |
mcuee
changed the title
|
It seems to me now you only need a small patch to get avrdude 7.2 build using your build script. The patch is a bit strange though. Looks like |
|
BTW, please help to check the following feature request for the Arduino crossbuild container. Thanks. |
|
The Linux issue is still there. Need to figure out which CMAKE_EXE_LINKER_FLAGS to use. Full build log -- failed in the linking step. full build log |
|
Sorry to disturb you here. Just wondring if you have some ideas here. Thanks. |
|
This one seems to work. I am not so sure if it is the right way though. I think it is just a hack. |
This is fine, should be the only required parameter.
Needing to pass these from command line is not fine. |
|
PR #1540 has been merged. This issue is still valid, therefore, I will keep it open. |
|
It is a bit strange that cross compile works fine with the script in PR #1540 (now merged). But it does not work if I adopt similar approach with native build under Linux. Just wondering if you have some ideas or not. This does not work with native build under Linux. The following cross-compile srcipt works fine. For example, the following is from latest git main github avrdude-packing build. |
|
I'm not sure what is not working? |
The linking step failed. failed build log with verbose output |
|
Basically the issue is that the linking step is missing |
Right, so those should be explicitly added in CMake for platforms that requires it. |
Take note this is only required for static linking under Linux/macOS/BSDs (the default is using dynamic linking. So probably this kind of extra options can be added for them. For Windows by default avrdude is using static linking and not really allowing dynamic linking as of now (maybe no point to add the support for dynamic linking either). |
|
I am not so sure if you have some interests in helping here as well. I believe the fix is not that difficult but unfortunately I do not know much about CMake. BTW, we may extend the subject for autotools build as well. |
|
@mcuee Yes, I am interested. I am using avrdude to learn cmake with, I want to build a good avrdude package for Fedora, and I want avrdude to have a good Autotools build. Thanks for mentioning me here. TLDR: A large number of different builds are possible, and we need to decide which ones make sense for us to support, and determine how to. I will connect the following general thoughts with the existing build system some time in the next few days. There are two build products: The libavrdude library and the Without precisely counting them all, that results in a huge number of combinations, and not all of them make sense. Here are a few of those combinations which make sense, starting with the most extreme ones:
So someone building the code might want to select:
Defaults could differ depending on the target OS, e.g. mostly static for Windows and mostly dynamic for BSD, Linux, OSX. |
|
Thanks a lot for the detailed analysis and I agree with you.
For avrdude binary, I think this is the case already for the CMake build (MSVC and MSYS2 mingw) under Windows, and CMake/autools build under other OS.
Yes there are pretty strong requests to have this types of static binaries for Linux and macOS. Arduino avrdude-packing project is distributing such binaries for Linux, macOS and Windows. We have the CI script based on Arduino avrdude-packing project as well but needs some improvement (eg: no libserialport support). For libavrdude library, currently it is not possible to build dynamic library for Windows and there may be such use case if we want to allow 3rd party apps developer who do not want to open source their implementation. There may be such use case (dynamic library) for other OS as well. This is a moot point as of now there are no real GUI application which leverage libarvdude library.
|
|
Example working static build under Linux. detail build log under Ubuntu Linux 20.04Results: Normal build outcome (using default |
|
@ndim Good analysis.
Isn't the critical part here that avrdude is linked with the libavrdude dynamic library that is shipped at the same time?Could that be enforced by correct versioning? I agree, linking avrdude statically against libavrdude guards against this type of error. Anyway, I have too little experience in building projects for different platforms to have a preference for which options the AVRDUDE project should have. |
|
I am not concerned about compatibility between the avrdude tool and the libavrdude library. Both are developed by the same people in the same repository after all. But if you offer libavrdude to a global audience to allow them to develop tools which make use of libavrdude, you must expect that third parties actually start using libavrdude in their software and prepare for that scenario. That is when incompatibly changing libavrdude can break those third party tools. Either by breaking the ABI (relevant for dynamic linking, mainly on BSD, Linux, OSX) or by breaking the API (relevant for all systems). Then libavrdude devs need to coordinate with the external libavrdude users, each distro needs to coordinate rebuilding third party tool packages, and users who have built third party tools dynamically linked against a OS package will have them broken. Or with API changes, third party software might stop building with a newer libavrdude. So with an unstable libavrdude API/ABI, fully dynamic builds do not provide much value. The loooong list of part specific constants macro definitions might need to be converted into data structures and functions acting on those which represent the semantics of those compile time definitions at runtime. So the API/ABI would only cover a few functions getting and dealing with those part specific data structures which are provided by libavrdude. Then those data structures and functions could remain stable in the longer term. You might need to add a new function from time to time which does not break existing API or ABI. Anyway, API/ABI design specifics are getting a little off topic for an issue which is primarily about build types. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Ref:
Indeed git main static link build for Linux and macOS failed. This may pose some challenges for Arduino people. Maybe @umbynos wants to take a look. I think the fix may not be too complicated but it does involve changing CMake files.
Failed build log:
https://github.com/mcuee/avrdude/actions/runs/3312632411
This will also help the following issue.
Take note this issue is different from the following issue.
Therefore the following pull-request will not help.
* https://github.com/avrdudes/avrdude/
The text was updated successfully, but these errors were encountered: