Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to ndk-r25 #11422

Merged
merged 35 commits into from
Aug 13, 2022
Merged

Update to ndk-r25 #11422

merged 35 commits into from
Aug 13, 2022

Conversation

Grimler91
Copy link
Member

@Grimler91 Grimler91 commented Aug 2, 2022

Here's a start of an update to ndk-r25. ndk-patches and toolchain wise it seems trivial, it's exactly the same as for ndk-r23c.

Packages like ndk-sysroot, ndk-multilib, libc++ need to be updated/bumped still.

I've re-factored a bit to make it easier to jump between NDK versions. With these changes, to try out a new version, we can just change TERMUX_NDK_VERSION, add a new termux_step_setup_toolchain_##.sh function, and new ndk-patches, and add relevant calls to the if statements.

We might want to keep possibility to use r23c around since it is the last version supporting -mfpu=vfpv3-d16.

In the future we might also add option to use r17c, as that was the last version with gcc, and gcc/gfortran is needed to compile packages like python-scipy over at termux-user-repository.

I don't want us to NDK-dependent packages like ndk-sysroot compatible with all possible ndk-versions though, such packages should still only be buildable with main NDK we use, so adding possibility for multiple NDK versions is really just to simplify testing/debugging/development.

@Grimler91 Grimler91 linked an issue Aug 2, 2022 that may be closed by this pull request
@Grimler91 Grimler91 requested review from licy183 and truboxl August 2, 2022 11:33
@finagolfin
Copy link
Member

Forgot to change the default NDK to 25 in scripts/properties.sh?

Copy link
Member

@licy183 licy183 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is pretty helpful for using a custom toolchain! Wonderful job!

@Grimler91
Copy link
Member Author

Forgot to change the default NDK to 25 in scripts/properties.sh?

Now it is actually updated in the PR.. :) and ndk-sysroot/multilib updated.

Not sure if multilib contains all files necessary for cross-compilation after the update (haven't tested), but file list looks reasonable:

--- ndk-multilib-23c.list	2022-08-03 08:25:06.621514731 +0200
+++ ndk-multilib-25.list	2022-08-03 08:24:57.611522729 +0200
@@ -86,6 +86,10 @@
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.asan-preinit-arm-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.asan-preinit-i686-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.asan-preinit-x86_64-android.a
+./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.asan_static-aarch64-android.a
+./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.asan_static-arm-android.a
+./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.asan_static-i686-android.a
+./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.asan_static-x86_64-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.asan-x86_64-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.asan-x86_64-android.a.syms
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.asan-x86_64-android.so
@@ -108,6 +112,11 @@
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.hwasan-aarch64-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.hwasan-aarch64-android.a.syms
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.hwasan-aarch64-android.so
+./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.hwasan_aliases_cxx-x86_64-android.a
+./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.hwasan_aliases_cxx-x86_64-android.a.syms
+./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.hwasan_aliases-x86_64-android.a
+./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.hwasan_aliases-x86_64-android.a.syms
+./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.hwasan_aliases-x86_64-android.so
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.hwasan_cxx-aarch64-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.hwasan_cxx-aarch64-android.a.syms
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.hwasan_cxx-x86_64-android.a
@@ -119,34 +128,13 @@
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.lsan-arm-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.lsan-i686-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.lsan-x86_64-android.a
+./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.orc-aarch64-android.a
+./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.orc-arm-android.a
+./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.orc-x86_64-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.profile-aarch64-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.profile-arm-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.profile-i686-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.profile-x86_64-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo-aarch64-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo-aarch64-android.so
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo-arm-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo-arm-android.so
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_cxx-aarch64-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_cxx-arm-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_cxx-i686-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_cxx_minimal-aarch64-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_cxx_minimal-arm-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_cxx_minimal-i686-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_cxx_minimal-x86_64-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_cxx-x86_64-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo-i686-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo-i686-android.so
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_minimal-aarch64-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_minimal-aarch64-android.so
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_minimal-arm-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_minimal-arm-android.so
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_minimal-i686-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_minimal-i686-android.so
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_minimal-x86_64-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo_minimal-x86_64-android.so
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo-x86_64-android.a
-./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.scudo-x86_64-android.so
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.stats-aarch64-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.stats-arm-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.stats_client-aarch64-android.a
@@ -157,12 +145,14 @@
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.stats-x86_64-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.tsan-aarch64-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.tsan-aarch64-android.a.syms
+./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.tsan-aarch64-android.so
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.tsan_cxx-aarch64-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.tsan_cxx-aarch64-android.a.syms
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.tsan_cxx-x86_64-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.tsan_cxx-x86_64-android.a.syms
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.tsan-x86_64-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.tsan-x86_64-android.a.syms
+./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.tsan-x86_64-android.so
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.ubsan_minimal-aarch64-android.a
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.ubsan_minimal-aarch64-android.a.syms
 ./data/data/com.termux/files/usr/opt/ndk-multilib/cross-compiler-rt/libclang_rt.ubsan_minimal-aarch64-android.so

@finagolfin
Copy link
Member

I'm guessing the lld 13+ issue with --gc-sections is going to break some packages, like the upcoming Swift 5.7 as detailed in swiftlang/swift-package-manager#5698, since we were using lld 12 till now in NDK 23c. I'm going to kick off builds of libllvm and Swift with this pull, to see if they work well.

@Grimler91 Grimler91 force-pushed the ndk-r25 branch 2 times, most recently from 70cb0d5 to 3da34b1 Compare August 5, 2022 09:11
@Grimler91
Copy link
Member Author

Grimler91 commented Aug 5, 2022

Force-pushed to rebase and remove the termux_setup_standalone_toolchain function.

Also added a separate termux_step_setup_toolchain function for ndk-r25, it seems that our $CPP wrapper has to be removed. With it, autoconf packages fail during configure with:

checking how to run the C preprocessor... aarch64-linux-android-cpp
configure: error: in `/home/builder/.termux-build/0verkill/src':
configure: error: C preprocessor "aarch64-linux-android-cpp" fails sanity check

and in config.log:

configure:4406: aarch64-linux-android-cpp  -I/data/data/com.termux/files/usr/include conftest.c
/home/builder/.termux-build/_cache/android-r25-api-24-v0/bin/aarch64-linux-android-cpp: line 4: /home/builder/.termux-build/_cache/android-r25-api-24-v0/bin/clang -E: No such file or directory

Don't know why, /home/builder/.termux-build/_cache/android-r25-api-24-v0/bin/clang exists and is a symlink for both r23c and r25. (clearly it tries to run /home/builder/.termux-build/_cache/android-r25-api-24-v0/bin/clang -E, and a file with that name doesn't exist, but as mentioned it looks like behaviour should be identical compared to r23c, but it isn't)

@truboxl
Copy link
Contributor

truboxl commented Aug 5, 2022

Weird... Shouldn't there be other packages coming out besides libc++ when they are set with $TERMUX_NDK_VERSION?

@Grimler91
Copy link
Member Author

Weird... Shouldn't there be other packages coming out besides libc++ when they are set with $TERMUX_NDK_VERSION?

What do you mean? You mean that packages depending on libc++ should be rebuilt, then yeah, that should be done, but we have to bump their revisions manually

With recently added commit for vulkan-loader-android all packages that depend on ndk libs should have been touched and will be rebuilt by CI

@truboxl
Copy link
Contributor

truboxl commented Aug 5, 2022

I mean when TERMUX_PKG_VERSION is bumped, I reckon the packages with that set to a higher TERMUX_NDK_VERSION will be auto built.

True for ndk-multilib and ndk-sysroot but not for vulkan-loader-android which need another TERMUX_PKG_REVISION...

I guess that wasnt the case here huh... Ok from the other thread I took note already

@Grimler91
Copy link
Member Author

I mean when TERMUX_PKG_VERSION is bumped, I reckon the packages with that set to a higher TERMUX_NDK_VERSION will be auto built.

Yeah, it's because our CI scripts aren't intelligent enough, they rely on git diff to find modified packages, if directory packages/libc++/ is untouched it won't notice that version has changed since TERMUX_PKG_VERSION=$TERMUX_NDK_VERSION, and TERMUX_NDK_VERSION is defined in scripts/properties.sh rather than packages/libc++/build.sh

@@ -20,6 +20,9 @@ termux_step_extract_into_massagedir() {
# replace vulkan headers with upstream version
rm -rf $TERMUX_PKG_MASSAGEDIR/$TERMUX_PREFIX/include/vulkan

# replace ICU headers with upstream version
rm -rf $TERMUX_PKG_MASSAGEDIR/$TERMUX_PREFIX/include/unicode
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebased and avoided this libicu collision.

@finagolfin
Copy link
Member

I built libllvm and swift with this pull for AArch64 and tested them on Android 12 without a problem, after applying the last two commits that I pushed to this pull.

We'll need to kick off a build of all packages with this pull since it is a major version bump, so we catch any regressions.

@tomty89
Copy link
Contributor

tomty89 commented Aug 10, 2022

If it's not too much of a distraction, please kindly include #11476 in the rebuilding of all packages.

Grimler91 and others added 21 commits August 13, 2022 12:21
Or else build of boinc fails due to undefined backtrace_symbols{,_fd}.
Source archive for 2.5.1 no longer exist, so update to 2.5.2.
asymptote tries to link against libtinfo and libreadline by using
linker commands like -Wl,-Bstatic -lreadline -Wl,-Bdynamic, which
means that asymptote needs the libs during build, but not runtime.
After update of ffmpeg to 5.1.  gegl, megacmd, mpd, timg, vlc, kid3
and vlc-qt have not been bumped in this commit as they were rebuilt as
part of ndk-r25 update already.
Or else build fails with undefined reference to backtrace.
To try to fix compilation error.
SSL certificate has expired on website.
We configure with --with-libtiff=internal (which was required by jxl),
so no need to depend on libtiff.

Also sort dependencies and configure args alphabetically.
We have to remove the -fno-integrated-as flag, and then also set
DENABLE_ASSEMBLY=off for aarch64 as well.
With ndk-r25 we cannot use the gnu assembler, and with the llvm
internal assembler we get build errors, at least for arm.
removing the note on rev bump for mpv since mpv hasn't been the
only package that has linkage to ffmpeg for a long time

ffmpeg-5.0-vp9-dts-correction.patch is in 5.1 as commit
68595b46cb374658432fff998e82e5ff434557ac.
Configure with --disable-vulkan to avoid error due to vulkan.h trying
to include vulkan_beta.h from a third_party directory in the ndk.
Remove -fno-integrated-as flag which no longer works with ndk-r25 as
the non-internal (non-llvm) assembler is not included in the ndk anymore.

Also disable vpx and x265 support, libvpx and libx265 has compilation
issues (cannot compile with llvm assembler).  Disable until we have
managed to fix that.
@Grimler91 Grimler91 merged commit 9aefd2d into master Aug 13, 2022
@Grimler91 Grimler91 deleted the ndk-r25 branch August 13, 2022 10:24
@Grimler91
Copy link
Member Author

Some packages have ended up in the wrong repo (termux-x11 vs termux-main) because of bad wildcard usage on my part, for example tsmuxer/tsmuxergui and qemu-system-x86-64-headless.

Will untangle the mess later

Grimler91 added a commit that referenced this pull request Sep 11, 2022
licy183 pushed a commit that referenced this pull request Sep 11, 2022
licy183 pushed a commit that referenced this pull request Sep 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: cec-client: library "libcec.so.6" not found LTS NDK 25 is now out
6 participants