v1.1.0 failed to build with aom -DCONFIG_TUNE_BUTTERAUGLI or -DCONFIG_TUNE_VMAF

[Dzhami-Jame-ALI]

Since v1.1.0, libavif failed to build with [aom-DCONFIG_TUNE_BUTTERAUGLI] and [aom-DCONFIG_TUNE_VMAF] (aom without butteraugli or vmaf works fine with libavif, and I can still use ssim). The libjxl-butteraugli and vmaf library is pre-compiled, can be recognized and built into aom (v3.9.1). However, libavif failed to compile after building aom with butteraugli or vmaf:

...
vmaf.c:(.text+0x2b5): undefined reference to `vmaf_picture_alloc'
/usr/bin/ld: vmaf.c:(.text+0x2e1): undefined reference to `vmaf_picture_alloc'
/usr/bin/ld: vmaf.c:(.text+0x3cd): undefined reference to `vmaf_read_pictures'
/usr/bin/ld: vmaf.c:(.text+0x3e5): undefined reference to `vmaf_read_pictures'
/usr/bin/ld: vmaf.c:(.text+0x3f7): undefined reference to `vmaf_picture_unref'
/usr/bin/ld: vmaf.c:(.text+0x401): undefined reference to `vmaf_picture_unref'
/usr/bin/ld: vmaf.c:(.text+0x417): undefined reference to `vmaf_score_at_index'
/usr/bin/ld: /usr/local/lib/libaom.a(vmaf.c.o): in function `aom_read_vmaf_image':
vmaf.c:(.text+0x52d): undefined reference to `vmaf_picture_alloc'
/usr/bin/ld: vmaf.c:(.text+0x559): undefined reference to `vmaf_picture_alloc'
/usr/bin/ld: vmaf.c:(.text+0x647): undefined reference to `vmaf_read_pictures'
/usr/bin/ld: vmaf.c:(.text+0x659): undefined reference to `vmaf_picture_unref'
/usr/bin/ld: vmaf.c:(.text+0x663): undefined reference to `vmaf_picture_unref'
/usr/bin/ld: /usr/local/lib/libaom.a(vmaf.c.o): in function `aom_calc_vmaf_at_index':
vmaf.c:(.text+0x72c): undefined reference to `vmaf_score_at_index'
/usr/bin/ld: /usr/local/lib/libaom.a(vmaf.c.o): in function `aom_flush_vmaf_context':
vmaf.c:(.text+0x75b): undefined reference to `vmaf_read_pictures'
/usr/bin/ld: /usr/local/lib/libaom.a(vmaf.c.o): in function `aom_close_vmaf_model':
vmaf.c:(.text+0x81): undefined reference to `vmaf_model_destroy'
collect2: error: ld returned 1 exit status
collect2: error: ld returned 1 exit status
gmake[2]: *** [CMakeFiles/avifdec.dir/build.make:106: avifdec] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:342: CMakeFiles/avifdec.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
gmake[2]: *** [CMakeFiles/avifenc.dir/build.make:106: avifenc] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:311: CMakeFiles/avifenc.dir/all] Error 2
[ 97%] Built target avif_static
gmake: *** [Makefile:136: all] Error 2

Aom-Butteraugli's compile error log is similar.
Command i use to build aom:
cmake -DCONFIG_TUNE_VMAF=1 -DCMAKE_BUILD_TYPE=Release -DENABLE_DOCS=0 -DENABLE_EXAMPLES=0 -DENABLE_TESTDATA=0 -DENABLE_TESTS=0 -DENABLE_TOOLS=0 ..
Command I used to build libavif:
cmake -S . -B build -DBUILD_SHARED_LIBS=OFF -DAVIF_CODEC_AOM=SYSTEM -DAVIF_LIBYUV=LOCAL -DAVIF_LIBSHARPYUV=LOCAL -DAVIF_JPEG=LOCAL -DAVIF_ZLIBPNG=LOCAL -DAVIF_BUILD_APPS=ON

[y-guyon]

I was able to reproduce with the following:

sudo apt install libjxl-dev

git clone https://github.com/AOMediaCodec/libavif.git
cd libavif

git clone -b v3.9.1 --depth 1 https://aomedia.googlesource.com/aom ext/aom
cmake -S ext/aom -B ext/aom/build.libavif -DCONFIG_TUNE_BUTTERAUGLI=1 -DCMAKE_BUILD_TYPE=Release -DENABLE_DOCS=0 -DENABLE_EXAMPLES=0 -DENABLE_TESTDATA=0 -DENABLE_TESTS=0 -DENABLE_TOOLS=0 -DBUILD_SHARED_LIBS=OFF
cmake --build ext/aom/build.libavif --parallel

cmake -S . -B build -DBUILD_SHARED_LIBS=OFF -DAVIF_CODEC_AOM=LOCAL -DAVIF_LIBYUV=LOCAL -DAVIF_BUILD_APPS=ON
cmake --build build --parallel

undefined reference to JxlButteraugliApiCreate

---

Note: I failed to use STATIC_LINK_JXL=1 even with these hacky steps

git clone -b v0.10.3 --depth 1 https://github.com/libjxl/libjxl.git ext/aom/third_party/libjxl
cd ext/aom/third_party/libjxl
./deps.sh
cd ../../../..
cmake -S ext/aom/third_party/libjxl -B ext/aom/third_party/libjxl/build \
  -DBUILD_TESTING=OFF -DJPEGXL_ENABLE_BENCHMARK=OFF -DJPEGXL_ENABLE_EXAMPLES=OFF -DJPEGXL_ENABLE_JPEGLI=OFF -DJPEGXL_ENABLE_OPENEXR=OFF -DJPEGXL_ENABLE_DEVTOOLS=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF
cmake --build ext/aom/third_party/libjxl/build --parallel

touch ext/aom/third_party/libjxl/build/third_party/highway/libskcms.a
cp ext/aom/third_party/libjxl/build/third_party/brotli/libbrotlicommon.a ext/aom/third_party/libjxl/build/third_party/brotli/libbrotlicommon-static.a
cp ext/aom/third_party/libjxl/lib/jxl/butteraugli/butteraugli.h ext/aom/third_party/libjxl/build/lib/include/
cp ext/aom/third_party/libjxl/lib/jxl/butteraugli/butteraugli.h ext/aom/third_party/libjxl/build/lib/include/jxl/

cmake -S ext/aom -B ext/aom/build.libavif -DCONFIG_TUNE_BUTTERAUGLI=1 -DSTATIC_LINK_JXL=1 -DCMAKE_BUILD_TYPE=Release -DENABLE_DOCS=0 -DENABLE_EXAMPLES=0 -DENABLE_TESTDATA=0 -DENABLE_TESTS=0 -DENABLE_TOOLS=0 -DBUILD_SHARED_LIBS=OFF \
-DCMAKE_PREFIX_PATH="third_party/libjxl/build/lib/;third_party/libjxl/build/third_party/highway/;third_party/libjxl/build/third_party/brotli/;third_party/libjxl/build/lib/include/"
cmake --build ext/aom/build.libavif --parallel

[y-guyon]

To fix this issue, I guess either:

• libaom should export vmaf/butteraugli symbols in a static context (STATIC_LINK_JXL seems to exist for that purpose but I failed to build it, and I did not see a STATIC_LINK_VMAF anyway)
• libavif should link to vmaf/butteraugli libraries in a dynamic context (I wonder if it can be detected by libavif's CMakeLists.txt when this is needed or if an explicit option set by the user is necessary)

[wantehchang]

Yannis: Thank you for looking into this bug.

@Dzhami-Jame-ALI Did this work for you in an older verison of libavif, such as v1.0.4?

If so, could you provide the steps you used to build the older version of libavif with aom -DCONFIG_TUNE_BUTTERAUGLI or -DCONFIG_TUNE_VMAF? Thanks.

[wantehchang]

@jzern James: This is the issue Yannis mentioned in our team meeting.

[jzern]

It looks like libavif is missing the library dependency for libvmaf or the butteraugli libs. With either option libaom should include the dependency in its pkg-config file. For instance with -DCONFIG_TUNE_VMAF=1:

# libaom pkg-config.
prefix=/var/jenkins/tmp/aom/.build/install
exec_prefix=${prefix}
includedir=${prefix}/include
libdir=${exec_prefix}/lib

Name: aom
Description: Alliance for Open Media AV1 codec library v3.9.1-288-g570e8c4266.
Version: 3.9.1
Requires: libvmaf
Conflicts:
Libs: -L${libdir} -laom
Libs.private: -lm -lpthread
Cflags: -I${includedir}

$ pkg-config --libs aom.pc 
-L.../lib -L.../vmaf-install/lib/x86_64-linux-gnu -laom -lvmaf

libaom does not install cmake module files, however.

[Dzhami-Jame-ALI]

@wantehchang Sure.
Starting from libavif v0.10.0, aom v3.3.0, jxl v0.6.1 (until v0.8.2 due to deprecation of butteraugli) , all latest stable releases are built and tested using similar command:
(build jxl) cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF -DJPEGXL_ENABLE_BENCHMARK=OFF ..
(build aom with butteraugli) cmake -DCONFIG_TUNE_BUTTERAUGLI=1 -DCMAKE_BUILD_TYPE=Release -DENABLE_DOCS=0 -DENABLE_EXAMPLES=0 -DENABLE_TESTDATA=0 -DENABLE_TESTS=0 -DENABLE_TOOLS=0 ..
(build libavif with aom) cmake -G Ninja .. -DBUILD_SHARED_LIBS=OFF -DAVIF_CODEC_AOM=ON -DAVIF_LOCAL_AOM=OFF -DAVIF_BUILD_EXAMPLES=ON -DAVIF_BUILD_APPS=ON -DCMAKE_BUILD_TYPE=Release.

[wantehchang]

@AMDmi3 @fdintino Please take a look (especially Frankie). Thanks!

@Dzhami-Jame-ALI Thanks for the info. Because of the deprecation of Butteraugli in the jxl library, I am going to deal with vmaf first. I don't know how to fix this. A workaround is to build libaom as a shared library (with -DBUILD_SHARED_LIBS=ON).

It is complicated, because two commits are responsible for this bug. Here are the details.

I verified that v1.0.4 works but v1.1.0 doesn't.

git bisect says:

28ba2afba7e969cc40fff70fe32316422b7c96ed is the first bad commit
commit 28ba2afba7e969cc40fff70fe32316422b7c96ed (HEAD)
Author: Dmitry Marakasov <amdmi3@amdmi3.ru>
Date:   Mon Sep 25 18:09:26 2023 +0300

    Don't add ldflags to aom library (#1581)
    
    To link with aom, only AOM_LIBRARY is needed. Don't append LDFLAGS
    from pkgconfig, which contain (duplicate) flags to link the very same
    library, also in error-prone -L/-l form.
    
    Link with aom before: `/usr/local/lib/libaom.so  -L/usr/local/lib -laom`
    Link with aom after: `/usr/local/lib/libaom.so`

 cmake/Modules/Findaom.cmake | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

The pull request was written by @AMDmi3. I think the pull request makes sense for the libaom shared library, but you are building libaom as a static library (without -DBUILD_SHARED_LIBS=ON).

Reverting the above commit 28ba2af worked for a while, but it stopped working at this commit (found by another git bisection):

bccbcc032ab6b6978deab794bb6ca2fa9c18ca78 is the first bad commit
commit bccbcc032ab6b6978deab794bb6ca2fa9c18ca78 (HEAD)
Author: Frankie Dintino <fdintino@gmail.com>
Date:   Mon Dec 4 05:17:16 2023 -0500

    ci: use target properties instead of variables for flags, dirs, sources, and link libraries (#1778)

 .github/workflows/ci-windows.yml      |   1 +
 CMakeLists.txt                        | 569 ++++++++++++----------------------
 cmake/Modules/Findaom.cmake           |  10 +
[...omitted...]
 24 files changed, 631 insertions(+), 457 deletions(-)
 create mode 100644 cmake/Modules/LocalAom.cmake
 [...omitted...]

[fdintino]

I think something like this would be sufficient to pull in the library dependencies from pkg-config (replacing the current if(AOM_LIBRARY) conditional at the end of FindAom.cmake)

if(AOM_LIBRARY)
    if("${AOM_LIBRARY}" MATCHES "\\${CMAKE_STATIC_LIBRARY_SUFFIX}$")
        add_library(aom STATIC IMPORTED GLOBAL)
        set(_AOM_PC_PREFIX "_AOM_STATIC")
    else()
        add_library(aom SHARED IMPORTED GLOBAL)
        set(_AOM_PC_PREFIX "_AOM")
    endif()
    set_target_properties(aom PROPERTIES IMPORTED_LOCATION "${AOM_LIBRARY}")
    target_include_directories(aom INTERFACE ${AOM_INCLUDE_DIR})

    target_link_directories(aom INTERFACE "${${_AOM_PC_PREFIX}_LIBRARY_DIRS}")

    set(_AOM_PC_LIBRARIES "${${_AOM_PC_PREFIX}_LIBRARIES}")
    # remove "aom" so we only have library dependencies
    list(REMOVE_ITEM _AOM_PC_LIBRARIES "aom")
    target_link_libraries(aom INTERFACE "${_AOM_PC_LIBRARIES}")
endif()

Something similar can be done for aom that was compiled into ext/aom/build.libavif in LocalAom.cmake, after these lines

libavif/cmake/Modules/LocalAom.cmake

Lines 17 to 21 in 7d9522d

	if(EXISTS "${LIB_FILENAME}")
	message(STATUS "${AOM_MESSAGE_PREFIX}: compiled library found at ${LIB_FILENAME}")
	add_library(aom STATIC IMPORTED GLOBAL)
	set_target_properties(aom PROPERTIES IMPORTED_LOCATION "${LIB_FILENAME}" AVIF_LOCAL ON)
	target_include_directories(aom INTERFACE "${AOM_EXT_SOURCE_DIR}")

By prepending the build.libavif directory to PKG_CONFIG_PATH and using the variables set by pkg_check_modules

    if(MSVC)
        set(ENV{PKG_CONFIG_PATH} "${AOM_EXT_SOURCE_DIR}/build.libavif;$ENV{PKG_CONFIG_PATH}")
    else()
        set(ENV{PKG_CONFIG_PATH} "${AOM_EXT_SOURCE_DIR}/build.libavif:$ENV{PKG_CONFIG_PATH}")
    endif()
    pkg_check_modules(_AOM QUIET aom)
    target_link_directories(aom INTERFACE ${_AOM_STATIC_LIBRARY_DIRS})
    set(_AOM_PC_LIBRARIES ${_AOM_STATIC_LIBRARIES})
    list(REMOVE_ITEM _AOM_PC_LIBRARIES "aom")
    target_link_libraries(aom INTERFACE ${_AOM_PC_LIBRARIES})

I could probably open a PR with these changes tomorrow or this weekend if it would be helpful.

[wantehchang]

In #2274 (comment), I said "A workaround is to build libaom as a shared library (with -DBUILD_SHARED_LIBS=ON)." That workaround did not work today, so I must have made a mistake when I investigated this bug last time.

Based on the incorrect info, I asked Frankie to restrct his fix to the libaom static library case. So his fix (#2304) only fixes the libaom static library case and does not fix the libaom shared library case. Sorry!

I verified that we just need to go back to an earlier version of his fix.

[wantehchang]

Here are the steps I used to reproduce the bug with libvmaf. Note that I use the /home/wtc/tmp/prefix directory as the install prefix. I am on Linux x86_64.

export PREFIX=/home/wtc/tmp/prefix
export PKG_CONFIG_PATH=$PREFIX/lib/x86_64-linux-gnu/pkgconfig:$PREFIX/lib/pkgconfig

rm -rf $PREFIX

cd vmaf/libvmaf
rm -rf build
# The default of libvmaf is to build both static and shared libraries.
# Add --default-library static to build only the static library.
meson setup build --buildtype release --prefix $PREFIX
ninja -C build
ninja -C build install
cd ../..

cd aom
rm -rf build.libavif
mkdir build.libavif
cd build.libavif
# The default of libaom is to build only the static library.
# Add -DBUILD_SHARED_LIBS=1 to build both static and shared libraries.
cmake .. -G Ninja -DCONFIG_TUNE_VMAF=1 -DCMAKE_BUILD_TYPE=Release -DENABLE_DOCS=0 -DENABLE_EXAMPLES=0 -DENABLE_TESTDATA=0 -DENABLE_TESTS=0 -DENABLE_TOOLS=0 -DCMAKE_INSTALL_PREFIX=$PREFIX
ninja
ninja install
cd ../..

cd libavif
rm -rf build
mkdir build
cd build
cmake -G Ninja .. -DBUILD_SHARED_LIBS=OFF -DAVIF_CODEC_AOM=SYSTEM -DAVIF_LIBYUV=OFF -DAVIF_BUILD_EXAMPLES=ON -DAVIF_BUILD_APPS=ON -DCMAKE_BUILD_TYPE=Release
ninja
cd ../..

[wantehchang]

I figured out what I did differently in my testing. Last time I set the LD_LIBRARY_PATH environment variable to /home/wtc/tmp/prefix/lib/x86_64-linux-gnu, the directory where libvmaf.so.3 is installed. Today I forgot to do that.

Frankie: I am no longer sure if we need to add back your fix for the shared library case. What do you think?

[wantehchang]

I am going to consider it a libvmaf installation error if libvmaf.so is not on the run-time linker's search path.

I attached two shell scripts, static.sh and shared.sh, which I used to reproduce the bug and verify the fix.

static.sh builds libvmaf and libaom as static libraries.

#!/bin/bash
export PREFIX=/home/wtc/tmp/prefix
export PKG_CONFIG_PATH=$PREFIX/lib/x86_64-linux-gnu/pkgconfig:$PREFIX/lib/pkgconfig

rm -rf $PREFIX

cd vmaf/libvmaf
rm -rf build
meson setup build --buildtype release --default-library static --prefix $PREFIX
ninja -C build
ninja -C build install
cd ../..

cd aom
rm -rf build.libavif
mkdir build.libavif
cd build.libavif
cmake .. -G Ninja -DCONFIG_TUNE_VMAF=1 -DCMAKE_BUILD_TYPE=Release -DENABLE_DOCS=0 -DENABLE_EXAMPLES=0 -DENABLE_TESTDATA=0 -DENABLE_TESTS=0 -DENABLE_TOOLS=0 -DCMAKE_INSTALL_PREFIX=$PREFIX
ninja
ninja install
cd ../..

cd libavif
rm -rf build
mkdir build
cd build
cmake -G Ninja .. -DBUILD_SHARED_LIBS=OFF -DAVIF_CODEC_AOM=SYSTEM -DAVIF_LIBYUV=OFF -DAVIF_BUILD_EXAMPLES=ON -DAVIF_BUILD_APPS=ON -DCMAKE_BUILD_TYPE=Release
ninja
cd ../..

shared.sh builds libvmaf and libaom as shared libraries.

#!/bin/bash
export PREFIX=/home/wtc/tmp/prefix
export PKG_CONFIG_PATH=$PREFIX/lib/x86_64-linux-gnu/pkgconfig:$PREFIX/lib/pkgconfig

rm -rf $PREFIX

cd vmaf/libvmaf
rm -rf build
meson setup build --buildtype release --prefix $PREFIX
ninja -C build
ninja -C build install
cd ../..

cd aom
rm -rf build.libavif
mkdir build.libavif
cd build.libavif
cmake .. -G Ninja -DCONFIG_TUNE_VMAF=1 -DCMAKE_BUILD_TYPE=Release -DENABLE_DOCS=0 -DENABLE_EXAMPLES=0 -DENABLE_TESTDATA=0 -DENABLE_TESTS=0 -DENABLE_TOOLS=0 -DBUILD_SHARED_LIBS=1 -DCMAKE_INSTALL_PREFIX=$PREFIX
ninja
ninja install
cd ../..

cd libavif
rm -rf build
mkdir build
cd build
cmake -G Ninja .. -DBUILD_SHARED_LIBS=OFF -DAVIF_CODEC_AOM=SYSTEM -DAVIF_LIBYUV=OFF -DAVIF_BUILD_EXAMPLES=ON -DAVIF_BUILD_APPS=ON -DCMAKE_BUILD_TYPE=Release
ninja
cd ../..