Building failure after switching from 3.1.47 to 3.1.57 #22008
Comments
|
Can you see which version of It seems that this function is being miscompiled somehow? Can you attach the resulting wasm file perhaps? Running |
|
If nothing else helps then bisection may be useful here, https://emscripten.org/docs/contributing/developers_guide.html#bisecting |
|
Using the very same build instructions, produced 3.1.47.wasm (working) is 26MB, 3.1.57.wasm is 23.7MB (failing runtime) I have tried to dig deeper into the problem using tracing symbols. 3.1.473.1.47.zip - attached contains .a, .h, .wasm files and demo.html 3.1.57It appears that both Any ideas how to proceed further? |
|
Almost certainly the produced .wasm file here is broken. Can you attach the broken wasm file perhaps? Also, can you try bisecting between 3.1.47 and 3.1.57 to find the exact change that broke this (See https://emscripten.org/docs/contributing/developers_guide.html#bisecting for details on how to do this). |
|
It is attached as 3.1.57.zip in the previous comment |
|
@sbc100 I would love to narrow down the exact version of emsdk which introduced the issue, but that is not possible since 3.1.48 to 3.1.56 are not available for mac M2 as I mentioned in the original post: The attached 3.1.57.zip includes broken .wasm and some more files, and can be compared to working 3.1.47.zip |
|
I have put together min reproducible build setup which I run in #!/bin/bash
# running all the commands in DOCKER:
# docker run -it -v $(pwd):/ffmpeg-wasm -w /ffmpeg-wasm debian:12.5
apt-get update
apt-get install -y git python3.11 build-essential cmake autoconf autogen automake libtool pkg-config ragel wget
git config --global pull.rebase false
ln -sf /usr/bin/python3.11 /usr/bin/python
# EMSDK
git clone --depth=1 --branch main https://github.com/emscripten-core/emsdk/
(cd emsdk && ./emsdk install 3.1.57)
(cd emsdk && ./emsdk activate 3.1.57)
source ./emsdk/emsdk_env.sh
# SETUP
export PATH=$PATH:$EMSDK/upstream/bin
ROOT_DIR=$PWD
WASM_DIR=$ROOT_DIR/wasm
BUILD_DIR=$ROOT_DIR/build
PKG_CONFIG_PATH=$BUILD_DIR/lib/pkgconfig
export EM_PKG_CONFIG_PATH=$PKG_CONFIG_PATH
TOOLCHAIN_FILE=$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake
export CFLAGS="-O0 -I$BUILD_DIR/include -pthread -msimd128"
export CXXFLAGS=$CFLAGS
export LDFLAGS="$CFLAGS -L$BUILD_DIR/lib"
mkdir -p $WASM_DIR
# AOM
git clone --depth=1 --branch v3.9.0 https://aomedia.googlesource.com/aom/
rm -rf aom/CMakeCache.txt
rm -rf aom/CMakeFiles
rm -rf aom/aom_build
mkdir -p aom/aom_build
(cd aom/aom_build && emmake cmake .. \
-DCMAKE_TOOLCHAIN_FILE=$TOOLCHAIN_FILE \
-DAOM_TARGET_CPU=generic \
-DENABLE_DOCS=0 -DENABLE_TESTS=0 \
-DCONFIG_RUNTIME_CPU_DETECT=0 \
-DCONFIG_WEBM_IO=0 \
-DCMAKE_INSTALL_PREFIX=$BUILD_DIR \
-DBUILD_SHARED_LIBS=0 \
-DENABLE_EXAMPLES=0 \
-DENABLE_TOOLS=0 \
-DCMAKE_BUILD_TYPE=Release \
-DAOM_EXTRA_C_FLAGS="$CFLAGS" \
-DAOM_EXTRA_CXX_FLAGS="$CXXFLAGS" \
-G"Unix Makefiles")
emmake make -C aom/aom_build clean
emmake make -C aom/aom_build install -j
# VPX
git clone --depth=1 --branch v1.14.0 https://github.com/webmproject/libvpx/
(cd libvpx && emconfigure ./configure \
--prefix=$BUILD_DIR \
--target=generic-gnu \
--disable-install-bins \
--disable-examples \
--disable-tools \
--disable-docs \
--disable-unit-tests \
--disable-dependency-tracking \
--disable-shared \
--disable-codec-srcs \
--disable-debug-libs \
--extra-cflags="$CFLAGS" \
--extra-cxxflags="$CXXFLAGS")
emmake make -C libvpx clean
emmake make -C libvpx install STRIP=emstrip -j
## FFmpeg
git clone --depth=1 --branch n7.0.1 https://github.com/FFmpeg/FFmpeg
(cd FFmpeg && emconfigure ./configure \
--target-os=none \
--arch=x86_32 \
--enable-cross-compile \
--enable-version3 \
--enable-libaom \
--disable-encoder=libaom_av1 \
--enable-libvpx \
--disable-x86asm \
--disable-inline-asm \
--disable-stripping \
--disable-programs \
--disable-doc \
--disable-debug \
--disable-runtime-cpudetect \
--disable-autodetect \
--extra-cflags="$CFLAGS" \
--extra-cxxflags="$CXXFLAGS" \
--extra-ldflags="$LDFLAGS" \
--pkg-config-flags="--static" \
--nm=emnm \
--ar=emar \
--ranlib=emranlib \
--cc=emcc \
--cxx=em++ \
--objcc=emcc \
--dep-cc=emcc)
emmake make -C FFmpeg -j4
(cd FFmpeg && emcc \
$LDFLAGS \
-I. -I./fftools -I$BUILD_DIR/include \
-Llibavcodec -Llibavdevice -Llibavfilter -Llibavformat -Llibavresample -Llibavutil -Llibswscale -Llibswresample \
-Wno-deprecated-declarations -Wno-pointer-sign -Wno-implicit-int-float-conversion -Wno-switch -Wno-parentheses -Qunused-arguments \
-lavdevice -lavfilter -lavformat -lavcodec -lswresample -lswscale -lavutil -lm \
-laom \
-lvpx \
fftools/cmdutils.c \
fftools/ffmpeg.c \
fftools/ffmpeg_dec.c \
fftools/ffmpeg_demux.c \
fftools/ffmpeg_enc.c \
fftools/ffmpeg_filter.c \
fftools/ffmpeg_hw.c \
fftools/ffmpeg_mux.c \
fftools/ffmpeg_mux_init.c \
fftools/ffmpeg_opt.c \
fftools/ffmpeg_sched.c \
fftools/objpool.c \
fftools/opt_common.c \
fftools/sync_queue.c \
fftools/thread_queue.c \
-lworkerfs.js \
-s USE_SDL=2 \
-s WASM_BIGINT \
-s INVOKE_RUN=0 \
-s EXIT_RUNTIME=1 \
-s MODULARIZE=1 \
-s EXPORT_NAME="createFFmpeg" \
-s EXPORTED_FUNCTIONS="[_main, ___wasm_init_memory_flag]" \
-s EXPORTED_RUNTIME_METHODS="[callMain, FS, WORKERFS]" \
-s INITIAL_MEMORY=128mb \
-s ALLOW_MEMORY_GROWTH=1 \
-s MAXIMUM_MEMORY=4gb \
-s ENVIRONMENT=worker \
-s PROXY_TO_PTHREAD=1 \
-s STACK_SIZE=5MB \
-s DEFAULT_PTHREAD_STACK_SIZE=2MB \
-o $WASM_DIR/ffmpeg-lgpl-simd.js)The output .wasm and .js is then used in demo.html <body>
<p>Watch console log.</p>
<script id="worker" type="javascript/worker">
self.onmessage = async (event) => {
const {args, wasmUrl, ffmpegUrl, ffmpegWorkerUrl} = event.data;
importScripts(ffmpegUrl);
const module = await createFFmpeg({
printErr:console.log,
locateFile:url => {
if(url.endsWith(".wasm")) return wasmUrl;
if(url.endsWith(".worker.js")) return ffmpegWorkerUrl;
return url;},
mainScriptUrlOrBlob:ffmpegUrl});
module.callMain(args);
};
</script>
<script>
(async () => {
if(typeof SharedArrayBuffer === "undefined")
return document.body.innerHTML = "SharedArrayBuffer is not available. Run <code style='background:#ccc'>chrome --enable-features=SharedArrayBuffer</code> or follow <a href='https://web.dev/cross-origin-isolation-guide/'>A guide to enable cross-origin isolation</a>.";
const wasmUrl = new URL("ffmpeg-lgpl-simd.wasm", document.location).href;
const ffmpegUrl = new URL("ffmpeg-lgpl-simd.js", document.location).href;
const ffmpegWorkerUrl = new URL("ffmpeg-lgpl-simd.worker.js", document.location).href;
const args = ["-codecs"];
const workerBlob = new Blob([document.querySelector('#worker').textContent], {type:"text/javascript"});
const worker = new Worker(URL.createObjectURL(workerBlob));
worker.postMessage({args, wasmUrl, ffmpegUrl, ffmpegWorkerUrl});
})()
</script>
</body>which eventually prints to console log The attached outputs are attached in 3.1.57.zip . Please scroll few comments up for more details. Following emscripten changelog there has been few upgrades to llvm version (between 3.1.47 and 3.1.57) but nothing else that looks suspicious. Considering the wasm-dis log with so many Can you please help me narrow down the problem in emscripten? |
|
That wasm file does indeed look invalid, wasm-opt says As this is a debug build that likely means a clang or wasm-ld issue. Bisection may still be best. Even if 3.1.48 to 3.1.56 are not available for mac M2, it sounds like you have this running in Docker so bisection should be possible for you? Otherwise, if not, then validating the inputs to the final link command may be a good way to narrow down which file is broken. I am actually unaware of a tool that scans a (I would try the Docker instructions here but I don't have Docker installed myself.) |
|
Is wasm-ld producing any warnings about signature mismatches at link time by any chance? |
|
I was able to run/bisect builds on few more recent versions of emscripten via shell.cloud.google.com which is x86_64, which led me to a rabbit hole of more build failures and different kind of errors:
Not sure how this bisecting helped here as there is another different error between the originally reported 3.1.47 and 3.1.57. Considering the most recent reliable/working version of emscripten is ~6 months old (~8 for aarch64), @kripken the attached build script can be executed outside the docker, please go ahead and give it a go even without docker being used. @sbc100 please find attached build log from 3.1.54 (the first emscripten version failing on making parsable .wasm) to verify any suspicious wasm-ld warnings |
Thanks, I tried that now. I do get the broken final wasm file. I then ran a check on all the object files and they are all valid, so the breakage seems to happen by
I agree, the current situation is in need of improvement. I'm surprised we are seeing so many issues here, but it could be that ffmpeg has a particularly tricky build setup and stresses things we lack coverage for atm in our main test suite. And usually bisection is magical in these situations, but if there are multiple regressions, as you ran into unfortunately, then yeah, it is not that great. As for how we can improve things, in general there are two ways to go here:
Either of those (after we get things working again now) should give us an early warning about possible issues. |
|
My initial was that the linker was attempting the link functions with mismatched signatures. However, in that case the linker should normally give an error. I think bisecting would certainly help, but I'm a little busy right now preparing the CG meeting next week so I might not have time to get to this until after next week. |
|
Makes sense @sbc100 , no rush. In hopes of speeding things up when you get back, I bisected now. It turns out that bisecting just on the link command worked (once I added I started from the last emsdk tot which is I bisected that against 86 is a lot, but there are just a few WebAssembly commits there, all from you @sbc100, so hopefully it is one of these: Note that the bad releases commit I also dug into the broken wasm output. That only occurs in LTO builds, I believe because LTO builds are built without assertions, so they do not hit that |
|
Great job! So it sounds like a wasm-ld bug for sure. If you could package up a reproducer (using |
|
@sbc100 Sure, I opened llvm/llvm-project#94077 and I'll email you the big |
|
It seems that I've faced similar issue in the project I'm working on. Are there any hints on when this could be fixed? |
|
@kripken @sbc100 is my understanding correct, that this issue depends solely on llvm-project bug and emscripten will not be stable until the denpendency is fixed? |
|
Correct. I'm afraid I haven't had a change to look at the upstream bug yet. |
|
Do you know someone cooperative in llvm-project we can reach out to? |
|
I'm afraid its probably myself that is best placed to deal with it, but if you can find anyone else who might be able to take a look that would be awesome. |
|
Oh, is there a chance this could get to top of your priority list? Please |
|
I tracked this down to #78658 and I have a (one line!) fix in flight. just working on a test case for it. |
This was broken back in llvm#78658 when we transitioned away from archive indexes to parsing lazy object files. Fixes: llvm#94077 Fixes: emscripten-core/emscripten#22008
This was broken back in llvm#78658 when we transitioned away from archive indexes to parsing lazy object files. Fixes: llvm#94077 Fixes: emscripten-core/emscripten#22008
|
You are a hero @sbc100 . |
|
Yes indeed. You can also use |
I am having issues building ffmpeg after switching from emscripten 3.1.47 to 3.57.
With 3.1.47 I am able to build ffmpeg.wasm with aom and libpvx libraries, however with emscripten 3.1.57 I am fighting some errors.
When building ffmpeg with
-O0, the builds finishes ok, but there are runtime issues (in createModule() phase) when running .wasm in browser:Following that, it does appears both aom and libvpx defines the same static
decoder_get_framesymbol:But I do not think I will be able to convince aom / libvpx teams to rename all the conflicting symbols. Actually, I do not think these symbols are eventually needed or used in final .wasm.
When building with
-O3the build fails withrunning
wasm-opt --debugprints a huge log, ending with these lines:Linking only single aom or vpx at a time, there is no error even with 3.1.57, so my thinking is the problem is some kind of conflict, and 57 being somehow less forgiving than 47.
I am not sure how to proceed from here, I want to stick to the latest emscripten version however:
I wonder what is the breaking change between these versions (47 to 57) and if there is any magical build flag to use with emscripten so the build succeeds?
Both aom/libvpx are build with
-DBUILD_SHARED_LIBS=0or--disable-sharedrespectively, and linked to final .wasm using these flags:The text was updated successfully, but these errors were encountered: