Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Completely remove REVERSE_DEPS / deps_info using new wasm-ld feature #18875

Closed
sbc100 opened this issue Mar 1, 2023 · 2 comments · Fixed by #18905
Closed

Completely remove REVERSE_DEPS / deps_info using new wasm-ld feature #18875

sbc100 opened this issue Mar 1, 2023 · 2 comments · Fixed by #18905

Comments

@sbc100
Copy link
Collaborator

sbc100 commented Mar 1, 2023

Today in emscripten we have a reverse dependency system that is made up of two main parts:

  1. Explicitly dependency info in deps_info.py and __deps entries in library files.
  2. Scanning of all inputs using llvm-nm do discover which deps are needed.

With #18849 having been merged we can now move the deps information out of deps_info.py and into the library files.

I was planning on land that next phase, #18865, which enables transitive dependencies, but unfortunately that cause a regression because the current error message produced by wasm-ld was less useful than the jsifier ones:

Current wasm-ld error for missing reverse deps:

wasm-ld: error: symbol exported via --export not found: foo

vs:

error: undefined symbol: bar (referenced by foo__deps: ['bar'], referenced by emscripten_runtime_keepalive_pop__deps: ['foo','$runtimeKeepalivePop'], referenced by top-level compiled C/C++ code)

I think if we can make the wasm-ld message more like this then we could finally switch over to wasm-ld for all symbol resolution:

wasm-ld: error: undefined symbols `foo` required by `bar` which was imported from `libemscripten.txt`

This will require a new wasm-ld input format, which is kind of like a dummy shared library, but is essentially just a text file representing external symbol and their dependencies. My plan is to use a format much like the current current format we use for our cached symbol lists. e.g.:

js_sym
js_sym_foo: malloc, free
@sbc100
Copy link
Collaborator Author

sbc100 commented Mar 1, 2023

I'm not sure why best to call this text file. We can think of it as a kind of linker script, or a dummy shared library.

@sbc100
Copy link
Collaborator Author

sbc100 commented Mar 5, 2023

@dschuff pointer out some prior art here in terms of stub objects: https://docs.oracle.com/cd/E36784_01/html/E36857/chapter2-22.html

I used that term in my wasm-ld implementation: https://reviews.llvm.org/D145308

sbc100 added a commit that referenced this issue Mar 5, 2023
… file. NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit to llvm/llvm-project that referenced this issue Mar 23, 2023
See the docs in lld/docs/WebAssembly.rst for more on this.

This feature unlocks a lot of simplification in the emscripten toolchain
since we can represent the JS libraries to wasm-ld as stub libraries.

See emscripten-core/emscripten#18875

Differential Revision: https://reviews.llvm.org/D145308
sbc100 added a commit that referenced this issue Mar 24, 2023
… file. NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Mar 24, 2023
… file. NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Mar 27, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 3, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 3, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 3, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 7, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 7, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 11, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 11, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 11, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 11, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 11, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 11, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 12, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 12, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 13, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 13, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 13, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 13, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 13, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 13, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 14, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 14, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 14, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 14, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 14, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 15, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 17, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 17, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 17, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 17, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 17, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 18, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 18, 2023
… NFC

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
sbc100 added a commit that referenced this issue Apr 18, 2023
…#18905)

This uses a new "stub object" construct to tell the linker (wasm-ld)
not only about the existence of the JS library symbols but the native
symbols on which they depend (a.k.a reverse dependencies).

This allows us to completely remove deps_info.py in favor of just using
normal `__deps` entries in the library files.  It also means we no
longer need to run `llvm-nm` on the linker inputs to discover the
symbols they use.

Depends on: https://reviews.llvm.org/D145308

Fixes: #18875
llvmbot pushed a commit to llvm/llvm-project-release-prs that referenced this issue May 31, 2023
See the docs in lld/docs/WebAssembly.rst for more on this.

This feature unlocks a lot of simplification in the emscripten toolchain
since we can represent the JS libraries to wasm-ld as stub libraries.

See emscripten-core/emscripten#18875

Differential Revision: https://reviews.llvm.org/D145308

(cherry picked from commit 3111784)
tstellar pushed a commit to llvm/llvm-project-release-prs that referenced this issue Jun 1, 2023
See the docs in lld/docs/WebAssembly.rst for more on this.

This feature unlocks a lot of simplification in the emscripten toolchain
since we can represent the JS libraries to wasm-ld as stub libraries.

See emscripten-core/emscripten#18875

Differential Revision: https://reviews.llvm.org/D145308

(cherry picked from commit 3111784)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant