feat(treesitter): add support for wasm parsers #28415
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lewis6991
force-pushed
the
tswasm
branch
2 times, most recently
from
7264715 to
be7cc1c
Compare
lewis6991
added
DO NOT MERGE
lewis6991
force-pushed
the
tswasm
branch
5 times, most recently
from
c2519a9 to
31f9b03
Compare
clason
reviewed
Apr 21, 2024
lewis6991
force-pushed
the
tswasm
branch
2 times, most recently
from
7b627ac to
ca9e93d
Compare
dundargoc
reviewed
Apr 22, 2024
lewis6991
commented
Apr 22, 2024
dundargoc
reviewed
Apr 22, 2024
dundargoc
reviewed
Apr 22, 2024
dundargoc
reviewed
Apr 22, 2024
dundargoc
reviewed
Apr 22, 2024
lewis6991
force-pushed
the
tswasm
branch
3 times, most recently
from
9f77fb6 to
2ecad21
Compare
|
Is this something we want to add to |
clason
force-pushed
the
tswasm
branch
2 times, most recently
from
cbfa704 to
5ccf96b
Compare
|
Note: Investigate how wasm parsers and native parsers for the same language interact (i.e., trying to (un)load a wasm parser after loading a native or vice versa). |
clason
force-pushed
the
tswasm
branch
3 times, most recently
from
2d414a3 to
3c30933
Compare
|
Note 2: Look at using sccache instead of EDIT Seems to make a noticeable difference: cold build from
Tested with |
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
clason
force-pushed
the
tswasm
branch
3 times, most recently
from
6c00a4c to
8799870
Compare
Problem: Installing treesitter parser is hard (harder than climbing to heaven). Solution: Add optional support for wasm parsers with `wasmtime`. Notes: * Needs to be enabled by setting `ENABLE_WASMTIME` for tree-sitter and Neovim. Build with `make CMAKE_EXTRA_FLAGS=-DENABLE_WASMTIME=ON DEPS_CMAKE_FLAGS=-DENABLE_WASMTIME=ON` * Adds optional Rust (obviously) and C11 dependencies. * Wasmtime comes with a lot of features that can negatively affect Neovim performance due to library and symbol table size. Make sure to build with minimal features and full LTO. * To reduce re-compilation times, install `sccache` and build with `RUSTC_WRAPPER=<path/to/sccache> make ...`
Does that mean it's possible to actually unload parsers now? I thought that wasn't really supported yet.
see #22313 |
This comment has been minimized.
This comment has been minimized.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Treesitter parsers are currently distributed as compiled shared objects which need to be compiled by the users (with the help of nvim-treesitter).
Solution
Allow loading wasm built parsers.
Notes:
ENABLE_WASMTIME) which will be dynamically linked.make CMAKE_EXTRA_FLAGS=-DENABLE_WASMTIME=ON DEPS_CMAKE_FLAGS=-DENABLE_WASMTIME=ONTREE_SITTER_FEATURE_WASMand also requireswasmtimeat build time. So no[uv_]dlopen😢wasmtimehas includes that require C11 (static_assert).sccacheand build withRUSTC_WRAPPER=sccache make ....Status
Waiting for the next tree-sitter release with the commit we're pulling in here to allow us to build against a wasmtime release with the build changes required to avoid unacceptable regressions on macOS. (Ideally, that release would also be built against the same version we use, but that is blocked by the same build system changes; luckily so far the versions have not become incompatible for our purpose.)