mini_racer 0.12.0 Segmentation fault on x86_64-linux with libv8-node 21.7.2.0 #300
Comments
|
Hey @D-system. A couple of question hopefully helping to narrow the issue down
|
|
I don't think jemalloc is enabled: $ ruby -r rbconfig -e "puts RbConfig::CONFIG['MAINLIBS']"
-lz -lrt -lrt -ldl -lcrypt -lm -lpthreadEven by using the $ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 ruby -r rbconfig -e "puts RbConfig::CONFIG['MAINLIBS']"
-lz -lrt -lrt -ldl -lcrypt -lm -lpthread
# Checking the directory
$ ls -l /usr/lib/x86_64-linux-gnu/libjemalloc*
-rw-r--r-- 1 root root 1119814 Jan 24 2022 /usr/lib/x86_64-linux-gnu/libjemalloc.a
-rw-r--r-- 1 root root 1119970 Jan 24 2022 /usr/lib/x86_64-linux-gnu/libjemalloc_pic.a
lrwxrwxrwx 1 root root 16 Jan 24 2022 /usr/lib/x86_64-linux-gnu/libjemalloc.so -> libjemalloc.so.2
-rw-r--r-- 1 root root 744440 Jan 24 2022 /usr/lib/x86_64-linux-gnu/libjemalloc.so.2Let me see if I can make a new app and get the same error. I didn't realized there was a more debug output:
```
-- Ruby level backtrace information ---------------------------------------- -- Machine register context ------------------------------------------------ -- C level backtrace information ------------------------------------------- |
|
I made a brand new Rails 6.1.7.7 app I got the same error with those changes:
Note: it does precompile the files then seg fault.
```
$ bundle ; ./bin/rails assets:clobber assets:precompile
Bundle complete! 18 Gemfile dependencies, 84 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
I, [2024-04-23T18:59:24.147478 #10018] INFO -- : Removed /home/circleci/new_rails_app/public/assets
yarn install v1.22.19
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.42s.
I, [2024-04-23T18:59:25.739888 #10018] INFO -- : Writing /home/circleci/new_rails_app/public/assets/manifest-b4bf6e57a53c2bdb55b8998cc94cd00883793c1c37c5e5aea3ef6749b4f6d92b.js
I, [2024-04-23T18:59:25.740896 #10018] INFO -- : Writing /home/circleci/new_rails_app/public/assets/manifest-b4bf6e57a53c2bdb55b8998cc94cd00883793c1c37c5e5aea3ef6749b4f6d92b.js.gz
I, [2024-04-23T18:59:25.741133 #10018] INFO -- : Writing /home/circleci/new_rails_app/public/assets/application-04024382391bb910584145d8113cf35ef376b55d125bb4516cebeb14ce788597.css
I, [2024-04-23T18:59:25.742186 #10018] INFO -- : Writing /home/circleci/new_rails_app/public/assets/application-04024382391bb910584145d8113cf35ef376b55d125bb4516cebeb14ce788597.css.gz
./bin/rails: [BUG] Segmentation fault at 0x00007fb3f7901008
ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [x86_64-linux]
-- Control frame information ----------------------------------------------- -- Machine register context ------------------------------------------------ -- C level backtrace information ------------------------------------------- -- Other runtime information -----------------------------------------------
[...] |
|
I also tried the same project with mini_racer 0.9. It works. |
|
It seems sprocket is used by default if no other assets pipeline exists. The Gemfile without the comments source 'https://rubygems.org'
git_source(:github) { |repo| "https://github.com/#{repo}.git" }
ruby '3.2.3'
gem 'rails', '~> 6.1.7', '>= 6.1.7.7'
gem 'sqlite3', '~> 1.4'
gem 'puma', '~> 5.0'
gem 'sass-rails', '>= 6'
gem 'turbolinks', '~> 5'
gem 'jbuilder', '~> 2.7'
gem "bootstrap", "~> 4.3" # <- the problem
gem "mini_racer", "~> 0.9.0" # <- added
gem 'bootsnap', '>= 1.4.4', require: false
group :development, :test do
gem 'byebug', platforms: [:mri, :mingw, :x64_mingw]
end
group :development do
gem 'web-console', '>= 4.1.0'
gem 'rack-mini-profiler', '~> 2.0'
gem 'listen', '~> 3.3'
gem 'spring'
end
group :test do
gem 'capybara', '>= 3.26'
gem 'selenium-webdriver', '>= 4.0.0.rc1'
gem 'webdrivers'
end
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby] |
|
@lloeki this looks urgent , should we roll out a release that puts us back at an earlier version while we work this out ? |
|
@D-system does this repro in Linux as well |
|
@SamSaffron it's all tested inside the CircleCI image linked above. So, it's an Ubuntu based linux on x86_64. |
So the idea would be to have 0.12.1 be 0.9.0? That would be odd. |
|
I'll try a repro. |
|
The official |
|
I'm not able to reproduce yet, can't spend much time on it right now though. The product I'm working on in my day job is using docker images based on the 🤔 |
|
I tried with I'm not able to reproduce the issue in a local docker either. |
In any case it appears that
I couldn't reproduce it either inside that |
|
@D-system any chance you can create a container that repros this consistently. Then we can |
|
I just wanted to add another case of this issue. I tried to update yesterday from 0.8 to 0.12 and started getting segfaults. I don't use docker or any version managers, just the system's 3.0.2p107 (the default ruby) on Ubuntu 22.04. This is on rails 7.0.8.1 with libv8 21.7.2.0. I've reverted to 0.8 and libv8 18.16.0.0 for now. I'm not using jemalloc (I have no idea what it is, and the mentioned libraries are not on my system). If you need any info or need anything tested let me know, but I don't want to figure out how to build or use docker images (have no interest in that). |
|
I also saw it yesterday when upgrading from 0.9 to 0.12 both on Fedora 40 with ruby 3.3.0 and also on Ubuntu 22.04 with ruby 3.0.2p107. |
|
Thanks for the additional reports. Is it Sprockets-related as well? |
|
It was using rails with sprockets yes, specifically using https://rubygems.org/gems/rtlcss in the sprockets chain which uses mini_racer to run https://www.npmjs.com/package/rtlcss on CSS files. |
|
I am using sprockets, but I had not checked whether it's involved or not. I first got these errors when rendering a js response to an ajax request, and also saw them when shutting down the server (running puma). |
|
@SamSaffron I was not able to reproduce on my local host or local docker yesterday. I will give a new try tomorrow. |
|
We were getting this on 0.9.0 -> 0.12.0 (production only) on Heroku on the '20' image with Ruby 3.3.0 and jemalloc and node I did see this in a prod console, though may not be related
and looking at prod crash logs
we're just using this to run embedded javascript
|
Are you using puma in clustered mode? forking is known to cause some troubles with mini_racer/v8, see https://github.com/rubyjs/mini_racer#fork-safety. |
|
I am indeed using puma in clustered mode, but it was working fine on 0.8.0, and starting puma in single mode gives me the same segfault in the same place. I have the suggested fork safety line in an initializer, and removing this makes no difference in any scenario. |
|
I'm still not able to reproduce it in a local Docker even when I use the Docker image of the CI. I continue to investigate on the CI directly. SSH and vi for the win! Anyway, I track down the issue. On a brand new Rails 6.1 app:
I don't know what else I could search. |
|
@D-system I wish I had more to add here, but so much depends on a standalone repro. In theory most people use mini_racer in a very simple way, if you can get a crash then you can monkey patch eval to save what it is evaling into a file and then we can use that to cause the crash. Once we have one we will very likely be able to sort this out quickly There is a chance here as well that libraries were using mini racer in a fork unsafe way and somehow got away with it in earlier versions due to luck. |
|
The same happened to me after upgrading to the latest libv8-node (21.7.2.0) and mini_racer (0.12.0) during running specs
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux] |
|
After some lengthy monkey patching session, I'm 99% sure it's due to this file: https://github.com/ai/autoprefixer-rails/blob/master/vendor/autoprefixer.js The file is 6.29MB which isn't the simplest to debug. |
|
Oh wow, does an eval of this file cause the crash?
…On Wed, 1 May 2024 at 6:28 PM, Thomas ***@***.***> wrote:
After some lengthy monkey patching session, I'm 99% sure it's due to this
file:
https://github.com/ai/autoprefixer-rails/blob/master/vendor/autoprefixer.js
I removed the source map at the last line, it does not change anything.
The file is 6.29MB which isn't the simplest to debug.
—
Reply to this email directly, view it on GitHub
<#300 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAABIXPXZDI2KX724Z3GTFTZACRTFAVCNFSM6AAAAABGUEE3QSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBYGE2DQNZUHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
0.13.0 has been released. @D-system: If you have a moment, could you check if the new version with upgraded Node fixes the issue for you? |
|
@tisba I think you meant 0.13.0. |
|
oh no, and yes we just saw some of these too cc @lloeki 😢 will try debugging more this week |
|
@SamSaffron you can still reproduce with #300 (comment)? Trying to reproduce with your example on GH Actions, in hopes this helps. So far without success 😞 https://github.com/tisba/mini_racer-issue-300/actions/runs/10193149124/job/28197243732#step:4:5 |
|
Sadly #300 (comment) no longer crashes... I need a new consistent repro |
|
also #300 (comment) no longer crashes... So I have no repro left here :( |
Note @lloeki ... mini_racer no longer support any Ruby earlier than ruby 3.1. who knows maybe there are multiple issues here... For starters though we need a consistent repro. |
|
@D-system does #300 (comment) still reproduce the issue for you? |
|
@lloeki I am trying here, but this is really hard for me due to poor background. I can easily reproduce On my local Discourse instance, just need to cook some markdown into html in a particular sequence. Trouble is I am yet to make a standalone program that will crash it. Where I am super stuck now is figuring out how to get symbols going all the way through so I can get at least a full backtrace from v8 to have some clue of where v8 is crashing. Will keep trying, but as it stands given the instability of these releases I am tempted to just do a new release that pins the version back to the version of v8 that works. |
|
I released 0.14 which now reverts the v8 upgrades. I really want to be on latest v8 but we can not yet, we need a stable release out there and that trumps wanting an upgrade 😢 |
|
Interesting datapoint @lloeki Somewhere between 18.13 and 18.19 we broke test/test_forking.rb, it used to work fine, now it segfaults. |
|
Thanks folks for the unrelenting work in trying to reproduce! ❤️
By and large this doesn't matter, we could be building libv8 on a Rubyless image for all it cares. I plan to actually do that and move to something that makes having reproducible builds with better libc coverage (something like manylinux or even Nix).
Super sorry I cannot help as it absolutely refuses to crash on any of my machines, whatever I throw at it :( By the way the latest linux-gnu builds have been done with GCC.
Huh, I didn't see that one, been running tests locally before pushing and they all passed (except the "libv8 now optimises the hell out of things" ones which are red herrings)
At least this way:
|
|
Note: |
|
@lloeki do you get segfaults on local from test_forking? Still struggling through this, I know how amazing it will be to have a nice standalone test that segfaults, it is just very hard to pin down. |
|
Well, I just built Had to make some minor change to mini_racer: And now the test that was crashing with the forking is working perfectly. I enabled pointer compression and c++20 which is now required. I am not sure I am trusting this node build anymore for our purpose. Will see if I can double test this on build as well. I had some trouble building though cause finding the right toolchain for older builds is a bit more tricky... will see what I can do. We need to get |
|
@lloeki my branch here builds v8 from source Lines 31 to 39 in f134478 This build works just fine both with localization AND with the test_forking that was failing before. So I suspect something about the way this node build is working is ending up with a corrupt ICU setup which is leading to the various faults. |
|
OK another repro of the crash is a blank install of Discourse then: something about multithreading is causing segfaults and this is also happening on my builds |
V8's default thread-isolated allocator has a bug on x64 Linux. It uses memory protection keys (see `man 7 pkeys`) to write-protect JIT code memory but in a way that is currently incompatible with how we use threads. Specifically, pkey permissions are inherited by child threads. Threads that are not descendants of the thread that allocates the pkey default to "no permissions" for that pkey. Concretely, if thread A creates the v8::Platform (and the pkey) and write-protects memory, then later thread B tries to access that memory, it segfaults due to the lack of permissions. The fix on V8's side is conceptually easy - call pkey_set(PKEY_DISABLE_WRITE) before accessing the memory, to flip the permissions from "none" to "can read" - but until it's actually fixed, disable thread-isolation. Fixes: rubyjs/mini_racer#300 Refs: https://issues.chromium.org/issues/360909072
|
This is resolved, we disabled memory protection keys in v8 Underlying issue is tracked at: https://issues.chromium.org/issues/360909072 New version of mini_racer is now based on latest version of v8 that is shipped with node.js |
|
That's great to hear! The build on aarch alpine is broken and mini_racer won't compile there, but that's probably unrelated. Will investigate a little and open a new issue. |
|
We are now seeing different and new issues: It is going to take us a bit longer to get this v8 upgrade completed |
|
Sorry for the extreme late reply. Interesting to see the Chromium issue ticket mentioning this page. |
Hello the team
Thank you for the update to libv8-node 21.7.2.0 yesterday.
I wanted to report an error.
Setup:
RAILS_ENV=production SECRET_KEY_BASE=1 bin/rails assets:clobber assets:precompileIt does work on Darwin Kernel Version 23.2.0 arm64 (Mac on arm) that compile from source.
The error is from eval -> eval_unsafe, so I suspect it is an issue with the
libv8-nodemini_racer/lib/mini_racer.rb
Line 228 in 5b2a25c
The text was updated successfully, but these errors were encountered: