jemalloc 5.1.0 causing problems in heroku deploys release phase #3

concept47 · 2018-08-23T10:42:00Z

Since the change of version of jemalloc, we've started getting failures in the release phase of our heroku deploys that look like this

  rake aborted!
  JSON::ParserError: 765: unexpected token at 'node[29]: pthread_create: Invalid argument
  ["ok"]'
  /app/vendor/bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs/external_runtime.rb:68:in `extract_result'
  /app/vendor/bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs/external_runtime.rb:39:in `exec'
  /app/vendor/bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs/external_runtime.rb:14:in `initialize'
  /app/vendor/bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs/runtime.rb:57:in `new'
  /app/vendor/bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs/runtime.rb:57:in `compile'
  /app/vendor/bundle/ruby/2.5.0/gems/execjs-2.7.0/lib/execjs/module.rb:27:in `compile'
  /app/vendor/bundle/ruby/2.5.0/gems/uglifier-4.1.10/lib/uglifier.rb:154:in `initialize'
  /app/config/environments/staging.rb:21:in `new'
  /app/config/environments/staging.rb:21:in `block in <main>'
  /app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/railtie.rb:209:in `instance_eval'
  /app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/railtie.rb:209:in `configure'
  /app/config/environments/staging.rb:1:in `<main>'
  /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:19:in `require'
  /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:19:in `block in require_with_bootsnap_lfi'
  /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
  /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:18:in `require_with_bootsnap_lfi'
  /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:27:in `require'
  /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:293:in `block in require'
  /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:259:in `load_dependency'
  /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:293:in `require'
  /app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/engine.rb:600:in `block (2 levels) in <class:Engine>'
  /app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/engine.rb:599:in `each'
  /app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/engine.rb:599:in `block in <class:Engine>'
  /app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/initializable.rb:30:in `instance_exec'
  /app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/initializable.rb:30:in `run'
  /app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/initializable.rb:55:in `block in run_initializers'
  /app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/initializable.rb:44:in `each'
  /app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/initializable.rb:44:in `tsort_each_child'
  /app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/initializable.rb:54:in `run_initializers'
  /app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/application.rb:352:in `initialize!'
  /app/config/environment.rb:5:in `<main>'
  /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:19:in `require'
  /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:19:in `block in require_with_bootsnap_lfi'
  /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
  /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:18:in `require_with_bootsnap_lfi'
  /app/vendor/bundle/ruby/2.5.0/gems/bootsnap-1.2.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:27:in `require'
  /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:293:in `block in require'
  /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:259:in `load_dependency'
  /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.0.6/lib/active_support/dependencies.rb:293:in `require'
  /app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/application.rb:328:in `require_environment!'
  /app/vendor/bundle/ruby/2.5.0/gems/railties-5.0.6/lib/rails/application.rb:448:in `block in run_tasks_blocks'
  /app/vendor/bundle/ruby/2.5.0/gems/bugsnag-5.0.1/lib/bugsnag/rake.rb:12:in `execute_with_bugsnag'
  Tasks: TOP => db:migrate => environment
  (See full trace by running task with --trace)

The line in question in staging.rb that seems to fail is this one

     config.assets.js_compressor = Uglifier.new(harmony: true)

All our deploys with the jemalloc buildpack removed work just fine. (Figured this because our review apps don't use jemalloc, but it seemed that we had to run

heroku plugins:install heroku-repo
heroku repo:purge_cache

To get it to work.

The text was updated successfully, but these errors were encountered:

concept47 · 2018-08-23T20:35:33Z

jemalloc/jemalloc#1313 (comment)

alainravet · 2018-08-27T09:19:01Z

For those who can no longer deploy because of this issue: revert to the previous version

Fork this repo and revert to jemallock 5.0.1

Fork this repo
$ git clone <your fork repo> ; cd heroku-buildpack-jemalloc
$ git reset --hard 08f77c0
$ git push -f

Use your modified buildpack

open https://dashboard.heroku.com/apps/<your app>/settings ;
scroll down to 'Buildbacks'
deletehttps://github.com/gaffneyc/heroku-buildpack-jemalloc
add https://github.com/<your-account>/heroku-buildpack-jemalloc

gaffneyc · 2018-09-04T20:25:26Z

You should be able to set JEMALLOC_VERSION=5.0.1 instead of having to fork the repo. I'm sorry you ran into issues, we had an open issue for about two months before upgrading... of course that requires people to be testing the new version against different environments to find anything. I also missed this since I apparently wasn't watching my own repo 🤦‍♂️

From your stack trace, the problem is actually coming from NodeJS and not Ruby. There was a note in #2 about NodeJS printing pthread_create: Invalid argument but otherwise working fine. My guess is that execjs is forking a NodeJS process then reading the output from the command and parsing the output as JSON. The problem is that the warning is breaking the parsing.

@concept47 If you're still in a position to, could you try setting JEMALLOC_VERSION to see if that fixes the issue?

concept47 · 2018-09-04T20:46:45Z

It worked a treat @gaffneyc. Thanks!

gaffneyc · 2018-09-05T18:14:13Z

I'm considering reverting the change to make 5.1.0 the default. If you've run into this then 👍 this comment and I'll take that into consideration. I hate to move backwards but I also don't want to break things for people.

jvdp · 2018-11-19T16:54:18Z

@gaffneyc I get a problem after release (running migrations) with jemalloc enabled, even with 5.0.1, on the heroku-16 stack. I'll experiment with other versions.

gaffneyc · 2018-11-19T17:02:19Z

@jvdp Can you give some specifics on the problem you're seeing? For example: are you getting a stack trace, memory usage is going up, or response times are worse, etc...

jvdp · 2018-11-19T17:24:29Z

@gaffneyc the same exception as OP pretty much. This is with ruby 2.5.3.

(It worked well with Heroku-18 but it turns out that breaks wkhtmltopdf.)

gaffneyc · 2018-11-19T17:38:06Z

@jvdp The issue itself is with NodeJS (being called through ExecJS). I haven't had a chance to dig into the problem with Node but there are two things worth experimenting with. If you're seeing it on 5.0.1 then that's a new issue we haven't seen before (make sure you're pushing a code change as just changing JEMALLOC_VERSION isn't enough since jemalloc is baked into the slug).

Upgrade NodeJS
The default version for Rails applications on Heroku is 8.10.0 it's possible the issue has been fixed in a newer version of Node. You might be able to use therubyracer or mini_racer to get a newer version of node for testing. Otherwise you'll probably want to add the nodejs buildpack and use it to configure the node version.
Use jemalloc.sh
Use the per dyno config for jemalloc where you disable it globally (JEMALLOC_ENABLED=false) then add jemalloc.sh at the beginning of your web and (maybe) worker lines. This will disable jemalloc when running migrations but use it to run the application.

jvdp · 2018-11-20T12:39:30Z

@gaffneyc thanks for the pointers! I think I wasn't actually running 5.0.1, I've retried with a code change after setting the version and now it seems to work OK!

adamlogic · 2018-11-21T17:34:14Z

The default version for Rails applications on Heroku is 8.10.0 it's possible the issue has been fixed in a newer version of Node. You might be able to use therubyracer or mini_racer to get a newer version of node for testing. Otherwise you'll probably want to add the nodejs buildpack and use it to configure the node version.

Just heads-up that I ran into this same issue using the nodejs buildpack and Node 9.5.0. JEMALLOC_VERSION=5.0.1 fixed it. 🤷‍♂️

Also, I got tripped up that simply setting JEMALLOC_VERSION and triggering a release is not sufficient. You must trigger a full build on your app to pull in the updated version. This made sense once I peeked at the buildpack code and understood how it worked, but I'm still confused because changing JEMALLOC_ENABLED does seem to have an immediate impact without running a full build.

gaffneyc · 2019-05-07T19:21:28Z

Has any one run into this issue recently? I tried to reproduce it a couple months ago and wasn't able to. Also, several things have changed in the months since the issue was first opened:

Ruby apps now default to Node 10.14.1
I've rebuilt the jemalloc 5.1.0 builds on Linux (instead of MacOS) with updated build containers from Heroku
Jemalloc 5.2.0 has been released

Would one of you that has run into the issue be willing to try out Jemalloc 5.1.0 again? Or even kick the tires on jemalloc 5.2.0?

ianwremmel · 2019-06-17T20:30:37Z

@gaffneyc I just ran into it with the default version today. Setting 5.0.1 fixed it.

  "engines": {
    "node": "10.13.0",
    "npm": "6.4.1",
    "yarn": "1.15.2"
  },

 "stack": "heroku-16",
  "buildpacks": [
    {
      "url": "https://github.com/gaffneyc/heroku-buildpack-jemalloc.git"
    },
    {
      "url": "heroku/nodejs"
    },
    {
      "url": "heroku/ruby"
    },
    {
      "url": "heroku-community/awscli"
    },
    {
      "url": "heroku-community/cli"
    }
  ]

This was the line in our code that threw, if that helps:

config.assets.js_compressor = Uglifier.new(harmony: true)

gaffneyc · 2019-06-17T20:33:28Z

@ianwremmel Would you be able to give jemalloc 5.2.0 a try? It's looking like it might be an issue with the Heroku 16 stack

ianwremmel · 2019-06-17T20:38:24Z

trying it now

ianwremmel · 2019-06-17T21:45:58Z

same problem with 5.2.0

gaffneyc · 2019-06-18T13:36:51Z

@ianwremmel Awesome, thank you for testing it!

gaffneyc · 2019-06-23T15:57:06Z

I've been able to reproduce the issue and I've created a minimal test repository for it if anyone wants to try to solve it.

I've done some initial digging and it appears the issue is fixed on the heroku-18 stack or when using NodeJS >= 12.4.0 on heroku-16 (or cedar-14).

	cedar-14	heroku-16	heroku-18
Node 10.13.0	❌	❌	✅
Node 10.14.1 (Rails Default)	❌	❌	✅
Node 10.16.0	❌	❌	✅
Node 12.0.0	❌	❌	✅
Node 12.3.1	❌	❌	✅
Node 12.4.0	✅	✅	✅

I've run a bisect on jemalloc and this appears to be the commit that introduced the issue:
jemalloc/jemalloc@9c05490

I am going to experiment with configure options to see if we can get builds working for cedar-14 and heroku-16. Since the problem will effectively go away as people move to Heroku 18 or newer versions of Node I'm not sure what else could be done.

Changes between 12.3.1 and 12.4.0

ianwremmel · 2019-06-24T20:06:23Z

@gaffneyc thanks for the update! We're looking to move to Heroku-18 in the coming weeks, so I think simply running on the older version is perfectly fine for us for the time being.

gaffneyc · 2019-10-02T20:00:35Z

After more research it looks like it was fixed in 12.4.0 with this commit and is triggered by thread stack size. Annoyingly, NodeJS is printing the message that is breaking Uglifier.

This is looking to be an issue in NodeJS (which has been fixed) that happens to be triggered by something in jemalloc. I've tested a number of build configs with jemalloc and none of them have addressed the issue.

Unfortunately I think the "fix" is to document the issue in the README and recommend folks use one of: Heroku 18, NodeJS 12.4.0+, or jemalloc 5.0.1 (in that order).

gaffneyc · 2020-06-23T18:58:52Z

Heroku should be updating Ruby apps to use Node 12.16.2 by default sometime soon which should close this out once and for all.

Initial release yesterday (June 22nd) that was rolled back: https://devcenter.herokuapp.com/changelog-items/1818

gaffneyc · 2020-07-14T18:23:20Z

Heroku finally rolled out the upgrade to Node 12.16.2 for Rails apps for all of their stacks. That should mean this is finally fixed for good (unless of course you're pinned to an old version of NodeJS with the NodeJS buildpack).

https://devcenter.herokuapp.com/changelog-items/1825

concept47 changed the title ~~Jemalloc 5.1.0 causing problems in heroku deploys release phase~~ jemalloc 5.1.0 causing problems in heroku deploys release phase Aug 23, 2018

ncri mentioned this issue Oct 17, 2018

How to validate buildpack is having an effect #5

Closed

gaffneyc mentioned this issue Oct 7, 2019

Make 5.2.0 the default version Shopify/heroku-buildpack-jemalloc#1

Merged

gaffneyc closed this as completed Jul 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

jemalloc 5.1.0 causing problems in heroku deploys release phase #3

jemalloc 5.1.0 causing problems in heroku deploys release phase #3

concept47 commented Aug 23, 2018

concept47 commented Aug 23, 2018

alainravet commented Aug 27, 2018

gaffneyc commented Sep 4, 2018 •

edited

Loading

concept47 commented Sep 4, 2018

gaffneyc commented Sep 5, 2018

jvdp commented Nov 19, 2018

gaffneyc commented Nov 19, 2018 •

edited

Loading

jvdp commented Nov 19, 2018

gaffneyc commented Nov 19, 2018

jvdp commented Nov 20, 2018

adamlogic commented Nov 21, 2018 •

edited

Loading

gaffneyc commented May 7, 2019 •

edited

Loading

ianwremmel commented Jun 17, 2019

gaffneyc commented Jun 17, 2019

ianwremmel commented Jun 17, 2019

ianwremmel commented Jun 17, 2019

gaffneyc commented Jun 18, 2019

gaffneyc commented Jun 23, 2019 •

edited

Loading

ianwremmel commented Jun 24, 2019

gaffneyc commented Oct 2, 2019 •

edited

Loading

gaffneyc commented Jun 23, 2020

gaffneyc commented Jul 14, 2020

jemalloc 5.1.0 causing problems in heroku deploys release phase #3

jemalloc 5.1.0 causing problems in heroku deploys release phase #3

Comments

concept47 commented Aug 23, 2018

concept47 commented Aug 23, 2018

alainravet commented Aug 27, 2018

gaffneyc commented Sep 4, 2018 • edited Loading

concept47 commented Sep 4, 2018

gaffneyc commented Sep 5, 2018

jvdp commented Nov 19, 2018

gaffneyc commented Nov 19, 2018 • edited Loading

jvdp commented Nov 19, 2018

gaffneyc commented Nov 19, 2018

jvdp commented Nov 20, 2018

adamlogic commented Nov 21, 2018 • edited Loading

gaffneyc commented May 7, 2019 • edited Loading

ianwremmel commented Jun 17, 2019

gaffneyc commented Jun 17, 2019

ianwremmel commented Jun 17, 2019

ianwremmel commented Jun 17, 2019

gaffneyc commented Jun 18, 2019

gaffneyc commented Jun 23, 2019 • edited Loading

ianwremmel commented Jun 24, 2019

gaffneyc commented Oct 2, 2019 • edited Loading

gaffneyc commented Jun 23, 2020

gaffneyc commented Jul 14, 2020

gaffneyc commented Sep 4, 2018 •

edited

Loading

gaffneyc commented Nov 19, 2018 •

edited

Loading

adamlogic commented Nov 21, 2018 •

edited

Loading

gaffneyc commented May 7, 2019 •

edited

Loading

gaffneyc commented Jun 23, 2019 •

edited

Loading

gaffneyc commented Oct 2, 2019 •

edited

Loading