Fatal memory error with large response

[mtoothman]

1. Newman Version (can be found via newman -v):
   4.1.0

2. OS details (type, version, and architecture):
   Alpine Linux 3.7.0, in docker container

3. Are you using Newman as a library, or via the CLI?
   CLI

4. Did you encounter this recently, or has this bug always been there:
   Recently

5. Expected behaviour:
   No timeouts during long collection or test script execution when options are omitted.
   No out-of-memory errors

6. Command / script used to run Newman:
   newman run ./collections/$COLLECTION_NAME.json -e ./env/$ENV_NAME.json

7. Sample collection, and auxiliary files (minus the sensitive details):

8. Screenshots (if applicable):

Steps to reproduce the problem:

1. Run attached collection (JSON, saved with bogus .txt extension). Randomly selected large JSON file from the internet.

→ get_all_episodes
  GET https://raw.githubusercontent.com/zemirco/sf-city-lots-json/master/citylots.json [200 OK, 21.79MB, 19.4s]

<--- Last few GCs --->

[1:0x55698c4e2000]    28027 ms: Mark-sweep 655.2 (666.6) -> 439.2 (450.6) MB, 280.1 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 3337 ms) allocation failure GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x4c364725ee1 <JSObject>
    1: toJSON [buffer.js:~938] [pc=0x15daddbc657d](this=0x2d8a4dcf68d1 <Uint8Array map = 0xee6886417b9>)
    3: /* anonymous */ [/usr/lib/node_modules/newman/node_modules/postman-collection/lib/collection/property-base.js:133] [bytecode=0x17720956b749 offset=100](this=0x3b2e4f48bde9 <JSGlobal Object>,accumulator=0x2d8a4dc84931 <Object map = 0x21f9f4330a39>,value=0x2d8a4dcf68d1 <Uint8Array map = 0...

FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory

episode-editor.json.txt

I noticed this problem initially when a job was failing with Script execution timed out, but adjusting timeout settings and upgrading to 4.10.0 to use its default Infinite setting didn't help.

[kunagpal]

@mtoothman As we can see from your shared CLI snippet, the 21.79MB response is not being handled all that well. As a workaround, could you try the following command and confirm that the collection runs correctly?

NODE_OPTIONS="--max-old-space-size=2048" newman run ./collections/$COLLECTION_NAME.json -e ./env/$ENV_NAME.json

Note that you may have to adjust the value of --max-old-space-size, which is in megabytes.

A proper long-term fix is to allow ignoring responses from the underlying collection run layer.

Similar to #1516.

[mtoothman]

Running this command with --max-old-space-size set to 2048 or above results in
FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory
(the same error as originally posted).

Using a smaller value results in

  $ NODE_OPTIONS="--max-old-space-size=1024" newman run collections/testfoo.json
newman

testfoo

→ testfoo
  GET https://raw.githubusercontent.com/zemirco/sf-city-lots-json/master/citylots.json [200 OK, 21.79MB, 1951ms]

<--- Last few GCs --->

[72738:0x102802400]     7133 ms: Mark-sweep 439.3 (479.1) -> 439.3 (479.1) MB, 135.4 / 0.0 ms  allocation failure GC in old space requested
[72738:0x102802400]     7270 ms: Mark-sweep 439.3 (479.1) -> 439.3 (445.6) MB, 137.4 / 0.0 ms  last resort GC in old space requested
[72738:0x102802400]     7404 ms: Mark-sweep 439.3 (445.6) -> 439.3 (445.6) MB, 133.9 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x387ed3525ec1 <JSObject>
    1: toJSON [buffer.js:~933] [pc=0x2658ff4553d](this=0x387ed43ffb39 <Uint8Array map = 0x387edfcc1c31>)
    3: /* anonymous */ [/usr/local/lib/node_modules/newman/node_modules/postman-collection/lib/collection/property-base.js:133] [bytecode=0x387e4ff89309 offset=100](this=0x387e3180c0b9 <JSGlobal Object>,accumulator=0x387e1720ce71 <Object map = 0x387e3efdcae9>,value=0x387ed43ffb39 <Uint8Array ...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

[kunagpal]

@mtoothman Closer inspection of your snippet reveals the following:

Security context: 0x387ed3525ec1 <JSObject>
    1: toJSON [buffer.js:~933] [pc=0x2658ff4553d](this=0x387ed43ffb39 <Uint8Array map = 0x387edfcc1c31>)
    3: /* anonymous */ [/usr/local/lib/node_modules/newman/node_modules/postman-collection/lib/collection/property-base.js:133] [bytecode=0x387e4ff89309 offset=100](this=0x387e3180c0b9 <JSGlobal Object>,accumulator=0x387e1720ce71 <Object map = 0x387e3efdcae9>,value=0x387ed43ffb39 <Uint8Array ...

1. This is happening when the large raw response is being processed for easier handling.
2. This isn't an issue with the overall available memory, it's got more to do with the size limits imposed on individual objects by Node itself. A similar problem has been discussed here: invalid array buffer length nodejs/node#3543

While we think of a better way to handle this problem, I've marked this issue as a bug.

[shamasis]

Node v9 and above increases these limits. Please try this out and let us know.

[DieGraueEminenz]

@shamasis The mentioned problem is still reproducable, even with node v11
Do you have any other suggestions to us, to solve this unfortunate issue?

[asi-postman]

Another customer has experienced a similar issue. They have tried the following and the problem still persists. I have attached a .txt of the error output as well.

newman version: 4.5.6
Node version: v12.13.1

[NODE_OPTIONS="--max-old-space-size=2048"](url
[error.txt](https://github.com/postmanlabs/newman/files/3913004/error.txt)
) newman run ./collections/$COLLECTION_NAME.json -e ./env/$ENV_NAME.json

Set NODE_OPTIONS first
SET NODE_OPTIONS="--max-old-space-size=2048"

Run newman next
newman run <collection_filenam> -e <environment_filename>

[codenirvana]

We've made significant performance improvements to handle large response payloads.
Is this issue still reproducible in the latest version of Newman?

[codenirvana]

Closing this issue as we are not able to reproduce this internally and we haven't seen other users facing the same.

Please feel free to reopen if the issue persists or you can help us with the steps to reproduce this issue.

[sahanakrishnaiyer]

I am seeing similar issue running collection using newman:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: node::Abort() [node] 2: 0x7f64f33ba011 [node] 3: v8::Utils::ReportOOMFailure(char const*, bool) [node] 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node] 5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [node] 6: v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [node] 7: v8::String::WriteUtf8(char*, int, int*, int) const [node] 8: node::StringBytes::Write(v8::Isolate*, char*, unsigned long, v8::Local<v8::Value>, node::encoding, int*) [node] 9: node::Buffer::New(v8::Isolate*, v8::Local<v8::String>, node::encoding) [node] 10: 0x7f64f33dfe46 [node] 11: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [node] 12: 0x7f64f2d38b62 [node] 13: 0x7f64f2d390af [node] 14: 0x1439df7040bd Aborted (core dumped)

[toolmania1]

Using Newman 4.6.1 or Newman 4.5.5 ( We don't have Node > 10 yet )

<--- Last few GCs --->

n 2643 steps since start of marking, biggest step 9.7 ms, walltime since start of marking 1759 ms) finalize incremental marking via stack guard GC i[13346:0x31c3840] 38894 ms: Mark-sweep 1046.6 (1081.3) -> 551.2 (564.9) MB, 3.5 / 0.0 ms (+ 40.3 ms in 8 steps since start of marking, biggest step 11.2 ms, walltime since start of marking 12114 ms) finalize incremental marking via stack guard GC in o[13346:0x31c3840] 38926 ms: Scavenge 551.9 (564.9) -> 551.4 (564.9) MB, 1.3 / 0.0 ms allocation failure

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x35e132d25729

1: utf8Slice(aka utf8Slice) [evalmachine.<anonymous>:~13632] [pc=0xd135aa36432](this=0x265a68d822d1 <undefined>,e=0x1a4820c02201 <Uint8Array map = 0x3b2c7547901>,t=0,r=129301023)

2: slowToString [evalmachine.<anonymous>:13632] [bytecode=0x89ae9f85931 offset=207](this=0x1a4820c02201 <Uint8Array map = 0x3b2c7547901>,e=0x35e132d34749 <String[4]: utf8>,t=0,r=129301023)

3: arguments adapt...

FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory

1: node::Abort() [node]

2: 0x11dd81c [node]

3: v8::Utils::ReportOOMFailure(char const*, bool) [node]

4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]

5: v8::internal::Heap::AllocateRawFixedArray(int, v8::internal::PretenureFlag) [node]

6: v8::internal::Heap::AllocateFixedArrayWithFiller(int, v8::internal::PretenureFlag, v8::internal::Object*) [node]

7: v8::internal::Factory::NewUninitializedFixedArray(int) [node]

8: 0xde8473 [node]

9: v8::internal::Runtime_GrowArrayElements(int, v8::internal::Object**, v8::internal::Isolate*) [node]

10: 0xd135a8842fd

Aborted (core dumped)

script returned exit code 134

[peter-dolkens]

Issue is happening for us too - happens even before we attempt to run any tests

  POST https://server.com/graphql [200 OK, 131.89MB, 29.3s]

<--- Last few GCs --->

[13168:00000234DA416A90]    96702 ms: Mark-sweep 1706.1 (1742.2) -> 1438.3 (1474.2) MB, 125.5 / 0.0 ms  (+ 19.9 ms in 5 steps since start of marking, biggest step 11.9 ms, walltime since start of marking 1160 ms) (average mu = 0.954, current mu = 0.929) a[13168:00000234DA416A90]    97422 ms: Mark-sweep 2012.1 (2048.0) -> 965.1 (998.7) MB, 193.1 / 0.0 ms  (average mu = 0.888, current mu = 0.732) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory
 1: 00007FF7210EA41F v8::internal::wasm::DisjointAllocationPool::~DisjointAllocationPool+74447
 2: 00007FF721092F26 v8::base::CPU::has_sse+59718
 3: 00007FF721093CF6 v8::base::CPU::has_sse+63254
 4: 00007FF7218F837E v8::Isolate::ReportExternalAllocationLimitReached+94
 5: 00007FF7218DD544 v8::SharedArrayBuffer::Externalize+772
 6: 00007FF72179C7FC v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1452
 7: 00007FF7217C41CE v8::internal::Factory::NewUninitializedFixedArray+110
 8: 00007FF72168CFD0 v8::debug::Script::GetIsolate+8112
 9: 00007FF721525CD7 v8::internal::interpreter::JumpTableTargetOffsets::iterator::operator=+167175
10: 00007FF72197FC2D v8::internal::SetupIsolateDelegate::SetupHeap+465869
11: 000001999262A4E1

[swedkg]

This issue still exists. This is a VM with 10GB of ram with NODE_OPTIONS="--max-old-space-size=8192". The response is a file of 137.98MB. The file is downloaded, but then we get the fatal memory error. Smaller responses are fine.

$ newman -v
5.2.2

$ node -v
v14.16.0

<--- Last few GCs --->

[7232:000000BAE331C240]    93192 ms: Scavenge 1163.6 (1200.8) -> 1163.6 (1200.8) MB, 105.5 / 0.0 ms  (average mu = 0.985, current mu = 0.981) allocation failure
[7232:000000BAE331C240]    94383 ms: Scavenge 1546.2 (1583.3) -> 1546.2 (1583.3) MB, 161.4 / 0.0 ms  (average mu = 0.985, current mu = 0.981) allocation failure
[7232:000000BAE331C240]    96250 ms: Scavenge 2120.0 (2157.1) -> 2120.0 (2157.1) MB, 243.3 / 0.0 ms  (average mu = 0.985, current mu = 0.981) allocation failure

<--- JS stacktrace --->

FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory
 1: 00007FF6E461052F napi_wrap+109311
 2: 00007FF6E45B5256 v8::internal::OrderedHashTable<v8::internal::OrderedHashSet,1>::NumberOfElementsOffset+33302
 3: 00007FF6E45B6026 node::OnFatalError+294
 4: 00007FF6E4E8163E v8::Isolate::ReportExternalAllocationLimitReached+94
 5: 00007FF6E4E664BD v8::SharedArrayBuffer::Externalize+781
 6: 00007FF6E4D1094C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1516
 7: 00007FF6E4D3547F v8::internal::Factory::NewUninitializedFixedArray+111
 8: 00007FF6E4BFB3C0 v8::Object::GetIsolate+8128
 9: 00007FF6E4A851F7 v8::internal::interpreter::JumpTableTargetOffsets::iterator::operator=+169671
10: 00007FF6E4F09FED v8::internal::SetupIsolateDelegate::SetupHeap+463949
11: 000003D543A02C98

[matuszabojnik]

Same issue here. 64 bit VM with 32GB available ram, NODE_OPTIONS="--max-old-space-size=8192".
$ newman -v
5.2.3

$ node -v
v10.24.0

Responses below 107.55MB are passing fine,
Responses above 107.69MB are failing.

<--- Last few GCs --->

[23820:0x238b920]    89082 ms: Mark-sweep 963.0 (1014.2) -> 525.7 (579.9) MB, 19.7 / 0.0 ms  (+ 17.5 ms in 4 steps since start of marking, biggest step 13.9 ms, walltime since start of marking 2307 ms) (average mu = 0.982, current mu = 0.989) finalize inc
[23820:0x238b920]    93302 ms: Mark-sweep 526.2 (579.9) -> 525.6 (578.9) MB, 12.3 / 0.0 ms  (+ 17.4 ms in 5 steps since start of marking, biggest step 9.3 ms, walltime since start of marking 3082 ms) (average mu = 0.989, current mu = 0.993) finalize incre

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x580db1d452b]
Security context: 0x395feb22ede9 <JSObject>
    1: E(aka E) [0x2d7f3cd8f9e1] [evalmachine.<anonymous>:~58] [pc=0x580db3c11df](this=0x1385aff025d9 <undefined>,e=0x3f7e3e787bd1 <Uint8Array map = 0x1be21f4639f1>,t=0,n=114353052)
    2: d [0x2d7f3cd8f761] [evalmachine.<anonymous>:58] [bytecode=0x8301cc5b659 offset=204](this=0x3f7e3e787bd1 <Uint8Array map = 0x1be21f4639f1>,e=0x395feb272a29 <String[4]: utf8>...

FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory
 1: 0x7f1fd66402d8 node::Abort() [/usr/lib/x86_64-linux-gnu/libnode.so.64]
 2: 0x7f1fd6640321  [/usr/lib/x86_64-linux-gnu/libnode.so.64]
 3: 0x7f1fd68215c2 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/lib/x86_64-linux-gnu/libnode.so.64]
 4: 0x7f1fd6821818 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/lib/x86_64-linux-gnu/libnode.so.64]
 5: 0x7f1fd6bb0542  [/usr/lib/x86_64-linux-gnu/libnode.so.64]
 6: 0x7f1fd6b92863 v8::internal::Factory::AllocateRawFixedArray(int, v8::internal::PretenureFlag) [/usr/lib/x86_64-linux-gnu/libnode.so.64]
 7: 0x7f1fd6b92c34 v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::Heap::RootListIndex, int, v8::internal::Object*, v8::internal::PretenureFlag) [/usr/lib/x86_64-linux-gnu/libnode.so.64]
 8: 0x7f1fd6b595fd  [/usr/lib/x86_64-linux-gnu/libnode.so.64]
 9: 0x7f1fd6db793c v8::internal::Runtime_GrowArrayElements(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/lib/x86_64-linux-gnu/libnode.so.64]
10: 0x580db1d452b 

[Seayon]

same issue here

[SandhyaJha]

newman -v
5.3.2
node -v
v15.13.0

Setting node options to 800 MB
SET NODE_OPTIONS="--max-old-space-size=800"

Get api call which download a file of size more than 700MB doesn't work.

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory