Newman CLI fatal error on large tasks

[Postman-Gabriel]

Describe the bug
When using Postman CLI integration with Newman, running many tasks at once over a long period of time causes a fatal error to occur where the Heap overflows.

My user is using running 50 jobs at once over 24 hours to look for something, which is causing the crash

To Reproduce
Steps to reproduce the behavior:

1. Look for information in a large data set over a long time
2. Do many jobs at once
3. See error

Expected behavior
Expected to be able to use Postman CLI with Newman to perform high-intensity tasks in a professional environment

Screenshots

Console log:

FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x10a0f5c [node]
3: v8::Utils::ReportApiFailure(char const*, char const*) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Dictionary<v8::internal::SeededNumberDictionary, v8::internal::SeededNumberDictionaryShape, unsigned int>::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [node]
6: 0xc3c94d [node]
7: 0xc3cb92 [node]
8: v8::internal::JSObject::NormalizeElements(v8::internal::Handlev8::internal::JSObject) [node]
9: v8::internal::Runtime_NormalizeElements(int, v8::internal::Object**, v8::internal::Isolate*) [node]
10: 0x31116a2092a7
^C

App information (please complete the following information):

• App Type: Native App
• Postman Version: 6.2.1
• OS: Win x64

[kunagpal]

Workaround

This can be overcome by specifying a suitably large heap space size, as follows:

For CLI runs:

NODE_OPTIONS="--max-old-space-size=2048" Newman run ...

Or, if you're using Newman as a library,

node --max-old-space-size=2048 newman-script.js

In the command above, newman-script.js contains code that programmatically runs Postman collections with Neman.

A similar discussion has happened here: postmanlabs/newman#1516

Thinking of an actual fix

As can be seen from the discussion here: postmanlabs/newman#1497, we could consider the possibility of introducing a light mode flag that limits memory usage to the bare essentials.

[laschalkh]

I'm a little confused, when calling newman, can you pass the heap option directly from that command or do you have to go into the newman config files somewhere?

[kunagpal]

@laschalkh Yes, passing the heap space size override with the run command is possible. See the first example from my comment above 😄

[laschalkh]

aww, went right past that, I'm sorry. I'll give that a try. thank you!

[laschalkh]

I just tried another run.

I tried the 2k size and it still failed, I then up'ed to 4196, here is the start of my command:

NODE OPTIONS="--max-old-space=4196" newman run

There are about 50 concurrent jobs running on my linux box. It's running a few thousand total. After a few minutes I get the following:

FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x10a0f5c [node]
3: v8::Utils::ReportApiFailure(char const*, char const*) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Dictionary<v8::internal::SeededNumberDictionary, v8::internal::SeededNumberDictionaryShape, unsigned int>::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [node]
6: 0xc3c94d [node]
7: 0xc3cb92 [node]
8: v8::internal::JSObject::NormalizeElements(v8::internal::Handlev8::internal::JSObject) [node]
9: v8::internal::Runtime_NormalizeElements(int, v8::internal::Object**, v8::internal::Isolate*) [node]
10: 0x2ae9120092a7

[laschalkh]

I reduced the amount of data a query was requesting and the errors have stopped. Thanks for the help.