Memory leak or low memory?

[MichalLebeda]

npx dukascopy-node -i eurusd -from 2019-01-01 -to 2022-11-28 -t tick -f csv -v   
----------------------------------------------------
Downloading historical price data for:
----------------------------------------------------
Instrument:     Euro vs US Dollar
Timeframe:      tick
From date:      Jan 1, 2019, 12:00:00 AM
To date:        Sep 28, 2022, 11:58:33 AM
Price type:     bid
Volumes:        true
UTC Offset:     0
Include flats:  false
Format:         csv
----------------------------------------------------
|█████████████████████████████████████████████| 100%
<--- Last few GCs --->

[199416:0x595ddc0]  5837650 ms: Mark-sweep 1885.5 (2081.0) -> 1870.4 (2078.5) MB, 453.6 / 0.d
[199416:0x595ddc0]  5837857 ms: Scavenge 1884.6 (2079.0) -> 1875.3 (2079.0) MB, 5.8 / 0.0 ms 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap oy
 1: 0xa89e60 node::Abort() [node]
 2: 0x9ade29 node::FatalError(char const*, char const*) [node]
 3: 0xc7583e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xc75bb7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, ]
 5: 0xe3f6d5  [node]
 6: 0xe4027c  [node]
 7: 0xe4dc0b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal:]
 8: 0xe514f4 v8::internal::Heap::AllocateExternalBackingStore(std::function<void* (unsigned ]
 9: 0xf63f42 v8::internal::BackingStore::Allocate(v8::internal::Isolate*, unsigned long, v8:]
10: 0xcee8da  [node]
11: 0xceecdd  [node]
12: 0xcef0c6 v8::internal::Builtin_ArrayBufferConstructor(int, unsigned long*, v8::internal:]
13: 0x15046d9  [node]
[1]    199405 IOT instruction (core dumped)  npx dukascopy-node -i eurusd -from 2019-01-01 --

[Leo4815162342]

@MichalLebeda ~3.5+ years of tick data could potentially be dozens of gigabaytes of output. For now I'd recommend narrowing down the time period to a shorter time frame.

see: https://github.com/Leo4815162342/dukascopy-node/wiki/Downloading-tick-data#keep-in-mind

meanwhile, we'll take a look at how we can save large files in a more memory-efficient way

[MichalLebeda]

Thanks for explanation. Will try to download separate periods and merge manually using cat

[Leo4815162342]

@MichalLebeda please update dukascopy-node to the latest version (1.26.0) - the issue should be fixed

[Leo4815162342]

dukascopy-node@1.31.3 contains more performance improvements and addresses the memory leak issue

details > #119