High memory usage

[TiKaWe]

Hi everyone,

I'm not a python programmer, so I hired from Upwork to have my existing trading script fit into VectorBT. Since my trading script (also built by another programmer) is almost entirely based on pandas, the resulting VectorBT script seems to work.

However I feel it is taking too long and using too much memory, but I don't exactly know how to troubleshoot this.

My windows pc has 16GB of ram, give or take about 11Gb available. When I ran the script, it almost maxed out the entire memory. And took 1.5hrs to backtest.

My test case is to gather 5 trading days' worth of tickers (about 854 tickers) that my InteractiveBrokers scanner picked up and processed by my trading script. And have two strategies tested against this batch of 854 tickers.

I am using 3 timeframes (e.g. 1min, 5min, 15min), and one time frame (e.g. 1min) for exit conditions, so that's 4 dataframes per ticker. I tried pulling 900+ candles for 1 ticker for one timeframe, the dataframe is about 300+kb. 914 rows by 84 columns. So in total, all 854 tickers should be taking up at most 854 x 4 x 300kb = 1,024,800kb or 1GB. But from screenshot above, it maxed out my system memory.

My developer has sliced off excess rows (more rows pulled initially to generate accurate indicator figures such as SMA 200).

Is this right? I looked around on Youtube, came across downcasting datatype. but the developer I hired from Upwork claims downcasting datatype will not work cos VectorBT will upcast the datatype again on the backend and is beyond his control.

Is this true? Any other way to improve memory usage?

[polakowo]

@TiKaWe depends on how you use that data, what's contained in those 84 columns? Do you need every column? Are you running the pipeline through a loop to test each ticker, or simulate all tickers at the same time? Are you backtesting or how are you using vectorbt? But also, keeping 1GB of data in memory is already critical, ideally your data should be not longer than 200mb (and that's already a lot).