🐛 Scraper Memory usage

[ulfgebhardt]

🐛 Bugreport

When using AsyncIterator i have a substential memory leak when used in for-x-of-y

I need this when scraping a HTML-Page which includes the information about the next HTML-Page to be scraped:

1. Scrap Data
2. Evaluate Data
3. Scrape Next Data

The async Part is needed since axios is used to obtain the HTML

Here is a repro, which allows to see the memory rising von ~4MB to ~25MB at the end of the script. The memory is not freed till the program terminates.

const scraper = async ():Promise<void> => {
    let browser = new BrowserTest();
    let parser = new ParserTest();

    for await (const data of browser){
        console.log(await parser.parse(data))
    }
}

class BrowserTest {
    private i: number = 0;

    public async next(): Promise<IteratorResult<string>> {
        this.i += 1;
        return {
            done: this.i > 1000,
            value: 'peter '.repeat(this.i)
        }
    }

    [Symbol.asyncIterator](): AsyncIterator<string> {
        return this;
    }
}

class ParserTest {
    public async parse(data: string): Promise<string[]> {
        return data.split(' ');
    }
}

scraper()

It looks like that the data of the for-await-x-of-y is dangling in memory. The callstack gets huge aswell.

In the repro the Problem could still be handled. But for my actual code a whole HTML-Page stays in memory which is ~250kb each call.

In this screenshot you can see the heap memory on the first iteration compared to the heap memory after the last iteration

The expected workflow would be the following:

• Obtain Data
• Process Data
• Extract Info for the next "Obtain Data"
• Free all Memory from the last "Obtain Data"
• Use extracted information to restart the loop with new Data obtained.

I am unsure an AsyncIterator is the right choice here to archive what is needed.

Any help/hint would be appriciated!

See: https://stackoverflow.com/questions/58454833/for-await-x-of-y-using-an-asynciterator-causes-memory-leak

[ulfgebhardt]

Contrary to the original Post i now believe that the memory is freed once the loop is completed.

[ulfgebhardt]

Requirements for our scrapers:

Data which is dependent on last Dataset:

a references b references c
The Request of the ScraperBrowser must be completed in order to obtain the next Dataset.
To reduce http requests it would be nice to return the html from the Browser and give it to the Parser.

Data which has an index:

index references n urls
The Request of the ScraperBrowser returns n datasets which might be paginated. The Datasets are not needed to obtain the next ones.
To enable parallelization it would be nice to return Promises for the HTML-Requests from the Browser and give it to the Parser.