How to specify concurrency based on request handler's LABEL?

[harshmaur]

Hi, I wanted to understand if there's a way in crawlee to control the concurrency of requests based on the label property.

For example, LISTING page can have a concurrency of 1, but PRODUCT page have a concurrency of 10 ?

Is that possible?

[janbuchar]

Hello, I don't think this is possible "out of the box". May I ask what the use case is?

In any case, it should be possible to implement the desired behavior by extending and adjusting the AutoscaledPool class and also overriding the _init method of your crawler, but it would take some research.

An alternative approach may be to run two crawlers in parallel, with separate request queues, each for one "concurrency group".