fix: Simplified RequestQueueV2 implementation #2775

janbuchar · 2024-12-17T14:58:49Z

closes Utilize queueHasLockedRequests to simplify RequestQueue v2 #2767
should resolve feedback from misc(RequestQueueV2): adding notes for request queue v2 implementation #2700

…nishes

packages/core/src/storages/request_queue_v2.ts

janbuchar · 2024-12-19T18:36:15Z

packages/core/src/storages/request_queue_v2.ts

+        const giveUpLock = async (id?: string, uniqueKey?: string) => {
+            if (id === undefined) {
+                return;
+            }
+
+            try {
+                await this.client.deleteRequestLock(id);
+            } catch {
+                this.log.debug('Failed to delete request lock', { id, uniqueKey });
+            }
+        };
+
+        // If we tried to read new forefront requests, but another client appeared in the meantime, we can't be sure we'll only read our requests.
+        // To retain the correct queue ordering, we rollback this head read.
+        if (hasPendingForefrontRequests && headData.hadMultipleClients) {
+            this.log.debug(`Skipping this read - forefront requests may not be fully consistent`);
+            await Promise.all(headData.items.map(({ id, uniqueKey }) => giveUpLock(id, uniqueKey)));
+        }
+


@barjin I'm pretty sure this is equivalent to the previous version, but please check it.

janbuchar · 2024-12-19T18:37:58Z

packages/core/src/storages/request_queue_v2.ts

+    /**
+     * @inheritDoc
+     */
+    override async isFinished(): Promise<boolean> {


@drobnikj I didn't remove the inheritance from RequestProvider completely, just overwrote this method. I don't think there's any other stuff in RequestProvider that could cause any trouble, but feel free to prove me wrong.

As I wrote, I would remove original implementation from Provider do not to confuse future developers.

drobnikj

Nice 💪

I would do some testing myself, but the first what about some unit tests, did you consider,add some? There are none -> https://github.com/apify/crawlee/blob/03951bdba8fb34f6bed00d1b68240ff7cd0bacbf/test/core/storages/request_queue.test.ts
Honesly, we are dealing with various bugs during time and we do not have any tests for these features still.

packages/core/src/storages/request_provider.ts

barjin · 2024-12-20T10:56:39Z

packages/core/src/storages/request_queue_v2.ts

+        // If we tried to read new forefront requests, but another client appeared in the meantime, we can't be sure we'll only read our requests.
+        // To retain the correct queue ordering, we rollback this head read.
+        if (hasPendingForefrontRequests && headData.hadMultipleClients) {
+            this.log.debug(`Skipping this read - forefront requests may not be fully consistent`);
+            await Promise.all(headData.items.map(async ({ id, uniqueKey }) => giveUpLock(id, uniqueKey)));
+        }


I was thinking about those scenarios during the last batch of forefront fixes, but I considered them too edge case-y to handle them.

I'm not sure about these changes, though - let's say two clients are using the queue. Won't this cause any forefront request to be locked and then immediately unlocked? headData.hadMultipleClients doesn't iirc say "a new client just appeared", once it's set, it's never false again, no? 🤔

I considered them too edge case-y to handle them

My point here: I'd be happy not to do this at all. Both the clients are using the same queue - as a user, I'd be fine with some request intermingling.

drobnikj · 2025-01-06T10:04:39Z

The build did not finish, can you check @janbuchar ?
I would like to test it in some Actors.

janbuchar · 2025-01-06T11:55:30Z

The build did not finish, can you check @janbuchar ? I would like to test it in some Actors.

I can, but only later this week - I have different stuff to finish first.

janbuchar · 2025-01-23T17:08:35Z

@drobnikj the unit tests are now passing so you should be able to build. I'm still working on some e2e tests, if you have any ideas for scenarios to test (e2e, unit, doesn't matter), I'd love to hear those.

drobnikj

Looks good, I did not find any issue, even during testing.
I have a few more comments, can you check pls? @janbuchar

drobnikj · 2025-01-30T10:44:47Z

packages/core/src/storages/request_queue_v2.ts


        const headData = await this.client.listAndLockHead({
-            limit: Math.min(forefront ? this.assumedForefrontCount : 25, 25),
+            limit: Math.min(hasPendingForefrontRequests ? this.assumedForefrontCount : 25, 25),
            lockSecs: this.requestLockSecs,


I checked where requestLockSecs was set and I think we should consider to change it.
The currrent is 2x of requestHandlerTimeoutSecs plus 5 secs.

crawlee/packages/basic-crawler/src/internals/basic-crawler.ts

Line 676 in af2966f

this.requestQueue.requestLockSecs = Math.max(this.internalTimeoutMillis / 1000 + 5, 60);

I think we should set it to
requestHandlerTimeoutSecs plus some safe buffer.

drobnikj · 2025-01-30T11:10:06Z

packages/core/src/storages/request_queue_v2.ts

@@ -361,7 +430,8 @@ export class RequestQueue extends RequestProvider {



I cannot comment it below but during code review, I see that we are removing locks one by one in _clearPossibleLock.
see

crawlee/packages/core/src/storages/request_queue_v2.ts

Line 446 in 092d60c

while ((requestId = this.queueHeadIds.removeFirst()) !== null) {

There is 200 rps rate limit. I would remove lock in some batches maybe 10 to speed it up.

drobnikj · 2025-01-30T11:18:04Z

packages/core/src/storages/request_queue_v2.ts

@@ -53,7 +53,8 @@ const RECENTLY_HANDLED_CACHE_SIZE = 1000;
 * @category Sources
 */
 export class RequestQueue extends RequestProvider {
-    private _listHeadAndLockPromise: Promise<void> | null = null;
+    private listHeadAndLockPromise: Promise<void> | null = null;
+    private queueHasLockedRequests: boolean | undefined = undefined;

    /**
     * Returns `true` if there are any requests in the queue that were enqueued to the forefront.


Can you please update this function regarding notes in
https://github.com/apify/apify-core/issues/19218#issuecomment-2621466723 ?

drobnikj · 2025-01-30T13:31:48Z

packages/core/src/storages/request_queue_v2.ts

+        }
+
+        if (this.queueHasLockedRequests !== undefined) {
+            return !this.queueHasLockedRequests;


Can we add at least a debug or info log if there are still locked requests? It means that there are locked requests by another client.

janbuchar · 2025-01-30T14:27:04Z

@drobnikj thanks! You really went above and beyond for this one. I'll try to wrap this up today/tomorrow.

janbuchar added 4 commits December 12, 2024 15:48

Make a separate isFinished implementation for RQv2

4b662c2

Do not use recentlyHandledRequestsCache in RQv2

0529e12

Add queueHasLockedRequests field

8407033

RQv2-specific implementation of isFinished()

7fbecc4

janbuchar added the t-tooling Issues with this label are in the ownership of the tooling team. label Dec 17, 2024

github-actions bot assigned janbuchar Dec 17, 2024

github-actions bot added this to the 105th sprint - Tooling team milestone Dec 17, 2024

janbuchar marked this pull request as draft December 17, 2024 14:59

janbuchar added 3 commits December 18, 2024 12:20

Track request batch addition so that we do not terminate before it fi…

b94a9c8

…nishes

Do not return additional requests when paused for migration

a40d04e

Slightly simplify forefront request handling

07b3717

janbuchar requested review from barjin, B4nan, drobnikj and vladfrangu December 19, 2024 18:34

janbuchar commented Dec 19, 2024

View reviewed changes

Resolve lint errors

092d60c

drobnikj reviewed Dec 20, 2024

View reviewed changes

packages/core/src/storages/request_provider.ts Outdated Show resolved Hide resolved

barjin reviewed Dec 20, 2024

View reviewed changes

janbuchar added 3 commits January 21, 2025 12:19

Merge remote-tracking branch 'origin/master' into simplify-rq-v2

edf73e9

Remove unused RequestProvider.isFinished implementation

c4b45cb

Update tests, remove too implementation-dependent assertions

efea4ff

github-actions bot added the tested Temporary label used only programatically for some analytics. label Jan 22, 2025

Bug

fe3e129

drobnikj self-requested a review January 30, 2025 10:41

drobnikj reviewed Jan 30, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Simplified RequestQueueV2 implementation #2775

fix: Simplified RequestQueueV2 implementation #2775

janbuchar commented Dec 17, 2024 •

edited

Loading

janbuchar Dec 19, 2024

janbuchar Dec 19, 2024

drobnikj Dec 20, 2024

drobnikj left a comment

barjin Dec 20, 2024 •

edited

Loading

barjin Dec 20, 2024

drobnikj commented Jan 6, 2025

janbuchar commented Jan 6, 2025

janbuchar commented Jan 23, 2025

drobnikj left a comment •

edited

Loading

drobnikj Jan 30, 2025

drobnikj Jan 30, 2025

drobnikj Jan 30, 2025

drobnikj Jan 30, 2025

janbuchar commented Jan 30, 2025

		@@ -361,7 +430,8 @@ export class RequestQueue extends RequestProvider {

fix: Simplified RequestQueueV2 implementation #2775

Are you sure you want to change the base?

fix: Simplified RequestQueueV2 implementation #2775

Conversation

janbuchar commented Dec 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drobnikj left a comment

Choose a reason for hiding this comment

barjin Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drobnikj commented Jan 6, 2025

janbuchar commented Jan 6, 2025

janbuchar commented Jan 23, 2025

drobnikj left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janbuchar commented Jan 30, 2025

janbuchar commented Dec 17, 2024 •

edited

Loading

barjin Dec 20, 2024 •

edited

Loading

drobnikj left a comment •

edited

Loading