Backfill stream returns after first iteration when parallelism is set to 1

[0xalex88]

Describe the bug

Not sure if it's expected but I was trying to use parallelism = 1 in a backfill job and in that case, the stream ends after returning a single chunk, instead of continuing executing the next chunk in the full backfill range.

Steps to reproduce

Simplest reproduction I've found is in the test_batch test, if you change the code to:

let factory = BackfillJobFactory::new(executor.clone(), blockchain_db.clone())
            .with_thresholds(ExecutionStageThresholds { max_blocks: Some(1), ..Default::default() });
        let mut backfill_stream = factory.backfill(1..=2).into_stream();
        let mut chain = backfill_stream.next().await.unwrap().unwrap();
        chain.execution_outcome_mut().state_mut().reverts.sort();

        assert!(chain.blocks_iter().eq(&blocks[0..1]));

        // expect no more blocks
        assert!(backfill_stream.next().await.is_none());

so backfilling one block at a time it fails because assert!(backfill_stream.next().await.is_none()) is not none (as expected) however when using

let factory = BackfillJobFactory::new(executor.clone(), blockchain_db.clone())
            .with_thresholds(ExecutionStageThresholds { max_blocks: Some(1), ..Default::default() })
            .with_stream_parallelism(1);
        let mut backfill_stream = factory.backfill(1..=2).into_stream();
        let mut chain = backfill_stream.next().await.unwrap().unwrap();
        chain.execution_outcome_mut().state_mut().reverts.sort();

        assert!(chain.blocks_iter().eq(&blocks[0..1]));

        // expect no more blocks
        assert!(backfill_stream.next().await.is_none());

so adding the stream parallelism 1,next() returns none, ending the stream

Node logs

2024-12-10T19:22:57.969204Z DEBUG exex::backfill: Executing block range range=0..=499999
2024-12-10T19:23:16.131201Z DEBUG exex::backfill: Finished executing block range range=0..=499999 block_fetch=2.753609197s execution=15.289312944s throughput="1.89 Ggas/second"
(and then the stream returns)

Platform(s)

No response

Container Type

Not running in a container

What version/commit are you on?

v1.1.3

What database version are you on?

2

Which chain / network are you on?

dev

What type of node are you running?

Archive (default)

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[mattsse]

the stream ends after returning a single chunk, instead of continuing executing the next chunk in the full backfill range.

so backfilling one block at a time it fails because assert!(backfill_stream.next().await.is_none()) is not none (as expected)

these two are conflicting, which one is it?

@0xalex88 to me it feels like you're mixing your actual issue together with this repro? but I'm not exactly sure

@shekhirin the reason this test setup fails is because we don't have that block 1 in the test database setup hence the stream returns an error. so imo this repro seems unrelated

[mattsse]

@shekhirin the second repro is a logic bug if parallelism is one

because we can end up returning None here

reth/crates/exex/exex/src/backfill/stream.rs

Line 113 in 37f3933

Poll::Ready(None)

but never spawn the second task for block 2

[0xalex88]

these two are conflicting, which one is it?

in the first quote I'm mentioning my use case, where backfilling from block 0 to block 13M (chain tip) spawns a task with range 0..=50k but nothing more, so the remaining 50k-13M blocks are not backfilled

in the second quote I'm pointing out the case where the test fails when parallelism = 2, which is correct (because there's one more block range to process) but the same test doesn't fail (even if it should) when parallelism = 1

[mattsse]

gotcha, was a bit confused at first, maybe because it's late here.

there's indeed an issue with parallelism 1

[0xalex88]

gotcha, was a bit confused at first, maybe because it's late here.

there's indeed an issue with parallelism 1

it wasn't very clear, my bad!
anyway thanks for the fix, I've tested it and it works as expected now!