Improve handling of progress conditions in non-polling fetch

[ballard26]

The use of ss::when_any and ss::promise<> for each progress condition resulted in unneeded allocations as well as expensive broken promises and exception propagation. This commit changes all progress conditions to signal the same ss::condition_variable instead. This simplifies their handling and greatly reduces their cost.

The profile for progress conditions before this PR;

And then after this PR;

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v24.1.x
• v23.3.x
• v23.2.x

Release Notes

• none

[vbotbuildovich]

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/50475#01903422-bccb-4400-b98a-c16d58307095

[vbotbuildovich]

new failures in https://buildkite.com/redpanda/redpanda/builds/50475#01903423-12e4-4950-bdd0-0a5cbe7823d7:

"rptest.tests.maintenance_test.MaintenanceTest.test_exclusive_maintenance.use_rpk=False"

new failures in https://buildkite.com/redpanda/redpanda/builds/50475#01903423-12e2-459e-ba79-81b4d832cc07:

"rptest.tests.raft_availability_test.RaftAvailabilityTest.test_leadership_transfer"

[ballard26]

/ci-repeat 5

[ballard26]

All failures look to be known issues;

• rptest.tests.raft_availability_test.RaftAvailabilityTest.test_leadership_transfer is CI Failure (MetricCheckFailed) in RaftAvailabilityTest. test_leadership_transfer  #19953
• rptest.tests.consumer_group_balancing_test.ConsumerGroupBalancingTest.test_coordinator_nodes_balance is CI Failure (RpkException: unable to produce record: records have timed out before they were able to be produced) in ConsumerGroupBalancingTest.test_coordinator_nodes_balance #17176
• rptest.tests.full_disk_test.FullDiskReclaimTest.test_full_disk_triggers_gc is CI Failure (Data never drops below 1MB) in FullDiskReclaimTest.test_full_disk_triggers_gc #18014
• rptest.tests.maintenance_test.MaintenanceTest.test_exclusive_maintenance is CI Failure (Node never became leader) in MaintenanceTest.test_exclusive_maintenance #20592

[piyushredpanda]

#20574 and #20592 in latest runs

[vbotbuildovich]

/backport v24.1.x

[vbotbuildovich]

/backport v23.3.x

[vbotbuildovich]

/backport v23.2.x

[vbotbuildovich]

Failed to create a backport PR to v24.1.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-19930-v24.1.x-291 remotes/upstream/v24.1.x
git cherry-pick -x 8fd8fcdf1c3a8d8ed1eb67d5649fddb63286c9c3

Workflow run logs.

[vbotbuildovich]

Failed to create a backport PR to v23.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-19930-v23.2.x-619 remotes/upstream/v23.2.x
git cherry-pick -x 8fd8fcdf1c3a8d8ed1eb67d5649fddb63286c9c3

Workflow run logs.

[vbotbuildovich]

Failed to create a backport PR to v23.3.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-19930-v23.3.x-368 remotes/upstream/v23.3.x
git cherry-pick -x 8fd8fcdf1c3a8d8ed1eb67d5649fddb63286c9c3

Workflow run logs.