[v24.1.x] Improve handling of progress conditions in non-polling fetch

[vbotbuildovich]

Backport #19930 to branch v24.1.x. Note that this issue was created as a placeholder, since the original PR's commit(s) could not be automatically cherry-picked.
Here are the commands to execute:

git checkout v24.1.x
git checkout -b vbotbuildovich/backport-19930-v24.1.x-856
git cherry-pick -x 8fd8fcdf1c3a8d8ed1eb67d5649fddb63286c9c3

git push origin vbotbuildovich/backport-19930-v24.1.x-856
gh pr create \
  --title "[v24.1.x] Improve handling of progress conditions in non-polling fetch" \
  --base "v24.1.x" \
  --label "kind/backport" \
  --head "vbotbuildovich/backport-19930-v24.1.x-856" \
  --draft \
  --repo "redpanda-data/redpanda" \
  --milestone "v24.1.x-next" \
  --body "Backport of PR https://github.com/redpanda-data/redpanda/pull/19930 "

JIRA Link: [CORE-5094](https://redpandadata.atlassian.net/browse/CORE-5094)

[BenPope]

@ballard26 is this blocked on anything?